docs/scripting.xml - master - githubmirror/nmap

docs/scripting.xml
3 993 строки178 KB
dmiller
Update docs for Lua 5.4.4 upgrade
01 май 2023, 20:44
01 май 2023, 20:446fdccee
<chapter id="nse"><title>Nmap Scripting Engine</title>
<indexterm class="startofrange" id="nse-indexterm"><primary>Nmap Scripting Engine (NSE)</primary></indexterm>
<indexterm><primary>scripting</primary><see>Nmap Scripting Engine</see></indexterm>
<indexterm><primary>NSE</primary><see>Nmap Scripting Engine</see></indexterm>

<sect1 id="nse-intro">
    <title>Introduction</title>

    <para>The Nmap Scripting Engine (NSE) is one of Nmap's most
    powerful and flexible features.  It allows users to write (and
    share) simple scripts to automate a wide variety of networking
    tasks.  Those scripts are then executed in parallel with the speed
    and efficiency you expect from Nmap.  Users can rely on the
    growing and diverse set of scripts distributed with Nmap, or write
    their own to meet custom needs.</para>

    <para>We designed NSE to be versatile, with the following tasks in mind:</para>

    <variablelist>
      <varlistentry>
        <term>Network discovery</term>
        <listitem>
          <para>This is Nmap's bread and butter. Examples include
           looking up whois data based on the target domain,
           querying ARIN, RIPE, or APNIC for the target IP to determine ownership,
           performing identd lookups on open ports, SNMP queries, and
           listing available NFS/SMB/RPC shares and services.</para>
        </listitem>

      </varlistentry>
      <varlistentry>
        <term>
        <indexterm><primary>version detection</primary><secondary>using NSE</secondary></indexterm>
        More sophisticated version detection</term>
        <listitem>
          <para>The Nmap version detection system (<xref linkend="vscan"/>)
       is able to recognize thousands of different services through
       its probe and regular expression signature based matching system, but it
       cannot recognize everything.  For example, identifying the Skype v2 service requires two
       independent probes, which version detection isn't flexible enough to handle.  Nmap could also recognize more SNMP services
       if it tried a few hundred different community names by brute
       force.  Neither of these tasks are well suited to traditional
       Nmap version detection, but both are easily accomplished with
       NSE.  For these reasons, version detection now calls NSE by
       default to handle some tricky services.  This is described in
       <xref linkend="nse-vscan"/>.</para>

        </listitem>
      </varlistentry>
      <varlistentry>
        <term>
        <indexterm><primary>vulnerability detection</primary></indexterm>
        Vulnerability detection</term>
        <listitem>

          <para>When a new vulnerability is discovered, you often want
       to scan your networks quickly to identify vulnerable systems
       before the bad guys do.  While Nmap isn't a
       comprehensive <ulink role="hidepdf" url="https://sectools.org/vuln-scanners.html">vulnerability scanner</ulink>,
       NSE is powerful enough to handle even demanding vulnerability
       checks. When the Heartbleed bug affected hundreds of thousands of
       systems worldwide, Nmap's developers responded with the
       <literal>ssl-heartbleed</literal> detection script within 2 days.
       Many vulnerability detection scripts are already available and we plan to distribute more as they are written.
       </para>

        </listitem>
      </varlistentry>
      <varlistentry>
        <term>Backdoor detection</term>
        <listitem>
          <para>
	   Many attackers and some automated worms leave backdoors to
	   enable later reentry.  Some of these can be detected by
     Nmap's regular expression based version detection, but more complex worms
     and backdoors require NSE's advanced capabilities to reliably detect.
     NSE has been used to detect the Double Pulsar NSA backdoor in SMB and
     backdoored versions of UnrealIRCd, vsftpd, and ProFTPd.
       </para>

        </listitem>
      </varlistentry>
      <varlistentry>
        <term>Vulnerability exploitation</term>
        <listitem>
          <para>
	   As a general scripting language, NSE can even
	   be used to exploit vulnerabilities rather than just find them.
	   The capability to add custom exploit scripts may be valuable
	   for some people (particularly
	   penetration testers),<indexterm><primary>penetration testing</primary></indexterm>
           though we aren't
	   planning to turn Nmap into an exploitation framework such as
	   <ulink url="http://www.metasploit.com">Metasploit</ulink>.<indexterm><primary><application>Metasploit</application></primary></indexterm>
       </para>
        </listitem>
      </varlistentry>
    </variablelist>
    <para>
      These listed items were our initial goals, and we expect Nmap
      users to come up with even more inventive uses for NSE.
    </para>

    <para>
      Scripts are written in the
      embedded
      <ulink url="https://lua.org/">Lua programming language</ulink>, version 5.4.<indexterm><primary>Lua programming language</primary><seealso>Nmap Scripting Engine</seealso></indexterm>
      The language itself is well documented in the books
<web>
      <citetitle><ulink url="http://www.amazon.com/dp/8590379868?tag=secbks-20">Programming
	in Lua, Fourth Edition</ulink></citetitle> and
      <citetitle><ulink url="http://www.amazon.com/dp/9888381229?tag=secbks-20">Lua
	5.2 Reference Manual</ulink></citetitle>.
</web>
<print>
      <citetitle>Programming in Lua, Fourth Edition</citetitle> and
      <citetitle>Lua 5.2 Reference Manual</citetitle>.
</print>

The reference manual, updated for Lua 5.4, is also
      <ulink url="https://lua.org/manual/5.4/">freely available
        online</ulink>, as is the
      <ulink url="https://lua.org/pil/">first edition of <citetitle>Programming in
  Lua</citetitle></ulink>. Given the availability of these excellent general
      Lua programming references, this document only covers aspects and
      extensions specific to Nmap's scripting engine.
    </para>
    <para>

      NSE is activated with the <option>-sC</option> option (or
      <option>--script</option> if you wish to specify a custom set of
      scripts) and results are integrated into Nmap
      normal<indexterm><primary>normal output</primary></indexterm>
      and XML output.<indexterm><primary>XML output</primary></indexterm>
    </para>

      <para>
        A typical script scan is shown in the
      <xref linkend="nse-ex1" xrefstyle="select: label nopage"/>.
        Service scripts producing output in this example are
      <literal>ssh-hostkey</literal>, which provides the system's RSA and DSA SSH keys, and <literal>rpcinfo</literal>, which queries
      portmapper to enumerate available services.  The only host
      script producing output in this example
      is <literal>smb-os-discovery</literal>, which collects a variety of
      information from SMB servers.<indexterm><primary>script names, examples
      of</primary></indexterm>  Nmap discovered all of this information in a third of a second.</para>

<example id="nse-ex1"><title>Typical NSE output</title><indexterm><primary><option>-sC</option></primary><secondary>example of</secondary></indexterm>
<screen>
# <userinput>nmap -sC -p22,111,139 -T4 localhost</userinput>

Starting Nmap ( https://nmap.org )
Nmap scan report for flog (127.0.0.1)
PORT     STATE SERVICE
22/tcp   open  ssh
| ssh-hostkey: 1024 b1:36:0d:3f:50:dc:13:96:b2:6e:34:39:0d:9b:1a:38 (DSA)
|_2048 77:d0:20:1c:44:1f:87:a0:30:aa:85:cf:e8:ca:4c:11 (RSA)
111/tcp  open  rpcbind
| rpcinfo:  
| 100000  2,3,4    111/udp  rpcbind  
| 100024  1      56454/udp  status   
|_100000  2,3,4    111/tcp  rpcbind  
139/tcp  open  netbios-ssn

Host script results:
| smb-os-discovery: Unix
| LAN Manager: Samba 3.0.31-0.fc8
|_Name: WORKGROUP

Nmap done: 1 IP address (1 host up) scanned in 0.33 seconds
</screen>
</example>

<para>A 38-minute video introduction to NSE is available at
<ulink url="https://nmap.org/presentations/BHDC10/"/>.  This
presentation was given by Fyodor and David Fifield at Defcon and the
Black Hat Briefings in 2010.</para>



  </sect1>
  <sect1 id="nse-usage">
    <title>Usage and Examples</title>
    <para>
      While NSE has a complex implementation for efficiency, it is
      strikingly easy to use.  Simply specify
      <option>-sC</option><indexterm><primary><option>-sC</option></primary></indexterm>
      to enable the most common scripts.  Or specify the
      <option>--script</option><indexterm><primary><option>--script</option></primary></indexterm>
      option to choose your own scripts to
      execute by providing categories, script file names, or the name of
      directories full of scripts you wish to execute.  You can customize 
	  some scripts by providing arguments to them via the
	  <option>--script-args</option><indexterm><primary><option>--script-args</option></primary></indexterm> and <option>--script-args-file</option><indexterm><primary><option>--script-args-file</option></primary></indexterm>
          options.
          The <option>--script-help</option><indexterm><primary><option>--script-help</option></primary></indexterm>
          shows a description of what each selected script does.
          The two remaining options,
	  <option>--script-trace</option><indexterm><primary><option>--script-trace</option></primary></indexterm>
          and <option>--script-updatedb</option>,<indexterm><primary><option>--script-updatedb</option></primary></indexterm>
          are generally only used for script debugging and development.  Script scanning is also included as part of the <option>-A</option> (aggressive scan) option.
    </para>

    <para>
    Script scanning is normally done in combination with a port scan,
    because scripts may be run or not run depending on the port states
    found by the scan. With the <option>-sn</option> option it is
    possible to run a script scan without a port scan, only host
    discovery. In this case only host scripts will be eligible to run.
    To run a script scan with neither a host discovery nor a port scan,
    use the <option>-Pn -sn</option> options together with
    <option>-sC</option> or <option>--script</option>. Every host will
    be assumed up and still only host scripts will be run. This
    technique is useful for scripts like
    <filename>whois-ip</filename><indexterm><primary><filename>whois-ip</filename> script</primary></indexterm>
    that only use the remote system's address and don't require it to be
    up.
    </para>

    <para>
    Scripts are not run in a sandbox and thus could accidentally or
    maliciously damage your system or invade your privacy.  Never run
    scripts from third parties unless you trust the authors or have
    carefully audited the scripts yourself.
    </para>

    <sect2 id="nse-categories"><title>Script Categories</title>
    <indexterm><primary>script categories</primary></indexterm>

    <para>NSE scripts define a list of categories they belong to.
    Currently defined categories are
    <literal>auth</literal>,
    <literal>broadcast</literal>,
    <literal>brute</literal>,
    <literal>default</literal>.
    <literal>discovery</literal>,
    <literal>dos</literal>,
    <literal>exploit</literal>,
    <literal>external</literal>,
    <literal>fuzzer</literal>,
    <literal>intrusive</literal>,
    <literal>malware</literal>,
    <literal>safe</literal>,
    <literal>version</literal>, and
    <literal>vuln</literal>.
    Category names are not case sensitive.  The following list describes each category.</para>

     <variablelist>

        <varlistentry>
          <term id="nse-category-auth">
            <indexterm><primary sortas="auth script category">&ldquo;<literal>auth</literal>&rdquo; script category</primary></indexterm>
            <option>auth</option>
          </term>
          <listitem>
            <para>These scripts deal with authentication credentials (or bypassing them) on the target system.  Examples include <literal>x11-access</literal>, <literal>ftp-anon</literal>, and <literal>oracle-enum-users</literal>.  Scripts which use brute force attacks to determine credentials are placed in the <literal>brute</literal> category instead.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term id="nse-category-broadcast">
            <indexterm><primary sortas="broadcast script category">&ldquo;<literal>broadcast</literal>&rdquo; script category</primary></indexterm>
            <option>broadcast</option>
          </term>
          <listitem>
            <para>Scripts in this category typically do discovery of hosts
            not listed on the command line by broadcasting on the local network.
            Use the
            <varname>newtargets</varname><indexterm><primary><varname>newtargets</varname> script argument</primary></indexterm>
            script argument to allow these scripts to automatically add the
            hosts they discover to the Nmap scanning queue.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term id="nse-category-brute">
            <indexterm><primary sortas="brute script category">&ldquo;<literal>brute</literal>&rdquo; script category</primary></indexterm>
            <option>brute</option>
          </term>
          <listitem>
            <para>These scripts use brute force attacks to guess authentication credentials of a remote server.  Nmap contains scripts for brute forcing dozens of protocols, including <literal>http-brute</literal>, <literal>oracle-brute</literal>, <literal>snmp-brute</literal>, etc.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term id="nse-category-default">
            <indexterm><primary sortas="default script category">&ldquo;<literal>default</literal>&rdquo; script category</primary></indexterm>
            <option>default</option>
          </term>
          <listitem>

            <para>These scripts are the default set and are run when
            using the <option>-sC</option> or <option>-A</option>
            options rather than listing scripts
            with <option>--script</option>. This category can also be
            specified explicitly like any other
            using <option>--script=default</option>.  Many factors are
            considered in deciding whether a script should be run by
            default:</para>

     <variablelist>

        <varlistentry>
        <term>Speed</term>
        <listitem><para>A default scan must finish quickly, which excludes brute force authentication crackers, web spiders, and any other scripts which can take minutes or hours to scan a single service.</para></listitem>
        </varlistentry>

        <varlistentry>
        <term>Usefulness</term>
        <listitem><para>Default scans need to produce valuable and
	actionable information.  If even the script author has trouble
	explaining why an average networking or security professional
	would find the output valuable, the script should not run by
	default.</para></listitem>
        </varlistentry>

        <varlistentry>
        <term>Verbosity</term>
        <listitem><para>Nmap output is used for a wide variety of
        purposes and needs to be readable and concise.  A script which
        frequently produces pages full of output should not be added
        to the <literal>default</literal> category.  When there is no
        important information to report, NSE scripts (particularly
        default ones) should return nothing.  Checking for an obscure
        vulnerability may be OK by default as long as it only produces output
        when that vulnerability is discovered.</para></listitem>
        </varlistentry>

        <varlistentry>
        <term>Reliability</term>
        <listitem><para>Many scripts use heuristics and fuzzy signature matching to reach conclusions about the target host or service.  Examples include <literal>sniffer-detect</literal> and <literal>sql-injection</literal>.  If the script is often wrong, it doesn't belong in the <literal>default</literal> category where it may confuse or mislead casual users.  Users who specify a script or category directly are generally more advanced and likely know how the script works or at least where to find its documentation.</para></listitem>
        </varlistentry>

        <varlistentry>
        <term>Intrusiveness</term>
        <listitem><para>Some scripts are very intrusive because they use significant resources on the remote system, are likely to crash the system or service, or are likely to be perceived as an attack by the remote administrators.  The more intrusive a script is, the less suitable it is for the <literal>default</literal> category.  Default scripts are almost always in the <literal>safe</literal> category too, though we occasionally allow <literal>intrusive</literal> scripts by default when they are only mildly intrusive and score well in the other factors.</para></listitem>
        </varlistentry>

        <varlistentry>
        <term>Privacy</term>
        <listitem><para>Some scripts, particularly those in the <literal>external</literal> category described later, divulge information to third parties by their very nature.  For example, the <literal>whois</literal> script must divulge the target IP address to regional whois registries.  We have also considered (and decided against) adding scripts which check target SSH and SSL key fingerprints against Internet weak key databases.  The more privacy-invasive a script is, the less suitable it is for <literal>default</literal> category inclusion.</para></listitem>
        </varlistentry>
      </variablelist>

      <para>We don't have exact thresholds for each of these criteria,
      and many of them are subjective.  All of these factors are
      considered together when making a decision whether to promote a
      script into the <literal>default</literal> category.  A few default scripts are <literal>identd-owners</literal> (determines the username running remote services using identd), <literal>http-auth</literal> (obtains authentication scheme and realm of web sites requiring authentication), and <literal>ftp-anon</literal> (tests whether an FTP server allows anonymous access).</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term id="nse-category-discovery">
            <indexterm><primary sortas="discovery script category">&ldquo;<literal>discovery</literal>&rdquo; script category</primary></indexterm>
            <option>discovery</option>
          </term>
          <listitem>
            <para>These scripts try to actively discover more about the
            network by querying public registries, SNMP-enabled
            devices, directory services, and the like.  Examples include <literal>html-title</literal> (obtains the title of the root path of web sites), <literal>smb-enum-shares</literal> (enumerates Windows shares), and <literal>snmp-sysdescr</literal> (extracts system details via SNMP).</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term id="nse-category-dos">
            <indexterm><primary sortas="dos script category">&ldquo;<literal>dos</literal>&rdquo; script category</primary></indexterm>
            <option>dos</option>
          </term>
          <listitem>
            <para>Scripts in this category may cause a denial of
            service.  Sometimes this is done to test vulnerability to
	    a denial of service method, but more commonly it is
	    an undesired by necessary side effect of testing for
	    a traditional vulnerability.  These tests sometimes crash
	    vulnerable services.</para>

          </listitem>
        </varlistentry>

        <varlistentry>
          <term id="nse-category-exploit">
            <indexterm><primary sortas="exploit script category">&ldquo;<literal>exploit</literal>&rdquo; script category</primary></indexterm>
            <option>exploit</option>
          </term>
          <listitem>
            <para>These scripts aim to actively exploit some vulnerability. Examples include <literal>jdwp-exec</literal> and <literal>http-shellshock</literal>.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term id="nse-category-external">
            <indexterm><primary sortas="external script category">&ldquo;<literal>external</literal>&rdquo; script category</primary></indexterm>
            <option>external</option>
          </term>
          <listitem>
            <para>Scripts in this category may send data to a
            third-party database or other network resource. An example
            of this is <literal>whois-ip</literal>, which makes a
            connection to
            whois<indexterm><primary>whois</primary></indexterm> servers
            to learn about the address of the target. There is always
            the possibility that operators of the third-party
            database will record anything you send to them, which in
            many cases will include your IP address and the address of
            the target. Most scripts involve traffic strictly between
            the scanning computer and the client; any that do not are
            placed in this category.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term id="nse-category-fuzzer">
            <indexterm><primary sortas="fuzzer script category">&ldquo;<literal>fuzzer</literal>&rdquo; script category</primary></indexterm>
            <option>fuzzer</option>
          </term>
          <listitem>
            <para>This category contains scripts which are designed to send server software unexpected or randomized fields in each packet. While this technique can useful for finding undiscovered bugs and vulnerabilities in software, it is both a slow process and bandwidth intensive.
            An example of a script in this category is <literal>dns-fuzz</literal>, which bombards a DNS server with slightly flawed domain requests until either the server crashes or a user specified time limit elapses.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term id="nse-category-intrusive">
            <indexterm><primary sortas="intrusive script category">&ldquo;<literal>intrusive</literal>&rdquo; script category</primary></indexterm>
            <option>intrusive</option>
          </term>
          <listitem>

            <para>These are scripts that cannot be classified in the
            <literal>safe</literal> category because the risks are too
            high that they will crash the target system, use up
            significant resources on the target host (such as
            bandwidth or CPU time), or otherwise be perceived as
            malicious by the target's system administrators.  Examples
            are <literal>http-open-proxy</literal> (which attempts to
            use the target server as an HTTP proxy)
            and <literal>snmp-brute</literal> (which tries to guess a
            device's SNMP community string by sending common values
            such
            as <literal>public</literal>, <literal>private</literal>,
            and <literal>cisco</literal>).  Unless a script is in the special <literal>version</literal> category, it should be categorized as either <literal>safe</literal> or <literal>intrusive</literal>.</para>

          </listitem>
        </varlistentry>

        <varlistentry>
          <term id="nse-category-malware">
            <indexterm><primary sortas="malware script category">&ldquo;<literal>malware</literal>&rdquo; script category</primary></indexterm>
            <option>malware</option>
          </term>
          <listitem>
            <para>These scripts test whether the target platform is
            infected by malware or backdoors.  Examples include <literal>smtp-strangeport</literal>, which watches for SMTP servers running on unusual port numbers, and <literal>auth-spoof</literal>, which detects identd spoofing daemons which provide a fake answer before even receiving a query.  Both of these behaviors are commonly associated with malware infections.</para>
          </listitem>
        </varlistentry>


        <varlistentry>
          <term id="nse-category-safe">
            <indexterm><primary sortas="safe script category">&ldquo;<literal>safe</literal>&rdquo; script category</primary></indexterm>
            <option>safe</option>
          </term>
          <listitem>
            <para>Scripts
	      which weren't designed to crash services, use large
	      amounts of network bandwidth or other resources, or
	      exploit security holes are categorized as <literal>safe</literal>. These are less likely to offend
	      remote administrators, though (as with all other Nmap
	      features) we cannot guarantee that they won't ever cause
	      adverse reactions.  Most of these perform general
              network discovery. Examples are
              <literal>ssh-hostkey</literal> (retrieves an SSH host key) and
              <literal>html-title</literal> (grabs the title from a
              web page).  Scripts in the <literal>version</literal> category are not categorized by safety, but any other scripts which aren't in <literal>safe</literal> should be placed in <literal>intrusive</literal>.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term id="nse-category-version">
            <indexterm><primary sortas="version script category">&ldquo;<literal>version</literal>&rdquo; script category</primary></indexterm>
            <indexterm><primary>version detection</primary><seealso>&ldquo;<literal>version</literal>&rdquo; script category</seealso></indexterm>
            <option>version</option>
          </term>
          <listitem>

            <para>The scripts in this special category are an
            extension to the version detection feature and cannot be
            selected explicitly. They are selected to run only if
            version detection (<option>-sV</option>) was requested.
            Their output cannot be distinguished from version
            detection output and they do not produce service or host
            script results.  Examples
            are <literal>skypev2-version</literal>, <literal>pptp-version</literal>,
            and <literal>iax2-version</literal>.</para>

          </listitem>
        </varlistentry>


        <varlistentry>
          <term id="nse-category-vuln">
            <indexterm><primary sortas="vuln script category">&ldquo;<literal>vuln</literal>&rdquo; script category</primary></indexterm>
            <option>vuln</option>
          </term>
          <listitem>
            <para>These scripts check for specific known vulnerabilities and
            generally only report results if they are found.  Examples include <literal>realvnc-auth-bypass</literal> and <literal>afp-path-vuln</literal>.</para>
          </listitem>
        </varlistentry>

    </variablelist>
    </sect2>

    <sect2 id="nse-script-types">
      <title>Script Types and Phases</title>
   <para>
   NSE supports four types of scripts, which are distinguished by the kind of targets they take and the scanning phase in which they are run.  Individual scripts may support multiple types of operation.
   </para>

    <variablelist>
      <varlistentry>
        <term>Prerule scripts</term>
        <listitem>
          <para>These scripts run before any of Nmap's scan phases, so
          Nmap has not collected any information about its targets
          yet.  They can be useful for tasks which don't depend on
          specific scan targets, such as performing network broadcast
          requests to query DHCP and DNS SD servers.  Some of these
          scripts can generate new targets for Nmap to scan (only if
          you specify
          the <ulink url="https://nmap.org/nsedoc/lib/target.html">newtargets</ulink>
          NSE argument).  For example, <ulink role="hidepdf"
          url="https://nmap.org/nsedoc/scripts/dns-zone-transfer.html">dns-zone-transfer</ulink>
          can obtain a list of IPs in a domain using a zone transfer
          request and then automatically add them to Nmap's scan
          target list.  Prerule scripts can be identified by containing a <literal>prerule</literal> function (see <xref linkend="nse-format-rules"/>).
          </para>
        </listitem>
      </varlistentry>


      <varlistentry>
        <term>Host scripts</term>
        <listitem>
          <para>Scripts in this phase run during Nmap's normal
          scanning process after Nmap has performed host discovery,
          port scanning, version detection, and OS detection against
          the target host.  This type of script is invoked once
          against each target host which matches
          its <literal>hostrule</literal> function.  Examples
          are <ulink role="hidepdf"
          url="https://nmap.org/nsedoc/scripts/whois-ip.html">whois-ip</ulink>,
          which looks up ownership information for a target IP,
          and <ulink role="hidepdf"
          url="https://nmap.org/nsedoc/scripts/path-mtu.html">path-mtu</ulink>
          which tries to determine the maximum IP packet size which
          can reach the target without requiring fragmentation.
          </para>
        </listitem>
      </varlistentry>

      <varlistentry>
        <term>Service scripts</term>
        <listitem>
          <para>These scripts run against specific services listening
          on a target host.  For example, Nmap includes more than 15
          http service scripts to run against web servers.  If a host
          has web servers running on multiple ports, those scripts may
          run multiple times (one for each port).  These are the most
          commong Nmap script type, and they are distinguished by
          containing a <literal>portrule</literal> function for
          deciding which detected services a script should run
          against.
          </para>
        </listitem>
      </varlistentry>

      <varlistentry>
        <term>Postrule scripts</term>
        <listitem>

          <para>These scripts run after Nmap has scanned all of its
          targets.  They can be useful for formatting and presenting
          Nmap output.  For example, <ulink role="hidepdf"
          url="https://nmap.org/nsedoc/scripts/ssh-hostkey.html">ssh-hostkey</ulink>
          is best known for its service (portrule) script which
          connects to SSH servers, discovers their public keys, and
          prints them.  But it also includes a postrule which checks
          for duplicate keys amongst all of the hosts scanned, then
          prints any that are found.  Another potential use for a
          postrule script is printing a reverse-index of the Nmap
          output&mdash;showing which hosts run a particular service
          rather than just listing the services on each host.
          Postrule scripts are identified by containing a
          <literal>postrule</literal> function.

          </para>

          <para>Many scripts could potentially run as either a prerule
          or postrule script.  In those cases, we recommend using a
          prerule for consistency.</para>

        </listitem>
      </varlistentry>
    </variablelist>
    </sect2>

    <sect2 id="nse-cmd-line-args">
      <title>Command-line Arguments</title>
      <para>
	These are the five command-line arguments specific to script scanning:
      </para>
      <variablelist>
        <varlistentry>
          <term>
            <indexterm><primary><option>-sC</option></primary></indexterm>
            <option>-sC</option>
          </term>
          <listitem>
          <para>Performs a script scan using the default set of scripts. It is 
		  equivalent to <option>--script=default</option>. Some of the
		  scripts in this <literal>default</literal> category are considered intrusive and should
		  not be run against a target network without permission. </para>
          </listitem>
        </varlistentry>

     <varlistentry>
        <term>
          <indexterm><primary><option>--script</option></primary></indexterm>
          <option>--script <replaceable>filename</replaceable>|<replaceable>category</replaceable>|<replaceable>directory</replaceable>/|<replaceable>expression</replaceable><optional>,...</optional></option></term>

<listitem>

<para>
Runs a script scan using the comma-separated list of filenames, script
categories, and directories. Each element in the list may also be a
Boolean expression describing a more complex set of scripts. Each
element is interpreted first as an expression, then as a category, and
finally as a file or directory name. The special argument
<literal>all</literal> makes every script in Nmap's script database
eligible to run.  The <literal>all</literal> argument should be used with caution as NSE may contain dangerous scripts including exploits, brute force authentication crackers, and denial of service attacks.
</para>

<para>
Each element in the script expression list may be prefixed with a
<literal>+</literal> character to force the given script(s) to run
regardless of the conditions in their <literal>portrule</literal> or
<literal>hostrule</literal> functions.  This is generally only done by
advanced users in special cases.  For example, you might want to do a
configuration review on a bunch of MS SQL servers, some of which are
running on nonstandard ports.  Rather than slow the Nmap scan by
running extensive version detection (<option>-sV
--version-all</option>) so that Nmap will recognize the <literal>ms-sql</literal>
service, you can force the <literal>ms-sql-config</literal> script to run against all the
targeted hosts and ports by specifying <option>--script
+ms-sql-config</option>.</para>

<para>
File and directory names may be relative or absolute. Absolute names are
used directly. Relative paths are searched for in the
<filename>scripts</filename> subdirectory of each of the following places until
found:
<indexterm><primary>data files</primary><secondary>directory search order</secondary></indexterm><indexterm><primary>scripts, location of</primary></indexterm>
<simplelist>
<member><option>--datadir</option></member>
<member><envar>$NMAPDIR</envar><indexterm><primary><envar>NMAPDIR</envar> environment variable</primary></indexterm></member>
<member><filename>~/.nmap</filename> (not searched on Windows)<indexterm><primary sortas="nmap directory"><filename>.nmap</filename> directory</primary></indexterm></member>
<member><filename><replaceable>APPDATA</replaceable>\nmap</filename> (only on Windows)<indexterm><primary sortas="nmap directory"><filename>.nmap</filename> directory</primary></indexterm></member>
<member>the directory containing the <filename>nmap</filename>
executable</member>
<member>the directory containing the <filename>nmap</filename>
executable, followed by <filename>../share/nmap</filename> (not searched on Windows)</member>
<member><varname>NMAPDATADIR</varname><indexterm><primary><varname>NMAPDATADIR</varname></primary></indexterm> (not searched on Windows)</member>
<member>the current directory.</member>
</simplelist>
</para>

<para>
When a directory name ending in <literal>/</literal> is given, Nmap loads every file in the directory
whose name ends with <filename>.nse</filename>. All other files are
ignored and directories are not searched recursively. When a filename is
given, it does not have to have the <filename>.nse</filename> extension;
it will be added automatically if necessary.
</para>

<para>
See <xref linkend="nse-script-selection"/> for examples and a full
explanation of the <option>--script</option> option.
</para>

<indexterm><primary>script database</primary><see><filename>script.db</filename></see></indexterm>
<para>Nmap scripts are stored in a <filename>scripts</filename>
subdirectory of the Nmap data directory by default (see
<xref linkend="data-files"/>). For efficiency, scripts are indexed in
a database stored
in <filename>scripts/script.db</filename>,<indexterm><primary><filename>script.db</filename></primary></indexterm>
which lists the category or categories in which each script belongs.
The argument <literal>all</literal> will execute all scripts in the
Nmap script database, but should be used cautiously since Nmap may contain exploits, denial of service attacks, and other dangerous scripts.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term>
            <indexterm><primary><option>--script-args</option></primary></indexterm>
            <option>--script-args <replaceable>args</replaceable></option>
          </term>
          <listitem>
            <para>Provides arguments to the scripts. See
            <xref linkend="nse-args"/> for a detailed explanation.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term>
            <indexterm><primary><option>--script-args-file</option></primary></indexterm>
            <option>--script-args-file <replaceable>filename</replaceable></option>
          </term>
          <listitem>
            <para>This option is the same as
            <option>--script-args</option> except that you pass the
            arguments in a file rather than on the command-line.  See
            <xref linkend="nse-args"/> for a detailed
            explanation.</para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term>
            <indexterm><primary><option>--script-help</option></primary></indexterm>
            <option>--script-help <replaceable>filename</replaceable>|<replaceable>category</replaceable>|<replaceable>directory</replaceable>|<replaceable>expression</replaceable>|all<optional>,...</optional></option>
          </term>
          <listitem>
            <para>
            Shows help about scripts. For each script matching the given
            specification, Nmap prints the script name, its categories, and its
            description. The specifications are the same as those accepted by
            <option>--script</option>; so for example if you want help about
            the <literal>ssl-enum-ciphers</literal> script, you would run
            <command>nmap --script-help ssl-enum-ciphers</command>. A sample of script
            help is shown in <xref linkend="nse-script-help"/>.
            </para>
            <example id="nse-script-help">
              <indexterm><primary><option>--script-help</option></primary><secondary>example of</secondary></indexterm>
              <title>Script help</title>
<screen>
$ nmap --script-help "afp-* and discovery"

Starting Nmap 7.40 ( https://nmap.org ) at 2017-04-21 14:15 UTC

afp-ls
Categories: discovery safe
https://nmap.org/nsedoc/scripts/afp-ls.html
  Attempts to get useful information about files from AFP volumes.
  The output is intended to resemble the output of <code>ls</code>.

afp-serverinfo
Categories: default discovery safe
https://nmap.org/nsedoc/scripts/afp-serverinfo.html
  Shows AFP server information. This information includes the server's
  hostname, IPv4 and IPv6 addresses, and hardware type (for example
  <code>Macmini</code> or <code>MacBookPro</code>).

afp-showmount
Categories: discovery safe
https://nmap.org/nsedoc/scripts/afp-showmount.html
  Shows AFP shares and ACLs.
</screen>
            </example>
            <para>
            If the
            <option>-oX</option><indexterm><primary><option>-oX</option></primary></indexterm>
            option is used, an XML representation of the script help will be
            written to the given file.
            </para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term>
            <indexterm><primary><option>--script-trace</option></primary></indexterm>
            <option>--script-trace</option>
          </term>
          <listitem>
            <para>

	      This option is similar to
              <option>--packet-trace</option>, but works at the
              application level rather than packet by packet. If this
              option is specified, all incoming and outgoing
              communication performed by scripts is printed. The
              displayed information includes the communication
              protocol, source and target addresses, and the
              transmitted data. If more than 5% of transmitted data is
              unprintable, hex dumps are given instead.
              Specifying <option>--packet-trace</option> enables script
              tracing too.

	    </para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term>
            <indexterm><primary><option>--script-updatedb</option></primary></indexterm>
            <option>--script-updatedb</option>
          </term>
          <listitem>

            <para>This option updates the script database found
            in <filename>scripts/script.db</filename> which is used by
            Nmap to determine the available default scripts and
            categories. It is only necessary to update the database if
            you have added or removed NSE scripts from the
            default <filename>scripts</filename> directory or if you
            have changed the categories of any script.  This option is
            used by
            itself without arguments: <command>nmap --script-updatedb</command>.</para>

          </listitem>
        </varlistentry>
      </variablelist>

      <para>
        Some other Nmap options have effects on script scans. The most
        prominent of these is
	<option>-sV</option>.<indexterm><primary><option>-sV</option></primary></indexterm>
        A version scan automatically executes
        the scripts in the
	<literal>version</literal> category.<indexterm><primary sortas="version script category">&ldquo;<literal>version</literal>&rdquo; script category</primary></indexterm>
        The scripts
        in this category are slightly different from other scripts because their
        output blends in with the version scan results and they do not produce any
        script scan output to the screen.  If the
        <option>-oX</option><indexterm><primary><option>-oX</option></primary></indexterm>
        option is used, typical script output will still be available in the
        XML output file.
      </para>
      <para>
        Another option which affects the scripting engine is
	<option>-A</option>.<indexterm><primary><option>-A</option></primary><secondary>features enabled by</secondary></indexterm>
        The aggressive Nmap mode implies
	the <option>-sC</option> option.
      </para>

      <para>
      </para>
    </sect2>

    <sect2 id="nse-script-selection">
      <title>Script Selection</title>
      <indexterm><primary><option>--script</option></primary></indexterm>
      <indexterm><primary>script selection</primary></indexterm>

      <para>
      The <option>--script</option> option takes a comma-separated list
      of categories, filenames, and directory names. Some simple
      examples of its use:
      </para>

      <variablelist>
        <varlistentry>
          <term><command>nmap --script default,safe</command></term>
          <listitem>
          <para>Loads all scripts in the <literal>default</literal> and
          <literal>safe</literal> categories.</para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term><command>nmap --script smb-os-discovery</command></term>
          <listitem>
          <para>Loads only the <filename>smb-os-discovery</filename>
          script. Note that the <filename>.nse</filename> extension is
          optional.</para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term><command>nmap --script default,banner,/home/user/customscripts</command></term>
          <listitem>
          <para>Loads the script in the <literal>default</literal>
          category, the <filename>banner</filename> script, and all
          <filename>.nse</filename> files in the directory
          <filename>/home/user/customscripts</filename>.</para>
          </listitem>
        </varlistentry>
      </variablelist>

      <indexterm><primary>wildcards</primary><secondary>in script selection</secondary></indexterm>
      <para>
      When referring to scripts from <filename>script.db</filename> by
      name, you can use a shell-style &lsquo;<literal>*</literal>&rsquo;
      wildcard.
      </para>

      <variablelist>
        <varlistentry>
          <term><command>nmap --script "http-*"</command></term>
          <listitem>
          <para>Loads all scripts whose name starts with
          <filename>http-</filename>, such as
          <filename>http-auth</filename> and
          <filename>http-open-proxy</filename>. The argument to
          <option>--script</option> had to be in quotes to protect the
          wildcard from the shell.</para>
          </listitem>
        </varlistentry>
      </variablelist>

      <indexterm><primary>Boolean expressions in script selection</primary></indexterm>
      <para>
      More complicated script selection can be done using the
      <literal>and</literal>, <literal>or</literal>, and
      <literal>not</literal> operators to build Boolean expressions. The
      operators have the same
      <ulink role="hidepdf" url="https://lua.org/manual/5.4/manual.html#3.4.8">precedence</ulink>
      as in Lua: <literal>not</literal> is the highest, followed by
      <literal>and</literal> and then <literal>or</literal>. You can
      alter precedence by using parentheses. Because expressions contain
      space characters it is necessary to quote
      them.
      </para>

      <variablelist>
        <varlistentry>
          <term><command>nmap --script "not intrusive"</command></term>
          <listitem>
          <para>Loads every script except for those in the
          <literal>intrusive</literal> category.</para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term><command>nmap --script "default or safe"</command></term>
          <listitem>
          <para>This is functionally equivalent to
          <command>nmap --script "default,safe"</command>. It loads all
          scripts that are in the <literal>default</literal> category or
          the <literal>safe</literal> category or both.</para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term><command>nmap --script "default and safe"</command></term>
          <listitem>
          <para>Loads those scripts that are in
          <emphasis>both</emphasis> the <literal>default</literal> and
          <literal>safe</literal> categories.</para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term><command>nmap --script "(default or safe or intrusive) and not http-*"</command></term>
          <listitem>
          <para>Loads scripts in the <literal>default</literal>,
          <literal>safe</literal>, or <literal>intrusive</literal>
          categories, except for those whose names start with
          <filename>http-</filename>.</para>
          </listitem>
        </varlistentry>
      </variablelist>

      <para>
      Names in a Boolean expression may be a category, a filename from
      <filename>script.db</filename>, or <literal>all</literal>. A name
      is any sequence of characters not containing
      &lsquo;<literal>&nbsp;</literal>&rsquo;,
      &lsquo;<literal>,</literal>&rsquo;,
      &lsquo;<literal>(</literal>&rsquo;,
      &lsquo;<literal>)</literal>&rsquo;, or
      &lsquo;<literal>;</literal>&rsquo;, except for the sequences
      <literal>and</literal>, <literal>or</literal>, and
      <literal>not</literal>, which are operators.
      </para>
    </sect2>

	<sect2 id="nse-args">
      <title>Arguments to Scripts</title>
      <indexterm><primary>script arguments</primary></indexterm>
      <para>
      Arguments may be passed to NSE scripts using the
      <option>--script-args</option> option. The arguments describe a table of
      key-value pairs and possibly array values. The arguments are provided to
      scripts as a table in the registry called
      <varname>nmap.registry.args</varname>, though they are normally accessed through the <literal>stdnse.get_script_args</literal> function.
      </para>

      <para>
      The syntax for script arguments is similar to Lua's table constructor
      syntax. Arguments are a comma-separated list of
      <literal>name=value</literal> pairs. Names and values may be strings not
      containing whitespace or the characters
      &lsquo;<literal>{</literal>&rsquo;,
      &lsquo;<literal>}</literal>&rsquo;,
      &lsquo;<literal>=</literal>&rsquo;, or
      &lsquo;<literal>,</literal>&rsquo;.
      To include one of these characters in a string, enclose the string in
      single or double quotes. Within a quoted string,
      &lsquo;<literal>\</literal>&rsquo; escapes a quote. A backslash is only
      used to escape quotation marks in this special case; in all other cases a
      backslash is interpreted literally.
      </para>
      <para>
      Values may also be tables enclosed in <literal>{}</literal>, just as in
      Lua. A table may contain simple string values, for example a list of proxy
      hosts; or more name-value pairs, including nested tables.
      </para>

      <para>Script arguments are often qualified with the relevant
      script name so that a user doesn't unintentionally affect multiple
      scripts with a single generic name.  For example, you can set
      the timeout for responses to the
      <literal>broadcast-ping</literal> script (and only that script)
      by setting <literal>broadcast-ping.timeout</literal> to the
      amount of time you're willing to wait.  Sometimes,
      however, you want a script argument applied more widely.  If you
      remove the qualification and specify just
      <literal>timeout=250ms</literal>, you will be setting the value
      for more than a dozen scripts in addition to
      <literal>broadcast-ping</literal>.  You can even combine
      qualified and unqualified arguments, and the most specific match
      takes precedence.  For example, you could specify
      <literal>rlogin-brute.timeout=20s,timeout=250ms</literal>.  In
      that case, the timeout will be 20 seconds for the
      <literal>rlogin-brute</literal> script, and 250 milliseconds for all other
      scripts which support this variable
      (<literal>broadcast-ping</literal>,
      <literal>lltd-discovery</literal>, etc.)</para>

      <para>Rather than pass the arguments on the command line with
      <option>--script-args</option>, you may store them in a file
      (separated by commas or newlines) and specify just the file name
      with <option>--script-args-file</option>.  Options specified
      with <option>--script-args</option> on the command-line take
      precedence over those given in a file.  The filename may be
      given as an absolute path or relative to Nmap's usual
      search path (NMAPDIR, etc.)
      </para>

      <para>Here is a typical Nmap invocation with script arguments:
<informalexample>
<indexterm><primary><option>--script-args</option></primary><secondary>example of</secondary></indexterm>
<literallayout>
  <command>nmap -sC --script-args 'user=foo,pass=",{}=bar",paths={/admin,/cgi-bin},xmpp-info.server_name=localhost'</command>
</literallayout>
</informalexample>
      Notice that the script arguments are surrounded in single quotes. For the
      Bash shell, this prevents the shell from interpreting the double quotes
      and doing automatic string concatenation. Naturally, different shells may
      require you to escape quotes or to use different quotes.  See your
      relevant manual. The command results in this Lua table:
<programlisting>
nmap.registry.args = {
  user = "foo",
  pass = ",{}=bar",
  paths = {
    "/admin",
    "/cgi-bin"
  },
  xmpp-info.server_name="localhost"
}
</programlisting>
      While you could access the values directly from <literal>nmap.registry.args</literal>, it is normally better to use the <literal>stdnse.get_script_args</literal> function like this:
<programlisting>
local server_name = stdnse.get_script_args("xmpp-info.server_name")
</programlisting>
      </para>
      <para>
      All script arguments share a global namespace, the
      <literal>nmap.registry.args</literal> table. For this reason, short or
      ambiguous names like <literal>user</literal> are not recommended. Some
      scripts prefix their arguments with their script name, like
      <literal>smtp-open-relay.domain</literal>.
      Arguments used by libraries, which can
      affect many scripts, usually have names beginning with the name of the
      library, like <literal>smbuser</literal> and
      <literal>creds.snmp</literal>.
      </para>
      <para>
      The online NSE Documentation Portal at <ulink
      url="https://nmap.org/nsedoc/"/> lists the arguments that each script
      accepts, including any library arguments that may influence the script.
      </para>
    </sect2>


    <sect2 id="nse-usage-examples">
      <title>Complete Examples</title>

      <variablelist>
        <varlistentry>
          <term><command>nmap -sC example.com</command></term>
          <listitem>
          <para>A simple script scan using the default set of
          scripts.</para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term><command>nmap -sn -sC example.com</command></term>
          <listitem>
          <para>A script scan without a port scan; only host scripts are
          eligible to run.</para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term><command>nmap -Pn -sn -sC example.com</command></term>
          <listitem>
          <para>A script scan without host discovery or a port scan. All
          hosts are assumed up and only host scripts are eligible to
          run.</para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term>
          <indexterm><primary><option>--script-trace</option></primary><secondary>example of</secondary></indexterm>
          <command>nmap --script smb-os-discovery --script-trace example.com</command></term>
          <listitem>
          <para>Execute a specific script with script tracing.</para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term>
          <indexterm><primary><option>--script-args</option></primary><secondary>example of</secondary></indexterm>
          <command>nmap --script snmp-sysdescr --script-args creds.snmp=admin example.com</command></term>
          <listitem>
          <para>Run an individual script that takes a script
          argument.</para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term><command>nmap --script mycustomscripts,safe example.com</command></term>
          <listitem>
          <para>Execute all scripts in the
          <filename>mycustomscripts</filename> directory as well as all
          scripts in the <literal>safe</literal> category.</para>
          </listitem>
        </varlistentry>
      </variablelist>
    </sect2>
  </sect1>

  <sect1 id="nse-script-format">
    <title>Script Format</title>
    <para>NSE scripts consist of a handful of descriptive fields, a rule defining when the script should be executed, and an <literal>action</literal> function containing the actual script instructions. Values can be assigned to the descriptive fields just as you would assign any other Lua variables.  Their names must be lowercase as shown in this section.</para>

    <sect2 id="nse-format-description">
      <title><literal>description</literal> Field</title>
      <indexterm><primary sortas="description script variable">&ldquo;<varname>description</varname>&rdquo; script variable</primary></indexterm>
      <para>The <literal>description</literal> field describes what a script is testing
      for and any important notes the user should be aware of. Depending on script complexity, descriptions may vary in length from a few sentences to a few paragraphs.  The first paragraph should be a brief synopsis of the script function suitable for stand-alone presentation to the user.  Further paragraphs may provide much more script detail.
      </para>
    </sect2>

    <sect2 id="nse-format-categories">
      <title><literal>categories</literal> Field</title>
      <indexterm><primary sortas="categories script variable">&ldquo;<varname>categories</varname>&rdquo; script variable</primary></indexterm>

      <para>The <literal>categories</literal> field defines one or
      more categories to which a script belongs (see
      <xref linkend="nse-categories"/>).  The categories are case-insensitive and may be specified in any order.  They are listed in an array-style Lua table as in this example:</para>

<programlisting>
categories = {"default", "discovery", "safe"}
</programlisting>

    </sect2>

    <sect2 id="nse-format-author">
      <title><literal>author</literal> Field </title>
      <indexterm><primary sortas="author script variable">&ldquo;<varname>author</varname>&rdquo; script variable</primary></indexterm>
      <para>
	The <literal>author</literal> field contains the script authors' names and can also contain contact information (such as home page URLs).  We no longer recommend including email addresses because spammers might scrape them from the NSEDoc web site.  This optional field is not used by NSE, but gives script authors their due credit or blame. 
      </para>
    </sect2>

    <sect2 id="nse-format-license">
      <title><literal>license</literal> Field </title>
      <indexterm><primary sortas="license script variable">&ldquo;<varname>license</varname>&rdquo; script variable</primary></indexterm>
      <indexterm><primary>copyright</primary><secondary>of scripts</secondary></indexterm>

      <para>Nmap is a community project and we welcome all sorts of
      code contributions, including NSE scripts.  So if you write a
      valuable script, don't keep it to yourself!
      The optional <literal>license</literal> field helps ensure that we have
      legal permission to distribute all the scripts which come with Nmap. All of those scripts
      currently use the standard Nmap license
      (described in <xref linkend="nmap-copyright"/>).  They include
      the following line:</para>

<programlisting>
license = "Same as Nmap--See https://nmap.org/book/man-legal.html"
</programlisting>

<para>The Nmap license is similar to the GNU GPL.  Script authors may
use a BSD-style license (no advertising clause) instead if they prefer
that.  For a BSD-style license, please include this line:

<programlisting>
license = "Simplified (2-clause) BSD license--See https://nmap.org/svn/docs/licenses/BSD-simplified"
</programlisting>

</para>

    </sect2>

    <sect2 id="nse-format-dependencies">
      <title><literal>dependencies</literal> Field</title>
      <indexterm><primary sortas="dependencies script variable">&ldquo;<varname>dependencies</varname>&rdquo; script variable</primary></indexterm>
      <indexterm><primary>script dependencies</primary></indexterm>
       <para>
        The <literal>dependencies</literal> field is an array containing the
        names of scripts that should run before this script, if they are also selected. This is used when
        one script can make use of the results of another. For example, most of
        the <filename>smb-*</filename> scripts depend on
        <filename>smb-brute</filename>,<indexterm><primary><filename>smb-brute</filename> script</primary></indexterm>
        because the accounts found by <filename>smb-brute</filename> may allow
        the other scripts to get more information. Listing a script in
        <literal>dependencies</literal> doesn't cause that script to be run; it
        still has to be selected through the <option>--script</option> option
        or otherwise. <literal>dependencies</literal> merely forces an ordering
        among the scripts that <emphasis>are</emphasis> selected. This is an
        example of a <literal>dependencies</literal> table, from
        <filename>smb-os-discovery</filename>:<indexterm><primary><filename>smb-os-discovery</filename> script</primary></indexterm>
<programlisting>
dependencies = {"smb-brute"}
</programlisting>
         The dependencies table is optional. NSE will assume
         the script has no dependencies if the field is omitted.
       </para>
       <para>
         Dependencies establish an internal ordering of scripts, assigning each
         one a number called a <quote>runlevel</quote><footnote><para>Up through
         Nmap version 5.10BETA2, dependencies didn't exist and script authors
         had to set a <varname>runlevel</varname> field manually.</para></footnote>.<indexterm><primary>runlevel</primary></indexterm>
         When
         running your scripts you will see the runlevel (along with the total number of
         runlevels) of each grouping of scripts run in NSE's output:
<screen>
NSE: Script scanning 127.0.0.1.
NSE: Starting runlevel 1 (of 3) scan.
Initiating NSE at 17:38
Completed NSE at 17:38, 0.00s elapsed
NSE: Starting runlevel 2 (of 3) scan.
Initiating NSE at 17:38
Completed NSE at 17:38, 0.00s elapsed
NSE: Starting runlevel 3 (of 3) scan.
Initiating NSE at 17:38
Completed NSE at 17:38, 0.00s elapsed
NSE: Script Scanning completed.
</screen>
       </para>
    </sect2>

    <sect2 id="nse-format-rules">
      <title>Rules</title>
      <indexterm><primary sortas="prerule script variable">&ldquo;<varname>prerule</varname>&rdquo; script variable</primary></indexterm>
      <indexterm><primary sortas="portrule script variable">&ldquo;<varname>portrule</varname>&rdquo; script variable</primary></indexterm>
      <indexterm><primary sortas="hostrule script variable">&ldquo;<varname>hostrule</varname>&rdquo; script variable</primary></indexterm>
      <indexterm><primary sortas="postrule script variable">&ldquo;<varname>postrule</varname>&rdquo; script variable</primary></indexterm>
      <indexterm><primary>rules in NSE</primary><see>&ldquo;<varname>prerule</varname>&rdquo;, &ldquo;<varname>portrule</varname>&rdquo;, &ldquo;<varname>hostrule</varname>&rdquo; and &ldquo;<varname>postrule</varname>&rdquo;</see></indexterm>
      <para>
      Nmap uses the script rules to determine whether a script should be
      run against a target. A rule is a Lua function that returns either
      <literal>true</literal> or <literal>false</literal>. The script
      <literal>action</literal> function is only performed if the rule
      evaluates to <literal>true</literal>.
      </para>
      <para>
      A script must contain one or more of the following functions that
      determine when the script will be run:
      <simplelist>
      <member><literal>prerule()</literal></member>
      <member><literal>hostrule(host)</literal></member>
      <member><literal>portrule(host, port)</literal></member>
      <member><literal>postrule()</literal></member>
      </simplelist>
      <literal>prerule</literal> scripts run once, before any hosts are
      scanned, during the script pre-scanning
      phase.<indexterm><primary>script
      pre-scanning</primary></indexterm> <literal>hostrule</literal> and
      <literal>portrule</literal> scripts run after each batch of hosts
      is scanned. <literal>postrule</literal> scripts run once after all
      hosts have been scanned, in the script post-scanning
      phase.<indexterm><primary>script post-scanning
      phase</primary></indexterm> A script may run in more than one
      phase if it has several rules.
      </para>
      <para>
      <literal>prerule</literal> and <literal>postrule</literal> do not
      accept arguments. <literal>hostrule</literal> accepts a host table
      and may test, for example, the IP address or hostname of the
      target. <literal>portrule</literal> accepts both a host table and
      a port table for any port in the
      <literal>open</literal><indexterm><primary><literal>open</literal> port state</primary></indexterm>, 
      <literal>open|filtered</literal><indexterm><primary><literal>open|filtered</literal> port state</primary></indexterm>,
      or <literal>unfiltered</literal><indexterm><primary><literal>unfiltered</literal> port state</primary></indexterm>
      port states.  Port rules generally test factors such as the port
      number, port state, or listening service name in deciding whether
      to run against a port.  Example rules are shown in <xref
      linkend="nse-tutorial-rule"/>.  
      </para>

      <para>Advanced users may force a script to run regardless of the
      results of these rule functions by prefixing the script name (or
      category or other expression) with a <literal>+</literal> in the
      <option>--script</option> argument.</para>

      <para>
      The current standard to choose between a
      <literal>prerule</literal> or a <literal>postrule</literal> is
      this: if the script is doing host discovery or any other network
      operation then the <literal>prerule</literal> should be used.
      <literal>postrule</literal> is reserved for reporting of data and
      statistics that were gathered during the scan.
      </para> 
    </sect2>

     <sect2 id="nse-format-action"><title>Action</title>
      <indexterm><primary sortas="action script variable">&ldquo;<varname>action</varname>&rdquo; script variable</primary></indexterm>

      <para>
The action is the heart of an NSE script. It contains all of the
instructions to be executed when the script's prerule, portrule, hostrule or postrule
triggers. It is a Lua function which accepts the same arguments as the
rule. The return value of the action value may be a table of
name&ndash;value pairs, a string, or <code>nil</code>. For an example of
an NSE action refer to <xref linkend="nse-tutorial-action"/>.
      </para>
      <para>
If the output of the action is a table, it is automatically formatted in
a structured fashion for inclusion in the normal (<option>-oN</option>)
and XML (<option>-oX</option>) output formats. If a string, the text is
displayed directly in normal output, and written as an XML attribute in
XML output, No output is produced if the script returns
<literal>nil</literal>. See <xref linkend="nse-structured-output"/> for
details of how different return values are handled.
      </para>
    </sect2>

     <sect2 id="nse-format-environment"><title>Environment Variables</title>
      <indexterm><primary sortas="environment script variable">&ldquo;<varname>environment</varname>&rdquo; script variable</primary></indexterm>

      <para>Each script has its own set of environment variables:</para>
      <variablelist>
        <varlistentry>
          <term><literal>SCRIPT_PATH</literal></term>
          <listitem>
            <para>
              The script path.
            </para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><literal>SCRIPT_NAME</literal></term>
          <listitem>
            <para>
              The script name. This variable can be used in debug output.
            </para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><literal>SCRIPT_TYPE</literal></term>
          <listitem>
            <para>
              Since a script can have multiple rule functions, this
              environment variable will show which rule has activated
              the script, this would be useful if the script wants to
              share some code between different Script Scan phases.
              It will take one of these four string values:
              <literal>"prerule"</literal>, <literal>"hostrule"</literal>, 
              <literal>"portrule"</literal> or 
              <literal>"postrule"</literal>.
              This variable is only available during and after the evaluation
              of the rule functions.
            </para>
          </listitem>
        </varlistentry>
      </variablelist>
      <para>
        This is an example of a debug code that uses the previous
        environment variables, followed by the output message, from dns-zone-transfer:
        <programlisting>
          stdnse.print_debug(3, "Skipping '%s' %s, 'dnszonetransfer.server' argument is missing.", SCRIPT_NAME, SCRIPT_TYPE)
        </programlisting>
        <screen>
          Initiating NSE at 15:31
          NSE: Skipping 'dns-zone-transfer' prerule, 'dnszonetransfer.server' argument is missing.
        </screen>
      </para>
    </sect2>

  </sect1>
  <sect1 id="nse-language">
    <title>Script Language</title>
    <indexterm><primary>Nmap Scripting Engine (NSE)</primary><secondary>parts of</secondary></indexterm>

    <para>
      The core of the Nmap Scripting Engine is an embeddable Lua
      interpreter. Lua is a lightweight language designed for
      extensibility.  It offers a powerful and well-documented API for
      interfacing with other software such as Nmap.
    </para>
    <indexterm><primary>Nmap Scripting Engine (NSE)</primary><secondary>library</secondary></indexterm>
    <para>
      The second part of the Nmap Scripting Engine is the NSE Library, which
      connects Lua and Nmap. This layer
      handles issues such as initialization of the Lua interpreter, 
      scheduling of parallel script execution, script retrieval and
      more. It is also the heart of the NSE network I/O framework and the
	  exception handling mechanism.  It also includes utility libraries to make scripts more powerful and convenient.  The utility library modules and extensions are described in <xref linkend="nse-library"/>.</para>

    <sect2 id="nse-lua">
    <title>Lua Base Language</title>
      <indexterm><primary>Lua programming language</primary></indexterm>
      <para>

	The Nmap scripting language is an embedded <ulink
	url="https://lua.org/">Lua</ulink> interpreter which is
	extended with libraries for interfacing with Nmap.  The Nmap
	API is in the Lua namespace <literal>nmap</literal>.  This
	means that all calls to resources provided by Nmap have an
	<literal>nmap</literal> prefix.<indexterm><primary><varname>nmap</varname> NSE library</primary></indexterm>
	<literal>nmap.new_socket()</literal>, for example, returns a
	new socket wrapper object. The Nmap library layer also takes
	care of initializing the Lua context, scheduling parallel
	scripts and collecting the output produced by completed
	scripts.  
	</para>
      <para>

	During the planning stages, we considered several programming
	languages as the base for Nmap scripting. Another option was to
	implement a completely new programming language. Our criteria
	were strict: NSE had to be easy to
	use, small in size, compatible with the Nmap license,
	scalable, fast and parallelizable.  Several
	previous efforts (by other projects) to design their own security auditing language from scratch
	resulted in awkward solutions, so we decided early not to follow that
	route. First the Guile Scheme interpreter was considered,
	but the preference drifted towards the Elk interpreter due to its more
	favorable license. But parallelizing Elk scripts would have been
	difficult. In addition, we expect that most Nmap users prefer procedural programming over functional languages such as Scheme.  Larger interpreters such as Perl, Python, and
	Ruby are well-known and loved, but are difficult to embed
	efficiently.  In the end, Lua excelled in all of our criteria.
	It is small, distributed under the liberal MIT open source license, has
	coroutines for efficient parallel script
	execution, was designed with embeddability in mind, has
	excellent documentation, and is actively developed by a large
	and committed community.
	Lua is now even embedded in other popular open source security tools including
	the <application>Wireshark</application> sniffer and <application>Snort</application> IDS.

      </para>
    </sect2>
	</sect1>

  <sect1 id="nse-scripts">
    <indexterm><primary>Nmap Scripting Engine (NSE)</primary><secondary>list of scripts</secondary></indexterm>
    <title>NSE Scripts</title>

    <print>
    <para>This section lists (alphabetically) all NSE scripts packaged
    with Nmap at the time of this writing.  It comes straight from the
    script source code thanks to the NSEDoc documentation system
    described in <xref linkend="nsedoc"/>. Because of space limitations,
    only script names, categories, and brief summaries of operation are
    shown.  Of course no paper documentation can stay current with
    software developed as actively as NSE is.  For complete and
    up-to-date documentation, including script arguments and output
    samples, see the online NSE Documentation Portal at
    <ulink url="https://nmap.org/nsedoc/"/>.
    </para>

    &nse-scripts;
    </print>

    <web>
    <para>This section (a long list of NSE scripts with brief
    summaries) is only provided in the printed edition of this book
    because we already provide a better online interface to the
    information at the <ulink url="https://nmap.org/nsedoc/">NSE
    Documentation Portal</ulink>.</para>
    </web>
  </sect1>

    <sect1 id="nse-library">
      <indexterm><primary>Nmap Scripting Engine (NSE)</primary><secondary>list of modules</secondary></indexterm>
      <title>NSE Libraries</title>
      <para>In addition to the significant built-in capabilities of
      Lua, we have written or integrated many extension libraries which make
      script writing more powerful and convenient. These libraries (sometimes called modules) are compiled if necessary and installed along with 
	  Nmap. They have their own directory, <filename>nselib</filename>, which
	  is installed in the configured Nmap data directory. Scripts need only 
      <ulink url="https://lua.org/manual/5.4/manual.html#pdf-require"><literal>require</literal></ulink> the default libraries in order to use them.
	  </para>

      <sect2 id="nse-library-list">
      <title>List of All Libraries</title>
      <para>
      This list is just an overview to give an idea of what libraries
      are available. Developers will want to consult the complete
      documentation at <ulink url="https://nmap.org/nsedoc/"/>.
      </para>

      &nse-modules;
      </sect2>

      <sect2 id="hacking-nse-libraries">
        <title>Hacking NSE Libraries</title>
        <para>
          A common mistake when editing libraries is to accidentally use a
          global variable instead of a local one. Different libraries using the
          same global variable can be the cause of mysterious bugs. Lua's scope
          assignment is global by default, so this mistake is easy to make.
        </para>
        <para>
          To help correct this problem, NSE uses a library adapted from
          the standard Lua distribution called
          <filename>strict.lua</filename>.<indexterm><primary><filename>strict</filename> NSE library</primary></indexterm>
          The library will
          raise a runtime error on any access or modification of a global
          variable which was undeclared in the file scope. A global variable is
          considered declared if the library makes an assignment to the global
          name (even <literal>nil</literal>) in the file scope.
        </para>
      </sect2>

    <sect2 id="nse-library-c-modules">
      <title>Adding C Modules to Nselib</title>
      <indexterm><primary>Nmap Scripting Engine (NSE)</primary><secondary>C modules</secondary></indexterm>

      <para>
      A few of the modules included in nselib are written in C or C++
      rather than Lua.  Two examples are <literal>bit</literal>
      and <literal>pcre</literal>.  We recommend that modules
      be written in Lua if possible, but C and C++ may be more
      appropriate if performance is critical or (as with
      the <literal>pcre</literal> and <literal>openssl</literal>
      modules) you are linking to an existing C library.  This section
      describes how to write your own compiled extensions to nselib.
      </para>

      <para>
      The Lua C API is described at length in 
      <web><ulink url="http://www.amazon.com/dp/8590379825?tag=secbks-20"><citetitle>Programming in Lua, Second Edition</citetitle></ulink>,</web>
      <print><citetitle>Programming in Lua, Second Edition</citetitle>,</print>
      so this is a short summary. C modules consist of functions that
      follow the protocol of the
      <ulink url="https://lua.org/manual/5.4/manual.html#lua_CFunction"><type>lua_CFunction</type></ulink>
      type. The functions are registered with Lua and assembled into a
      library by calling the
      <function>luaL_newlib</function><indexterm><primary><function>luaL_newlib</function></primary></indexterm>
      function. A special initialization function provides the interface
      between the module and the rest of the NSE code. By convention the
      initialization function is named in the form
      <function>luaopen_<replaceable>module</replaceable></function>.
      </para>

      <para>
      The most straightforward compiled module that comes with NSE is
      <literal>openssl</literal>.<indexterm><primary><varname>openssl</varname> NSE library</primary></indexterm>
      This module serves as a good example for a beginning module
      writer. The
      source code for
      <literal>openssl</literal> source is in <filename>nse_openssl.cc</filename> and
      <filename>nse_openssl.h</filename>. Most of the other compiled modules
      follow this <literal>nse_<replaceable>module name</replaceable>.cc</literal> naming convention.
      </para>
      <para>
      Reviewing the <literal>openssl</literal> module shows that one of the
      functions in <filename>nse_openssl.cc</filename> is
      <function>l_md5</function>, which calculates an MD5 digest. Its
      function prototype is:</para>
<programlisting>
static int l_md5(lua_State *L);
</programlisting>
      <para>The prototype shows that <function>l_md5</function> matches the
      <type>lua_CFunction</type> type. The function is static because it
      does not have to be visible to other compiled code.  Only an address is required
      to register it with Lua. Later in the file,
      <function>l_md5</function> is entered into an array of type
      <type>luaL_Reg</type> and associated with the name
      <function>md5</function>:</para>
<programlisting>
static const struct luaL_Reg openssllib[] = {
  { "md5", l_md5 },
  { NULL, NULL }
};
</programlisting>

 <para>This function will now be known as <function>md5</function> to NSE.  Next the library is registered with a call to
      <function>luaL_newlib</function> inside the initialization
      function <function>luaopen_openssl</function>, as shown
      next. Some lines relating to the registration of
      OpenSSL <type>BIGNUM</type> types have been omitted:</para>

<programlisting>
LUALIB_API int luaopen_openssl(lua_State *L) {
  luaL_newlib(L, openssllib);
  return 1;
}
</programlisting>

<para>The function <function>luaopen_openssl</function>
      is the only function in the file that is exposed in
      <filename>nse_openssl.h</filename>.  <varname>OPENSSLLIBNAME</varname> is simply the string
      <literal>"openssl"</literal>.
      </para>

      <para>
      After a compiled module is written, it must be added to NSE by including
      it in the list of standard libraries in
      <filename>nse_main.cc</filename>.  Then the module's
      source file names must be added to
      <filename>Makefile.in</filename> in the appropriate places. For both these tasks you can
      simply follow the example of the other C modules. For the
      Windows build, the new source files must be added to the
      <filename>mswin32/nmap.vcproj</filename> project file using MS Visual Studio (see <xref linkend="inst-win-source"/>).
      </para>
    </sect2>

  </sect1>


  <sect1 id="nse-api">
    <title>Nmap API</title>
    <indexterm class="startofrange" id="nse-nmap-indexterm"><primary><varname>nmap</varname> NSE library</primary></indexterm>
    <indexterm><primary>Nmap Scripting Engine (NSE)</primary><secondary>API</secondary></indexterm>
    <para>
      NSE scripts have access to several Nmap facilities for writing
      flexible and elegant scripts. The API provides target host
      details such as port states and version detection results.  It
      also offers an interface to the Nsock<indexterm><primary>Nsock</primary><secondary>in NSE</secondary></indexterm>
      library
      for efficient network I/O.
    </para>

    <sect2 id="nse-api-arguments">
      <title>Information Passed to a Script</title>
      <para>
	An effective Nmap scripting engine requires more than just a
	Lua interpreter. Users need easy access to the information
	Nmap has learned about the target hosts. This data is passed
	as arguments to the NSE script's
	<literal>action</literal> method.<indexterm><primary sortas="action script variable">&ldquo;<varname>action</varname>&rdquo; script variable</primary></indexterm>
        The arguments, <literal>host</literal> and
	<literal>port</literal>, are Lua tables which contain
	information on the target against which the script is
	executed. If a script matched a hostrule, it gets only the
	<literal>host</literal> table, and if it matched a portrule it
	gets both <literal>host</literal> and <literal>port</literal>.
	The following list describes each variable in these two tables.
      </para>

      <para>
	<variablelist>
	  <varlistentry>
	    <term><option>host</option>
	      </term>
	    <listitem>
	      <para>
		This table is passed as a parameter to the rule and action
		functions. It contains information on the operating system run by
		the host (if the <option>-O</option> switch was supplied), the
		IP address and the host name of the scanned target.
	      </para>
	    </listitem>
	  </varlistentry>

	  <varlistentry>
	    <term><option>host.os</option>
	      </term>
	    <listitem>
	      <para>
		An array of OS match tables. An OS match consists of a
		human-readable name and an array of OS classes. Each OS
		class consists of a vendor, OS family, OS generation,
		device type, and an array of
                CPE<indexterm><primary>CPE</primary></indexterm> entries
                for the class. (See <xref linkend="osdetect-ref-format"/>
                for a description of OS match fields.) Fields may be
                <varname>nil</varname> if they are not defined. The
                <varname>host.os</varname> table has this overall
                structure:
	      </para>

<programlisting>
host.os = {
  {
    name = <replaceable>string</replaceable>,
    classes = {
      {
        vendor = <replaceable>string</replaceable>,
        osfamily = <replaceable>string</replaceable>,
        osgen = <replaceable>string</replaceable>,
        type = <replaceable>string</replaceable>,
        cpe = {
          "cpe:/<replaceable>...</replaceable>",
          <optional>More CPE</optional>
        }
      },
      <optional>More classes</optional>
    },
  },
  <optional>More OS matches</optional>
}
</programlisting>

	      <para>
	        For example, an OS match on this
	        <filename>nmap-os-db</filename><indexterm><primary><filename>nmap-os-db</filename></primary></indexterm>
	        entry:
	      </para>

<programlisting>
Fingerprint Linux 2.6.32 - 3.2
Class Linux | Linux | 2.6.X | general purpose
CPE cpe:/o:linux:linux_kernel:2.6
Class Linux | Linux | 3.X | general purpose
CPE cpe:/o:linux:linux_kernel:3
</programlisting>

              <para>
                will result in this <varname>host.os</varname> table:
              </para>

<programlisting>
host.os = {
  {
    name = "Linux 2.6.32 - 3.2",
    classes = {
      {
        vendor = "Linux",
        osfamily = "Linux",
        osgen = "2.6.X",
        type = "general purpose",
        cpe = { "cpe:/o:linux:linux_kernel:2.6" }
      },
      {
        vendor = "Linux",
        osfamily = "Linux",
        osgen = "3.X",
        type = "general purpose",
        cpe = { "cpe:/o:linux:linux_kernel:3" }
      }
    },
  }
}
</programlisting>

              <para>
                Only entries corresponding to perfect OS matches are put
                in the <varname>host.os</varname> table. If Nmap was run
                without the <option>-O</option> option, then
                <literal>host.os</literal> is <literal>nil</literal>.
	      </para>
	    </listitem>
	  </varlistentry>
	  
	  <varlistentry>
	    <term><option>host.ip</option>
	      </term>
	    <listitem>
	      <para>Contains a string representation of the IP address of the
		target host. If the scan was run against a host name and its
		DNS lookup returned more than one IP addresses, then the
		same IP address is used as the one chosen for the scan.
	      </para>
	    </listitem>
	  </varlistentry>
	  
	  <varlistentry>
	    <term><option>host.name</option>
	      </term>
	    <listitem>
	      <para>Contains the reverse DNS entry of the scanned target host
		represented as a string. If the host has no reverse DNS entry,
		the value of the field is an empty string.
	      </para>
	    </listitem>
	  </varlistentry>
	  <varlistentry>
	    <term><option>host.targetname</option>
	      </term>
	    <listitem>
	      <para>Contains the name of the host as specified on the command line.
		If the target given on the command line contains a netmask or is an IP
		address the value of the field is <literal>nil</literal>.
	      </para>
	    </listitem>
	  </varlistentry>
	  <varlistentry>
	    <term><option>host.reason</option>
	      </term>
	    <listitem>
	      <para>
    Contains a string representation of the reason why the target host is in
    its current state. The reason is given by the type of the packet that
    determined the state. For example, an <literal>echo-reply</literal> from
    an alive host.
	      </para>
	    </listitem>
	  </varlistentry>
	  <varlistentry>
	    <term><option>host.reason_ttl</option>
	      </term>
	    <listitem>
	      <para>
    Contains the TTL value of the response packet, that was used to determine 
    the status of the target host, when it arrived. This response packet is the 
    packet that is also used to set <literal>host.reason</literal>.
	      </para>
	    </listitem>
	  </varlistentry>
	  <varlistentry>
	    <term><option>host.directly_connected</option>
	      </term>
	    <listitem>
	      <para> A Boolean value indicating whether or not the target host is
		  directly connected to (i.e. on the same network segment as) the host running Nmap.
		  </para>
	    </listitem>
	  </varlistentry>
	  <varlistentry>
	    <term><option>host.mac_addr</option>
	      </term>
	    <listitem>
	      <para>MAC address<indexterm><primary>MAC address</primary></indexterm>
                  of the destination host (six-byte-long binary
		  string) if available, otherwise <literal>nil</literal>.  The MAC address is generally only available for hosts directly connected on a LAN and only if Nmap is doing a raw packet scan such as SYN scan.
		  </para>
	    </listitem>
	  </varlistentry>
	  <varlistentry>
	    <term><option>host.mac_addr_next_hop</option>
	      </term>
	    <listitem>
	      <para>MAC address
	      of the first hop in the route to the host, or
	      <literal>nil</literal> if not available.
	      </para>
	    </listitem>
	  </varlistentry>
	  <varlistentry>
	    <term><option>host.mac_addr_src</option>
	      </term>
	    <listitem>
	      <para>Our own MAC address, which was used to connect to the
		  host (either our network card's, or (with
		  <option>--spoof-mac</option>)<indexterm><primary><option>--spoof-mac</option></primary></indexterm>
                  the spoofed address).
		  </para>
	    </listitem>
	  </varlistentry>
	  <varlistentry>
	    <term><option>host.interface</option>
	      </term>
	    <listitem>
	      <para>A string containing the interface name
	      (dnet-style)<indexterm><primary>libdnet</primary></indexterm>
              through 
		  which packets to the host are sent.
		  </para>
	    </listitem>
	  </varlistentry>
	  <varlistentry>
	    <term><option>host.interface_mtu</option>
	      </term>
	    <listitem>
	      <para>The MTU (maximum transmission unit) for <literal>host.interface</literal>,
		or 0 if not known.
		  </para>
	    </listitem>
	  </varlistentry>
	  <varlistentry>
	    <term><option>host.bin_ip</option>
	      </term>
	    <listitem>
	      <para>The target host's IP address as a 4-byte (IPv4) or 16-byte (IPv6) string.
		  </para>
	    </listitem>
	  </varlistentry>
	  <varlistentry>
	    <term><option>host.bin_ip_src</option>
	      </term>
	    <listitem>
	      <para>Our host's (running Nmap) source IP address as a 4-byte (IPv4) or 16-byte (IPv6) string.
		  </para>
	    </listitem>
	  </varlistentry>

	  <varlistentry>
	    <term><option>host.times</option>
	      </term>
	    <listitem>
	      <para>This table contains Nmap's timing data for the host (see
		<xref linkend="scan-methods-rtt"/>).  Its keys are <literal>srtt</literal> (smoothed
		round trip time), <literal>rttvar</literal> (round trip time variance), and <literal>timeout</literal>
		(the probe timeout), all given in floating-point seconds.
		  </para>
	    </listitem>
	  </varlistentry>

	  <varlistentry>
	    <term><option>host.traceroute</option>
	      </term>
	    <listitem>
	      <para>
                This is an array of traceroute hops, present when the
                <option>--traceroute</option> option was used. Each entry is a
                host table with fields <literal>name</literal>,
                <literal>ip</literal> and <literal>srtt</literal> (round
                trip time). The TTL for an entry is implicit given its position
                in the table. An empty table represents a timed-out hop.
	      </para>
	    </listitem>
	  </varlistentry>

	  <varlistentry>
	    <term><option>host.os_fp</option>
	      </term>
	    <listitem>
        <para>If OS detection was performed, this is a string containing the OS
          fingerprint for the host. The format is described in
          <xref linkend="osdetect-fingerprint-format"/>.
		  </para>
	    </listitem>
	  </varlistentry>

	  <varlistentry>
	    <term><option>port</option>
	      </term>
	    <listitem>
	      <para>
		The port table is passed to an NSE service script (i.e. only those with a portrule rather than a hostrule) in the same
		fashion as the host table. It contains information about the port
		against which the script is running.  While this table is not passed to host scripts, port states on the target can still be requested from Nmap
		using the <literal>nmap.get_port_state()</literal> and <literal>nmap.get_ports()</literal> calls.
	      </para>
	    </listitem>
	  </varlistentry>

	  <varlistentry>
	    <term><option>port.number</option>
	      </term>
	    <listitem>
	      <para>
		Contains the port number of the target port.
	      </para>
	    </listitem>
	  </varlistentry>

	  
	  <varlistentry>
	    <term><option>port.protocol</option>
	      </term>
	    <listitem>
	      <para>
		Defines the protocol of the target port. Valid values are
		<literal>"tcp"</literal> and <literal>"udp"</literal>.
	      </para>
	    </listitem>
	  </varlistentry>

	  
	  <varlistentry>
	    <term><option>port.service</option>
	      </term>
	    <listitem>
	      <para>
		Contains a string representation of the service running on
		<literal>port.number</literal> as detected by the Nmap service
		detection. If the <literal>port.version.service_dtype</literal> field is
		<literal>"table"</literal>, Nmap has guessed the service based
		on the port number. Otherwise version detection was able to determine the listening service and this field is equal to
		<literal>port.version.name</literal>.
	      </para>
	    </listitem>
	  </varlistentry>
	  
	  <varlistentry>
	    <term><option>port.reason</option>
	      </term>
	    <listitem>
	      <para>
    Contains a string representation of the reason why the target port is in 
    its current state (given by <literal>port.state</literal>). The reason is 
    given by the type of the packet that determined the state. For example, a 
    <literal>RST</literal> packet from a closed port or 
    <literal>SYN-ACK</literal> from an open port.
	      </para>
	    </listitem>
	  </varlistentry>

	  <varlistentry>
	    <term><option>port.reason_ttl</option>
	      </term>
	    <listitem>
	      <para>
    Contains the TTL value of the response packet, that was used to determine 
    the status of the target port, when it arrived. This response packet is the 
    packet that is also used to set <literal>port.reason</literal>.
	      </para>
	    </listitem>
	  </varlistentry>

	  <varlistentry>
	    <term><option>port.version</option>
	      </term>
	    <listitem>
	      <para>
		This entry is a table which contains information
		retrieved by the Nmap version scanning engine.  Some
		of the values (such as service name, service type
		confidence, and the RPC-related values) may be retrieved by
		Nmap even if a version scan was not performed.  Values
		which were not determined default to
		<literal>nil</literal>.  The meaning of each value is given in the following table:</para>

<table id="scripting-tbl-port-version-values">
 <title><literal>port.version</literal> values</title>
 <tgroup cols="2">
 <colspec colwidth="2*" />
 <colspec colwidth="5*" />
   <thead><row>
    <entry>Name</entry>
    <entry>Description</entry>
   </row></thead>
   <tbody>
   <row>
    <entry align="left"><literal>name</literal></entry>
    <entry>Contains the service name Nmap decided on for the port.</entry>
   </row>

   <row>
    <entry align="left"><literal>name_confidence</literal></entry>
    <entry>Evaluates how confident Nmap is about the accuracy of
      <literal>name</literal>, from 1 (least confident) to 10. If
      <literal>port.version.service_dtype</literal> is
      <literal>"table"</literal>, this is 3.</entry>
   </row>

   <row>
    <entry align="left"><literal>product</literal>, <literal>version</literal>, <literal>extrainfo</literal>, <literal>hostname</literal>, <literal>ostype</literal>, <literal>devicetype</literal></entry>
    <entry>These five variables are the same as those described under <replaceable>versioninfo</replaceable> in <xref linkend="vscan-db-match"/>.
   </entry>
   </row>

   <row>
    <entry align="left"><literal>service_tunnel</literal></entry>
    <entry>Contains the string <literal>"none"</literal> or <literal>"ssl"</literal> based on whether or not Nmap used SSL tunneling to detect the service.</entry>
   </row>

   <row>
    <entry align="left"><literal>service_fp</literal></entry>
    <entry>The service fingerprint, if any, is provided in this value.  This is described in 
<xref linkend="vscan-community"/>.
</entry>
   </row>

   <row>
    <entry align="left"><literal>service_dtype</literal></entry>
    <entry>Contains the string <literal>"table"</literal> or
      <literal>"probed"</literal> based on whether or not Nmap deduced
      <literal>port.version.name</literal> from the
      <filename>nmap-services</filename> file or from a service probe match.
    </entry>
   </row>

   <row>
    <entry align="left"><literal>cpe</literal></entry>

    <entry>List of CPE codes for the detected service. As described in the
    <ulink url="http://cpe.mitre.org">official CPE specification</ulink> these strings
    all start with the <literal>cpe:/</literal> prefix.</entry>
   </row>

</tbody></tgroup></table>

	    </listitem>
	  </varlistentry>
	  
	  <varlistentry>
	    <term><option>port.state</option>
	      </term>
	    <listitem>
	      <para>
		Contains information on the state of the port.
		Service scripts are only run against ports in the
		<literal>open</literal> or
		<literal>open|filtered</literal> states, so
		<literal>port.state</literal> generally contains one
		of those values. Other values might appear if the port
		table is a result of the
		<literal>get_port_state</literal> or <literal>get_ports</literal>
		functions.  You can adjust the port state using the
		<literal>nmap.set_port_state()</literal> call.  This is
		normally done when an <literal>open|filtered</literal>
		port is determined to be <literal>open</literal>.
	      </para>
	    </listitem>
	  </varlistentry>


	</variablelist>
      </para>

    </sect2>

    <sect2 id="nse-api-networkio">
      <title>Network I/O API</title>
      <para>
	To allow for efficient and parallelizable network I/O, NSE
	provides an interface to Nsock, the Nmap socket library. The
	smart callback mechanism Nsock uses is fully transparent to
	NSE scripts. The main benefit of NSE's sockets is that they
	never block on I/O operations, allowing many scripts to be run in parallel.
	The I/O parallelism is fully transparent to authors of NSE scripts.  
       In NSE you can either program as if you were using a single
        non-blocking socket or you can program as if your connection is
	blocking. Even blocking I/O calls return once a
	specified timeout has been exceeded. Two flavors of Network I/O are
       supported: connect-style and raw packet.
	</para>
	<sect3 id="nse-api-networkio-connect">
	  <title>Connect-style network I/O</title>
          <indexterm><primary>sockets in NSE</primary></indexterm>
	  <para>This part of the network API should be suitable for most
	  classical network uses: Users create a socket, connect it to a
	  remote address, send and receive data and finally close the socket.
	  Everything up to the Transport layer (which is either TCP, UDP or
	  SSL) is handled by the library.
        </para>
        <para>
          An NSE socket is created by calling
          <function>nmap.new_socket</function>, which returns a socket object.
          The socket object supports the usual <function>connect</function>,
          <function>send</function>, <function>receive</function>, and
          <function>close</function> methods. Additionally the functions
          <function>receive_bytes</function>,
          <function>receive_lines</function>, and
          <function>receive_buf</function> allow greater control
          over data reception.
          <xref linkend="nse-api-networkio-connect-example" xrefstyle="select: label nopage"/>
          shows the use of connect-style network operations. The
          <function>try</function> function is used for error handling, as described in
          <xref linkend="nse-exceptions"/>.
        </para>
        <example id="nse-api-networkio-connect-example">
          <title>Connect-style I/O</title>
<programlisting>
require("nmap")

local socket = nmap.new_socket()
socket:set_timeout(1000)
try = nmap.new_try(function() socket:close() end)
try(socket:connect(host.ip, port.number))
try(socket:send("login"))
response = try(socket:receive())
socket:close()
</programlisting>
        </example>
        </sect3>

	<sect3 id="nse-api-networkio-raw">
	  <title>Raw packet network I/O</title>
          <indexterm><primary>raw packets</primary><secondary>in NSE</secondary></indexterm>
	  <para>For those cases where the connection-oriented approach is too high-level,
	  NSE provides script developers with the
	  option of raw packet network I/O.</para>

          <para>Raw packet reception is handled through a
	  Libpcap<indexterm><primary>libpcap</primary></indexterm>
	  wrapper inside the Nsock
	  library.<indexterm><primary>Nsock</primary></indexterm>
	  The steps are to open a capture device, register listeners
	  with the device, and then process packets as they are
	  received.</para>

          <para>The <function>pcap_open</function> method creates a handle for raw socket reads from an
          ordinary socket object. This method takes a
          callback function, which computes a packet hash from
          a packet (including its headers). This hash can return any
          binary string, which is later compared to the strings
          registered with the <function>pcap_register</function>
          function. The packet hash callback will normally extract some
          portion of the packet, such as its source address.</para>

          <para>The pcap reader is instructed to listen for certain
          packets using the <function>pcap_register</function> function.
          The function takes a binary string which is compared against
          the hash value of every packet received. Those packets whose
          hashes match any registered strings will be returned by the
          <function>pcap_receive</function> method. Register the empty
          string to receive all packets.</para>

          <para>A script receives all packets for which a listener has
          been registered by calling the
          <function>pcap_receive</function> method. The method blocks
          until a packet is received or a timeout occurs.</para>

          <para>The more general the packet hash computing function is
          kept, the more scripts may receive the packet and proceed with
          their execution. To handle packet capture inside your
          script you first have to create a socket with
          <function>nmap.new_socket</function> and later close the socket
          with <function>socket_object:close</function>&mdash;just like
          with the connection-based network I/O.</para>

          <para>While receiving packets is important, sending them is certainly
          a key feature as well.  To accomplish this, NSE provides access to
          sending at the IP and Ethernet layers.  Raw packet writes do not use
          the same socket object as raw packet reads, so the <function>nmap.new_dnet</function>
          function is called to create the required object for sending.  After
          this, a raw socket or Ethernet interface handle can be opened for use.</para>

          <para>Once the dnet object is created, the function <function>ip_open</function>
          can be called to initialize the object for IP sending.  <function>ip_send</function>
          sends the actual raw packet, which must start with the IP header.
          The dnet object places no restrictions on which IP hosts may be sent
          to, so the same object may be used to send to many different hosts
          while it is open.  To close the raw socket, call <function>ip_close</function>.</para>

          <para>For sending at a lower level than IP, NSE provides functions for
          writing Ethernet frames.  <function>ethernet_open</function> initializes
          the dnet object for sending by opening an Ethernet interface.  The raw
          frame is sent with <function>ethernet_send</function>.  To close the
          handle, call <function>ethernet_close</function>.</para>

          <para>Sometimes the easiest ways to understand complex APIs is by
          example.  The
          <filename>ipidseq</filename><indexterm><primary><filename>ipidseq</filename> script</primary></indexterm>
          script included with
          Nmap uses raw IP packets to test hosts for suitability for Nmap's
          Idle Scan (<option>-sI</option>).  The
          <filename>sniffer-detect</filename><indexterm><primary><filename>sniffer-detect</filename> script</primary></indexterm>
          script also included with Nmap uses raw Ethernet frames in an attempt
          to detect promiscuous-mode machines on the network (those running
          sniffers).</para>

	</sect3>
	</sect2>

    <sect2 id="nse-structured-output">
      <title>Structured and Unstructured Output</title>
      <indexterm>structured script output</indexterm>

      <para>
        NSE scripts should usually return a table representing their
        output, one that is nicely organized and has thoughtfully chosen
        keys. Such a table will be automatically formatted for screen
        output and will be stored as nested elements in XML output.
        Having XML output broken down logically into keys and values
        makes it easier for other tools to make use of script output.
        It is possible for a script to return only a string, but doing
        so is deprecated. In the past, scripts could only return a
        string, and their output was simply copied to the XML as a blob
        of text&ndash;this is now known as <quote>unstructured
        output</quote>.
      </para>

      <para>
        Suppose a script called <filename>user-list</filename> returns a
        table as shown in this code sample. The following paragraphs
        show how it appears in normal and XML output.
      </para>

<programlisting>
local output = stdnse.output_table()
output.hostname = "slimer"
output.users = {}
output.users[#output.users + 1] = "root"
output.users[#output.users + 1] = "foo"
output.users[#output.users + 1] = "bar"
return output
</programlisting>

      <para>
        A Lua table is converted to a string for normal output. The way
        this works is: each nested table gets a new level of
        indentation. Table entries with string keys are preceded by the
        key and a colon; entries with integer keys simply appear in
        order.
        Unlike normal Lua tables, which are unordered, a table that
        comes from <code>stdnse.output_table</code> will keep its keys in
        the order they were inserted.
        <xref linkend="nse-normal-structured-output"/> shows how the
        example table appears in normal output.
      </para>

      <example id="nse-normal-structured-output">
        <title>Automatic formatting of NSE structured output</title>
<screen>
PORT     STATE SERVICE
1123/tcp open  unknown
| user-list:
|   hostname: slimer
|   users:
|     root
|     foo
|_    bar
</screen>
      </example>

      <para>
        The XML representation of a Lua table is constructed as follows.
        Nested table become <code>table</code> elements. Entries of
        tables that are not themselves tables become <code>elem</code>
        elements. Entries (whether <code>table</code> or
        <code>elem</code>) with string keys get a <code>key</code>
        attribute (e.g.
        <code>&lt;elem key="username"&gt;foo&lt;/elem&gt;</code>);
        entries with integer keys have no <code>key</code> element and
        their key is implicit in the order in which they appear.
      </para>
      <para>
        In addition to the above, whatever normal output the script
        produces (even if automatically generated) is copied to the
        <code>output</code> attribute of the <code>script</code>
        element. Newlines and other special characters will be encoded
        as XML character entities, for example <code>&amp;#xa;</code>.
        <xref linkend="nse-xml-structured-output"/> shows how the example
        table appears in XML.
      </para>

      <example id="nse-xml-structured-output">
        <title>NSE structured output in XML</title>
<screen><![CDATA[<script id="t" output="&#xa;hostname: slimer&#xa;users: &#xa;  root&#xa;  foo&#xa;  bar">
  <elem key="hostname">slimer</elem>
  <table key="users">
    <elem>root</elem>
    <elem>foo</elem>
    <elem>bar</elem>
  </table>
</script>
]]></screen>
      </example>

      <para>
        Some scripts need more control their normal output. This is the
        case, for example, with scripts that need to display complex
        tables. For complete control over the output, these scripts may
        do either of these things:
        <simplelist>
          <member>return a string as second return value, or</member>
          <member>set the <code>__tostring</code> metamethod on the
          returned table.</member>
        </simplelist>
        The resulting string will be used in normal output, and the
        table will be used in XML as usual. The formatted string may
        contain newline characters to appear as multiple lines.
      </para>
      <para>
        If the above code example were modified in this way to return a
        formatted string,
<programlisting>
local output = stdnse.output_table()
output.hostname = "slimer"
output.users = {}
output.users[#output.users + 1] = "root"
output.users[#output.users + 1] = "foo"
output.users[#output.users + 1] = "bar"
local output_str = string.format("hostname: %s\n", output.hostname)
output_str = output_str .. "\n" .. stringaux.strjoin(", ", output.users)
return output, output_str
</programlisting>
        then the normal output would appear as follows:
<screen>
PORT     STATE SERVICE
1123/tcp open  unknown
| user-list:
|   hostname: slimer
|_  users: root, foo, bar
</screen>
      </para>

      <sect3 id="nse-structured-output-conventions">
        <para>
        There are conventions regarding the formatting of certain kinds
        of data in structured output. Users of NSE output benefit by
        being able to assume that some kinds of data, for instance dates
        and times, are formatted the same way, even in different
        scripts.
        </para>
        <para>
        Network addresses, for example IPv4, IPv6, and MAC, are
        represented as strings.
        </para>
        <para>
        Long hexadecimal strings such as public key fingerprints should
        be written using lower-case alphabetical characters and without
        separators such as colons.
        </para>
        <para>
        Dates and times are formatted according to
        <ulink role="hidepdf" url="http://www.rfc-editor.org/rfc/rfc3339.txt">RFC
        3339</ulink><indexterm><primary>RFC 3339</primary></indexterm>.
        If the time zone offset is known, they should appear like these
        examples:
<screen>
2012-09-07T23:37:42+00:00
2012-09-07T23:37:42+02:00
</screen>
        If the time zone offset is not known (representing some
        unspecified local time), leave off the offset part:
<screen>
2012-09-07T23:37:42
</screen>
        The library function
        <code>datetime.format_timestamp</code> code exists to format times
        for structured output. It takes an optional time zone offset in
        seconds and automatically shifts the date to be correct within
        that offset.
<screen>
datetime.format_timestamp(os.time(), 0) --> "2012-09-07T23:37:42+00:00"
</screen>
        </para>
      </sect3>
    </sect2>

    <sect2 id="nse-exceptions">
      <title>Exception Handling</title>
      <indexterm><primary>exceptions in NSE</primary></indexterm>
      <para>
	  NSE provides an exception handling mechanism which is not present in
	  the base Lua language. It is tailored
	  specifically for network I/O operations, and
	  follows a functional programming paradigm rather than an
	  object-oriented one. The <function>nmap.new_try</function> API method is used to
          create an exception handler.  This method returns a function which takes a variable
          number of arguments that are assumed to be the return values of
          another function. If an exception is detected in the return
          values (the first return value is false),
	  then the script execution is aborted and no
	  output is produced. Optionally, you can pass a function to
	  <function>new_try</function> which will be called
	  if an exception is caught. The function would generally perform any required cleanup operations.
	  </para>

      <para>
	  <xref linkend="nse-exception-handling" xrefstyle="select: label nopage"/> shows cleanup
	  exception handling at work.  A new function named
	  <function>catch</function> is defined to simply close the
	  newly created socket in case of an error.  It is then used
	  to protect connection and communication attempts on that
	  socket. If no catch function is specified, execution of the
	  script aborts without further ado&mdash;open sockets will
	  remain open until the next run of Lua's garbage
	  collector. If the verbosity level is at least one or if the
	  scan is performed in debugging mode, a description of the
	  uncaught error condition is printed on standard output.
	  Note that it is currently not easily possible to group
	  several statements in one try block.

	</para>

        <example id="nse-exception-handling">
          <title>Exception handling example</title>
<programlisting>
local result, socket, try, catch

result = ""
socket = nmap.new_socket()
catch = function() 
socket:close() 
end
try = nmap.new_try(catch)

try(socket:connect(host.ip, port.number))
result = try(socket:receive_lines(1))
try(socket:send(result))
</programlisting>
       </example>
	
	  <para>
	  Writing a function which is treated properly by the
	  try/catch mechanism is straightforward. The function should
	  return multiple values.  The first value should be a Boolean
	  which is <literal>true</literal> upon successful completion of the function and
	  <literal>false</literal> (or <literal>nil</literal>) otherwise. If the function completed successfully, the try
	  construct consumes the indicator value and returns the
	  remaining values.  If the function failed then the second
	  returned value must be a string describing the error
	  condition. Note that if the value is not
          <literal>nil</literal> or <literal>false</literal> it is
	  treated as <literal>true</literal> so you can return your
	  value in the normal case and return <literal>nil, <replaceable>error description</replaceable></literal>
	  if an error occurs.
	</para>

    </sect2>

    <sect2 id="nse-api-registry">
      <title>The Registry</title>
      <indexterm><primary>registry (NSE)</primary></indexterm>
	<para>Scripts can share information by storing values in a
	<firstterm>register</firstterm>, which is a special table that can be
	accessed by all scripts. There is a global registry with the name
	<varname>nmap.registry</varname>, shared by all scripts. Each host
	additionally has its own registry called
	<varname>host.registry</varname>, where <varname>host</varname> is the
	<link linkend="nse-api-arguments">host table</link> passed to a script.
	Information in the registries is not stored between Nmap
	executions.</para>

	<para>The global registry persists throughout an entire scan session.
	Scripts can use it, for example, to store values that will later be
	displayed by a postrule script. The per-host registries, on the other
	hand, only exist while a host is being scanned. They can be used to send
	information from one script to another one that runs against the same
	host. When possible, use the per-host registry; this not only saves you
	from having to make key names unique across hosts, but also allows the
	memory used by the registry to be reclaimed when it is no longer
	needed.</para>

	<para>
	Here are examples of using both registries:
	<simplelist>
	<member>The portrule of the <filename>ssh-hostkey</filename> script collects SSH key fingerprints
	and stores them in the global <varname>nmap.registry</varname> so they
	can be printed later by the postrule.</member>
	<member>The <filename>ssl-cert</filename> script collects SSL certificates and
	stores them in the per-host registry so that the
	<filename>ssl-google-cert-catalog</filename> script can use them without
	having to make another connection to the server.</member>
	</simplelist>
	</para>

	<para>Because every script can write to the global registry table, it is
	important to make the keys you use unique, to avoid overwriting the keys
	of other scripts (or the same script running in parallel).</para>

	<para>Scripts that use the results of another script must declare it using
	the <literal>dependencies</literal> variable to make sure that the earlier
	script runs first.</para>
    </sect2>
    <indexterm class="endofrange" startref="nse-nmap-indexterm"/>
  </sect1>

  <sect1 id="nse-tutorial">
    <title>Script Writing Tutorial</title>
    <indexterm class="startofrange" id="nse-tutorial-indexterm"><primary>Nmap Scripting Engine (NSE)</primary><secondary>tutorial</secondary></indexterm>

    <para>
      Suppose that you are convinced of the power of NSE. How do you
      go about writing your own script?  Let's say
      that you want to extract information from an identification
      server<indexterm><primary>auth service</primary></indexterm> to determine the owner of the process listening on a TCP port.
      This is not really the purpose of identd (it is meant for querying the owner of outgoing connections, not listening daemons), but many identd servers allow it anyway.  Nmap used to have this functionality (called ident scan), but it was removed
      while transitioning to a new scan engine architecture.  The protocol identd uses is pretty simple, but still too
      complicated to handle with Nmap's version detection
      language. First, you connect to the identification server and
      send a query of the form <literal><replaceable>port-on-server</replaceable>,
      <replaceable>port-on-client</replaceable></literal> and
      terminated with a newline character. The server should then
      respond with a string containing the server port, client port,
      response type, and address information.  The address information
      is omitted if there is an error. More details are available
      in <ulink role="hidepdf"
      url="http://www.rfc-editor.org/rfc/rfc1413.txt">RFC
      1413</ulink>, but this description is sufficient for our
      purposes.  The protocol cannot be modeled in Nmap's version
      detection language for two reasons. The first is that you need
      to know both the local and the remote port of a
      connection. Version detection does not provide this data.  The
      second, more severe obstacle, is that you need two open
      connections to the target&mdash;one to the identification server
      and one to the listening port you wish to query. Both obstacles
      are easily overcome with NSE.</para>

    <para>
      The anatomy of a script is described in <xref linkend="nse-script-format"/>.
      In this section we will show how the described structure is utilized.
    </para>
    
    <sect2 id="nse-tutorial-head">
      <title>The Head</title>
      <para>

      The head of the script is essentially its meta information. This
      includes the
      fields: <literal>description</literal>, <literal>categories</literal>, <literal>dependencies</literal>, <literal>author</literal>, and <literal>license</literal> as well as
      initial NSEDoc information such as usage, args, and output
      tags (see <xref linkend="nsedoc"/>).
      </para>

      <para>
	The description field should contain a paragraph or more describing what the script does.  If anything about the script results might confuse or mislead users, and you can't eliminate the issue by improving the script or results text, it should be documented in the <literal>description</literal>.  If there are multiple paragraphs, the first is used as a short summary where necessary.  Make sure that first paragraph can serve as a stand alone abstract.  This description is short because it is such a simple script:
      </para>
      <para>
<indexterm><primary><literal>auth-owners</literal> script</primary></indexterm>
<indexterm><primary sortas="description script variable">&ldquo;<varname>description</varname>&rdquo; script variable</primary></indexterm>
<programlisting>
description = [[
Attempts to find the owner of an open TCP port by querying an auth
(identd - port 113) daemon which must also be open on the target system.
]]
</programlisting>
      </para>

<para>Next comes NSEDoc information.  This script is missing the
common <literal>@usage</literal> and <literal>@args</literal> tags
since it is so simple, but it does have an
NSEDoc <literal>@output</literal> tag:</para>

<programlisting>
---
--@output
-- 21/tcp   open     ftp       ProFTPD 1.3.1
-- |_ auth-owners: nobody
-- 22/tcp   open     ssh       OpenSSH 4.3p2 Debian 9etch2 (protocol 2.0)
-- |_ auth-owners: root
-- 25/tcp   open     smtp      Postfix smtpd
-- |_ auth-owners: postfix
-- 80/tcp   open     http      Apache httpd 2.0.61 ((Unix) PHP/4.4.7 ...)
-- |_ auth-owners: dhapache
-- 113/tcp  open     auth?
-- |_ auth-owners: nobody
-- 587/tcp  open     submission Postfix smtpd
-- |_ auth-owners: postfix
-- 5666/tcp open     unknown
-- |_ auth-owners: root
</programlisting>


      <para>

        Next come the <literal>author</literal>, <literal>license</literal>, and <literal>categories</literal> tags.
        This script belongs to the
        <literal>safe</literal><indexterm><primary><literal>safe</literal>
        script category</primary></indexterm> because we are not using
        the service for anything it was not intended for. Because this
        script is one that should run by default it is also in the
        <literal>default</literal><indexterm><primary><literal>default</literal>
        script category</primary></indexterm>
        category.  Here are the variables in context:</para>

<indexterm><primary sortas="categories script variable">&ldquo;<varname>categories</varname>&rdquo; script variable</primary></indexterm>
<programlisting>
author = "Diman Todorov"

license = "Same as Nmap--See https://nmap.org/book/man-legal.html"

categories = {"default", "safe"}
</programlisting>
    </sect2>

    <sect2 id="nse-tutorial-rule">
      <title>The Rule</title>
      <para>
	The rule section is a Lua method which decides whether to skip
    or execute the script's action.  This decision is usually based on
    the type of the rule and the host and port information passed to
    it.  A <literal>prerule</literal> or a
    <literal>postrule</literal> will always evaluate to true.  In the
    case of the identification script, it is slightly more complicated
	than that. To decide whether to run the identification script
	against a given port we need to know if there is an auth
	server running on the target machine. In other words, the
	script should be run only if the currently scanned TCP port is open and
	TCP port 113 is also open. For now we will rely on the fact that
	identification servers listen on TCP port 113. Unfortunately NSE
	only gives us information about the currently scanned port.</para>

	<para>To find out if port 113 is open, we use the
	<function>nmap.get_port_state</function> function. If the auth
	port was not scanned, the <literal>get_port_state</literal>
	function returns <literal>nil</literal>. So we check that
	the table is not <literal>nil</literal>. We also
	check that both ports are in the <literal>open</literal> state.
	If this is the case, the action is executed, otherwise we skip
	the action.
      </para>
      <para>
<indexterm><primary sortas="portrule script variable">&ldquo;<varname>portrule</varname>&rdquo; script variable</primary></indexterm>
<programlisting>
portrule = function(host, port)
	local auth_port = { number=113, protocol="tcp" }
	local identd = nmap.get_port_state(host, auth_port)

	return identd ~= nil
		and identd.state == "open"
		and port.protocol == "tcp"
		and port.state == "open"
end
</programlisting>
      </para>

    </sect2>

    <sect2 id="nse-tutorial-action">
      <title>The Action</title>
      <para>
	At last we implement the actual functionality! The script 
	first connects to the port on which we expect to find the
	identification server, then it will connect to the port we
	want information about. Doing so involves first creating two socket options by calling <function>nmap.new_socket</function>.  Next we define an error-handling <function>catch</function> function which closes those sockets if failure is detected.  At this point we can safely use object methods such as <function>open</function>, 
	<function>close</function>, 
	<function>send</function> and
	<function>receive</function> to operate on the network socket.  In this case we call <function>connect</function> to make the connections.  NSE's exception handling mechanism<indexterm><primary>exceptions in NSE</primary></indexterm>
        is used to avoid excessive error-handling code.  We simply wrap the networking calls in a <function>try</function> call which will in turn call our <function>catch</function> function if anything goes wrong.</para>


     <para>If the two connections succeed, we construct a query string
	and parse the response. If we received a satisfactory
	response, we return the retrieved information.
      </para>
      
      <para>
        <indexterm><primary sortas="action script variable">&ldquo;<varname>action</varname>&rdquo; script variable</primary></indexterm>
<programlisting>
action = function(host, port)
        local owner = ""

        local client_ident = nmap.new_socket()
        local client_service = nmap.new_socket()

        local catch = function()
                client_ident:close()
                client_service:close()
        end

        local try = nmap.new_try(catch)

        try(client_ident:connect(host.ip, 113))
        try(client_service:connect(host.ip, port.number))

        local localip, localport, remoteip, remoteport =
                try(client_service:get_info())

        local request = port.number .. ", " .. localport .. "\r\n"

        try(client_ident:send(request))

        owner = try(client_ident:receive_lines(1))

        if string.match(owner, "ERROR") then 
                owner = nil
        else
                owner = string.match(owner,
                        "%d+%s*,%s*%d+%s*:%s*USERID%s*:%s*.+%s*:%s*(.+)\r?\n")
        end

        try(client_ident:close())
        try(client_service:close())

        return owner
end
</programlisting>
      </para>

<para>Note that because we know that the remote port is stored
in <literal>port.number</literal>, we could have ignored the last two
return values of <literal>client_service:get_info()</literal> like
this:</para>

<programlisting>
local localip, localport = try(client_service:get_info())
</programlisting>

<para>In this example we exit quietly if the service responds with an error.  This is done by assigning <literal>nil</literal> to the <varname>owner</varname> variable which will be returned.  NSE scripts generally only return messages when they succeed, so they don't flood the user with pointless alerts.</para>

    </sect2>
    <indexterm class="endofrange" startref="nse-tutorial-indexterm"/>
  </sect1>

  <sect1 id="nsedoc">
    <title>Writing Script Documentation (NSEDoc)</title>
    <indexterm class="startofrange" id="nsedoc-indexterm"><primary>Nmap Scripting Engine (NSE)</primary><secondary>documentation in</secondary></indexterm>
    <indexterm class="startofrange" id="nse-nsedoc-indexterm"><primary>NSEDoc</primary></indexterm>

    <para>
    Scripts are used by more than just their authors, so they require good
    documentation. NSE modules need documentation so developers can
    use them in their scripts. NSE's documentation system, described in
    this section, aims to meet both these needs. While reading this
    section, you may want to browse NSE's online documentation, which is
    generated using this system. It is at
    <ulink url="https://nmap.org/nsedoc/"/>.
    </para>

    <para>
    NSE uses a customized version of the
    <ulink url="http://luadoc.luaforge.net/">LuaDoc</ulink><indexterm><primary>LuaDoc</primary></indexterm>
    documentation system called NSEDoc.
    The documentation for scripts
    and modules is contained in their source code, as
    comments with a special form.
    <xref linkend="nsedoc-comment" xrefstyle="select: label nopage"/>
    is an NSEDoc comment taken from the
    <function>stdnse.print_debug()</function> function.
    </para>

    <!-- From stdnse.lua. -->
    <!-- Be careful to change <code> to &lt;code&gt; when you copy code.
    <code> is a DocBook tag so it will disappear within a programlisting! -->
    <example id="nsedoc-comment">
      <title>An NSEDoc comment for a function</title>
<programlisting>
---
-- Prints a formatted debug message if the current verbosity level is greater
-- than or equal to a given level.
-- 
-- This is a convenience wrapper around
-- &lt;code&gt;nmap.log_write&lt;/code&gt;. The first optional numeric
-- argument, &lt;code&gt;level&lt;/code&gt;, is used as the debugging level necessary
-- to print the message (it defaults to 1 if omitted). All remaining arguments
-- are processed with Lua's &lt;code&gt;string.format&lt;/code&gt; function.
-- @param level Optional debugging level.
-- @param fmt Format string.
-- @param ... Arguments to format.
</programlisting>
    </example>

    <para>
    Documentation comments start with three dashes:
    <literal>---</literal>. The body of the comment is the description
    of the following code. The first paragraph of the description should
    be a brief summary, with the following paragraphs providing more
    detail. Special tags starting with <literal>@</literal> mark off
    other parts of the documentation. In the above example you see
    <literal>@param</literal>, which is used to describe each parameter
    of a function. A complete list of the documentation tags is found
    in <xref linkend="nsedoc-tags"/>.
    </para>

    <para>
    Text enclosed in the HTML-like <literal>&lt;code&gt;</literal> and
    <literal>&lt;/code&gt;</literal> tags will be rendered in a
    monospace font. This should be used for variable and function names,
    as well as multi-line code examples. When a sequence of lines start
    with the characters <quote><literal>* </literal></quote>, they will
    be rendered as a bulleted list. Each list item must be entirely on
    one physical line.
    </para>

    <para>
    It is good practice to document every public function and table in a
    script or module. Additionally every script and module should have
    its own file-level documentation. A documentation comment at the
    beginning of a file (one that is not followed by a function or table
    definition) applies to the entire file. File-level documentation can
    and should be several paragraphs long, with all the high-level
    information useful to a developer using a module or a user running a
    script.
    <xref linkend="nsedoc-module" xrefstyle="select: label nopage"/>
    shows documentation for the <literal>comm</literal> module (with a
    few paragraphs removed to save space).
    </para>

    <example id="nsedoc-module">
      <title>An NSEDoc comment for a module</title>
<programlisting>
---
-- Common communication functions for network discovery tasks like
-- banner grabbing and data exchange.
--
-- These functions may be passed a table of options, but it's not required. The
-- keys for the options table are &lt;code&gt;"bytes"&lt;/code&gt;, &lt;code&gt;"lines"&lt;/code&gt;,
-- &lt;code&gt;"proto"&lt;/code&gt;, and &lt;code&gt;"timeout"&lt;/code&gt;. &lt;code&gt;"bytes"&lt;/code&gt; sets
-- a minimum number of bytes to read. &lt;code&gt;"lines"&lt;/code&gt; does the same for
-- lines. &lt;code&gt;"proto"&lt;/code&gt; sets the protocol to communicate with,
-- defaulting to &lt;code&gt;"tcp"&lt;/code&gt; if not provided. &lt;code&gt;"timeout"&lt;/code&gt;
-- sets the socket timeout (see the socket function &lt;code&gt;set_timeout&lt;/code&gt;
-- for details). 
--
-- @author Kris Katterjohn 04/2008
-- @copyright Same as Nmap--See https://nmap.org/book/man-legal.html
</programlisting>
    </example>

    <para>
    There are some special considerations for documenting scripts rather than
    functions and modules. In particular, scripts have special variables for some information which
    would otherwise belongs in @-tag comments (script variables are described in
    <xref linkend="nse-script-format"/>). In particular, a script's
    description belongs in the <varname>description</varname> variable
    rather than in a documentation comment, and the information that
    would go in <literal>@author</literal> and
    <literal>@copyright</literal> belong in the variables
    <varname>author</varname> and <varname>license</varname> instead.
    NSEDoc knows about these variables and will use them in preference
    to fields in the comments. Scripts should also have
    <varname>@output</varname> and <varname>@xmloutput</varname> tags showing sample output, as well as <varname>@args</varname> and <varname>@usage</varname> where appropriate.
    <xref linkend="nsedoc-script" xrefstyle="select: label nopage"/>
    shows proper form for script-level documentation, using a
    combination of documentation comments and NSE variables.
    </para>

    <!-- From asn-query.nse. -->
    <example id="nsedoc-script">
      <title>An NSEDoc comment for a script</title>
<programlisting>
description = [[
Maps IP addresses to autonomous system (AS) numbers.

The script works by sending DNS TXT queries to a DNS server which in
turn queries a third-party service provided by Team Cymru
(team-cymru.org) using an in-addr.arpa style zone set up especially for
use by Nmap. The responses to these queries contain both Origin and Peer
ASNs and their descriptions, displayed along with the BGP Prefix and
Country Code. The script caches results to reduce the number of queries
and should perform a single query for all scanned targets in a BGP
Prefix present in Team Cymru's database.

Be aware that any targets against which this script is run will be sent
to and potentially recorded by one or more DNS servers and Team Cymru.
In addition your IP address will be sent along with the ASN to a DNS
server (your default DNS server, or whichever one you specified with the
&lt;code&gt;dns&lt;/code&gt; script argument).
]]

---
-- @usage
-- nmap --script asn-query [--script-args dns=&lt;DNS server&gt;] &lt;target&gt;
-- @args dns The address of a recursive nameserver to use (optional).
-- @output
-- Host script results:
-- |  asn-query:
-- |  BGP: 64.13.128.0/21 | Country: US
-- |    Origin AS: 10565 SVCOLO-AS - Silicon Valley Colocation, Inc.
-- |      Peer AS: 3561 6461
-- |  BGP: 64.13.128.0/18 | Country: US
-- |    Origin AS: 10565 SVCOLO-AS - Silicon Valley Colocation, Inc.
-- |_     Peer AS: 174 2914 6461

author = "jah, Michael"
license = "Same as Nmap--See https://nmap.org/book/man-legal.html"
categories = {"discovery", "external", "safe"}
</programlisting>
    </example>

    <indexterm><primary>NSEDoc</primary><secondary>for C modules</secondary></indexterm>
    <para>
    Compiled NSE modules are also documented with NSEDoc, even though
    they have no Lua source code. Each compiled module has a file
    <filename><replaceable>modulename</replaceable>.luadoc</filename><indexterm><primary sortas="luadoc filename extension"><filename>.luadoc</filename> filename extension</primary></indexterm>
    that is kept in the <filename>nselib</filename> directory alongside
    the Lua modules. This file lists and documents the functions and
    tables in the compiled module as though they were written in Lua.
    Only the name of each function is required, not its definition (not
    even <literal>end</literal>). You must use the
    <literal>@name</literal> and <literal>@class</literal> tags when
    documenting a table to assist the documentation parser in
    identifying it. There are several examples of this method of
    documentation in the Nmap source distribution (including <literal>nmap.luadoc</literal>, <literal>lfs.luadoc</literal>, and <literal>pcre.luadoc</literal>).
    </para>

    <sect2 id="nsedoc-tags">
      <title>NSE Documentation Tags</title>
      <para>
      The following tags are understood by NSEDoc:
      </para>
      <variablelist>
        <varlistentry>
          <term><option>@param</option></term>
          <listitem>
            <para>
              Describes a function parameter. The first word following
              <literal>@param</literal> is the name of the parameter
              being described. The tag should appear once for each
              parameter of a function.
            </para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term><option>@see</option></term>
          <listitem>
            <para>
              Adds a cross-reference to another function or table.
            </para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term><option>@return</option></term>
          <listitem>
            <para>
              Describes a return value of a function.
              <literal>@return</literal> may be used multiple times for
              multiple return values.
            </para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term><option>@usage</option></term>
          <listitem>
            <para>
              Provides a usage example of a function, script, or module. In
              the case of a function, the example is Lua code; for a
              script it is an Nmap command line; and for a module it is usually
              a code sample.
              <literal>@usage</literal> may be given more than once. If it is
              omitted in a script, NSEDoc generates a default standardized
              usage example.
            </para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term><option>@name</option></term>
          <listitem>
            <para>
              Defines a name for the function or table being documented.
              This tag is normally not necessary because NSEDoc infers
              names through code analysis.
            </para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term><option>@class</option></term>
          <listitem>
            <para>
              Defines the <quote>class</quote> of the object being
              documented: <literal>function</literal>,
              <literal>table</literal>, or <literal>module</literal>.
              Like <literal>@name</literal>, this is normally inferred
              automatically.
            </para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term><option>@field</option></term>
          <listitem>
            <para>
              In the documentation of a table, <varname>@field</varname> describes the value of a
              named field.
            </para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term><option>@args</option></term>
          <listitem>
            <para>
              Describes a script argument, as used with the
              <option>--script-args</option> option (see
              <xref linkend="nse-args"/>). The first word after
              <literal>@args</literal> is the name of the argument, and
              everything following that is the description. This tag is
              special to script-level comments.
            </para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term><option>@output</option></term>
          <listitem>
            <para>
              This tag, which is exclusive to
              script-level comments, shows sample output from a script.
            </para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term><option>@xmloutput</option></term>
          <listitem>
            <para>
	      Shows what the script's
	      <xref linkend="nse-structured-output">structured output</xref>
	      looks like when written to XML. The XML sample should not include
	      the enclosing <literal>&lt;script&gt;</literal> and
	      <literal>&lt;/script&gt;</literal> tags and should be indented to
	      show hierarchy.
            </para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term><option>@author</option></term>
          <listitem>
            <para>
              This tag, which may be given multiple times, lists the authors of an NSE module. For scripts, use the 
              <varname>author</varname> variable instead.
            </para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term><option>@copyright</option></term>
          <listitem>
            <para>
              This tag describes the copyright status of a module. For scripts,
              use the <varname>license</varname>
              variable instead.
            </para>
          </listitem>
        </varlistentry>
      </variablelist>

      <!-- These tags are undocumented here: @description, @summary, and
      @release. @documentation and @summary are automatically extracted
      from the contents of a comment. @release has not been used with
      NSEDoc. -->
    </sect2>
    <indexterm class="endofrange" startref="nsedoc-indexterm"/>
    <indexterm class="endofrange" startref="nse-nsedoc-indexterm"/>
  </sect1>

  <sect1 id="nse-parallelism">
    <title>Script Parallelism in NSE</title>
    <para>
      In <xref linkend="nse-api-networkio"/>, it was mentioned that NSE
      automatically parallelizes network operations. Usually this process is
      transparent to a script author, but there are some advanced techniques
      that require knowledge of how it works. The techniques covered in this
      section are controlling how multiple scripts interact in a library, using
      multiple threads in parallel, and disabling parallelism for special
      cases.
    </para>
    <para>
      The standard mechanism for parallel execution is a thread. A thread
      encapsulates the execution flow and data of a script.
      Lua thread may be yielded at arbitrary locations to continue
      work on another script. Typically, these yield locations are blocking
      socket operations in the
      <literal>nmap</literal><indexterm><primary><literal>nmap</literal> NSE library</primary></indexterm>
      library. The yield back to the script is also transparent, a side effect
      of the socket operation.
    </para>
    <para>
      Let's go over some common terminology. A <emphasis>script</emphasis> is
      analogous to a binary executable; it holds the information necessary to
      execute a script. A <emphasis>thread</emphasis> (a Lua coroutine) is
      analogous to a process; it runs a script against a host and possibly
      port. Sometimes we abuse terminology and refer to a running thread
      as a running <quote>script</quote>, but what this really means is an
      instantiation of a script, in the same way that a process is the
      instantiation of an executable.
    </para>
    <para>
      NSE provides the bare-bone essentials needed to expand parallelism
      basic model of one thread per script: new independent threads,
      mutexes, and condition variables.
    </para>
    <sect2 id="nse-parallelism-threads">
      <title>Worker Threads</title>
      <para>
        There are several instances where a script needs finer control with
        respect to parallel execution beyond what is offered by default with a
        generic script. A common need is to read from multiple sockets
        concurrently. For example, an HTTP
        spidering script may want to have multiple Lua threads querying web
        server resources in parallel.  To answer this need, NSE offers the
        function <literal>stdnse.new_thread</literal> to create worker threads.
        These worker threads have all the power of independent scripts with the
        only restriction that they may not report script output.
      </para>
      <para>
        Each worker thread launched by a script is given a main function and
        a variable number of arguments to be passed to the main function by
        NSE:
      </para>
      <para>
        <literal>worker_thread, status_function = stdnse.new_thread(main, ...)</literal>
      </para>
      <para>
        <literal>stdnse.new_thread</literal> returns two values: the Lua thread
        (coroutine) that uniquely identifies your worker thread, and a status
        query function that queries the status of your new worker.
        The status query function returns two values:
      </para>
      <para>
        <literal>status, error_object = status_function()</literal>
      </para>
      <para>
        The first return value is simply the return
        value of <literal>coroutine.status</literal> run on the worker thread
        coroutine. (More precisely, the <literal>base</literal> coroutine. Read
        more about <literal>base</literal> coroutine in <xref
        linkend="nse-parallelism-base"/>.) The second return value contains
        an error object that caused the termination of the worker thread, or
        <literal>nil</literal> if no error was thrown. This object is typically
        a string, like most Lua errors. However, any Lua type can
        be an error object, even <literal>nil</literal>.  Therefore
        inspect the error object, the second return value, only if the status
        of the worker is <literal>"dead"</literal>.
      </para>
      <para>
        NSE discards all return values from the main function when the worker
        thread finishes execution. You should communicate with your worker
        through the use of <literal>main</literal> function parameters,
        upvalues, or function environments. See
        <xref linkend="nse-worker-example" xrefstyle="select: label nopage"/>
        for an example.
      </para>
      <para>
        Finally, when using worker threads you should always use condition
        variables or mutexes to coordinate them. Nmap is single-threaded so
        there are no memory synchronization issues to worry about; but there
        <emphasis>is</emphasis> contention for resources.
        These resources include usually network bandwidth and sockets.
        Condition variables are also useful if the work for any
        single thread is dynamic. For example, a web server spider script with
        a pool of workers will initially have a single root HTML document.
        Following the retrieval of the root document, the set of resources to
        be retrieved (the worker's work) may become very large as each new
        document adds new URLs to fetch.
      </para>
      <example id="nse-worker-example">
        <title>Worker threads</title>
<programlisting>
local requests = {"/", "/index.html", --[[ long list of objects ]]}

function thread_main (host, port, responses, ...)
  local condvar = nmap.condvar(responses);
  local what = {n = select("#", ...), ...};
  local allReqs = nil;
  for i = 1, what.n do
    allReqs = http.pGet(host, port, what[i], nil, nil, allReqs);
  end
  local p = assert(http.pipeline(host, port, allReqs));
  for i, response in ipairs(p) do responses[#responses+1] = response end
  condvar "signal";
end

function many_requests (host, port)
  local threads = {};
  local responses = {};
  local condvar = nmap.condvar(responses);
  local i = 1;
  repeat
    local j = math.min(i+10, #requests);
    local co = stdnse.new_thread(thread_main, host, port, responses,
        unpack(requests, i, j));
    threads[co] = true;
    i = j+1;
  until i > #requests;
  repeat
    for thread in pairs(threads) do
      if coroutine.status(thread) == "dead" then threads[thread] = nil end
    end
    if ( next(threads) ) then
      condvar "wait"
    end
  until next(threads) == nil;
  return responses;
end
</programlisting>
      </example>
      <para>
        For brevity, this example omits typical behavior of a traditional web
        spider. The requests table is assumed to contain enough objects
        to warrant the use of worker threads.  The code in this example
        dispatches a new thread with as many as 11 relative URLs.
        Worker threads are cheap, so don't be afraid to create a lot of them.
        After dispatching all these threads, the code waits on a
        condition variable until every thread
        has finished, then finally return the responses table.
      </para>
      <para>
        You may have noticed that we did not use the status function returned
        by <literal>stdnse.new_thread</literal>. You will typically use this
        for debugging or if your program must stop based on the error thrown by
        one of your worker threads. Our simple example did not require this but
        a more fault-tolerant library may.
      </para>
    </sect2>
    <sect2 id="nse-parallelism-mutex">
      <title>Mutexes</title>
      <indexterm><primary>threads in NSE</primary></indexterm>
      <indexterm><primary>mutexes in NSE</primary></indexterm>
      <para>
        Recall from the beginning of this section that each script execution
        thread (e.g. <literal>ftp-anon</literal> running against an FTP server
        on a target host) yields to other scripts whenever it makes a call
        on network objects (sending or receiving data). Some scripts require
        finer concurrency control over thread execution. An example is the
        <literal>whois</literal><indexterm><literal>whois</literal> script</indexterm>
        script which queries
        whois<indexterm><primary>whois</primary></indexterm> servers for each
        target IP address. Because many concurrent queries can get your
        IP banned for abuse, and because a single query may
        return the same information another instance of the script is about to
        request, it is useful to have other threads pause while one thread
        performs a query.
      </para>
      <para>
        To solve this problem, NSE includes a <literal>mutex</literal> function
        which provides a <ulink
        url="http://en.wikipedia.org/wiki/Mutual_exclusion">mutex</ulink>
        (mutual exclusion object) usable by scripts. The mutex allows for only
        one thread to be working on an object at a time. Competing threads
        waiting to
        work on this object are put in the waiting queue until they can get a
        <quote>lock</quote> on the mutex. A solution for the <literal>whois</literal>
        problem above is to have each thread block on a mutex using a common
        string, ensuring that only one thread at a time is querying a server.
        When finished querying the remote servers, the thread can store
        results in the NSE registry and unlock the mutex. Other scripts waiting
        to query the remote server can then obtain a lock, check for the cache
        for a usable result from a previous query, make their own queries, and
        unlock the mutex.  This is a good example of serializing access to a
        remote resource.
      </para>

      <para>
        The first step in using a mutex is to create one with a call to
        <literal>nmap.mutex</literal>.
      </para>
<programlisting>
mutexfn = nmap.mutex(object)
</programlisting>

      <para>
        The <literal>mutexfn</literal> returned is a function which works as a
        mutex for the <literal>object</literal> passed in.  This object can be
        any <ulink role="hidepdf"
        url="https://lua.org/manual/5.4/manual.html#2.1">Lua data
        type</ulink> except <literal>nil</literal>,
        Boolean, and number.  The
        returned function allows you to lock, try to lock, and release the
        mutex. Its sole argument must be one of the
        following:
      </para>

      <variablelist>
        <varlistentry>
          <term><literal>"lock"</literal></term>
          <listitem>
            <para>
              Makes a blocking lock on the mutex. If the mutex is busy (another
              thread has a lock on it), then the thread will yield and
              wait. The function returns with the mutex locked.
            </para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><literal>"trylock"</literal></term>
          <listitem>
            <para>
              Makes a non-blocking lock on the mutex. If the mutex is busy then
              it immediately returns with a return value of
              <literal>false</literal>. Otherwise, locks the mutex and
              returns <literal>true</literal>.
            </para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><literal>"done"</literal></term>
          <listitem>
            <para>
              Releases the mutex and allows another thread to lock it.  If the
              thread does not have a lock on the mutex, an error will be
              raised.
            </para>
          </listitem>
        </varlistentry>

        <varlistentry>
          <term><literal>"running"</literal></term>
          <listitem>
            <para>
              Returns the thread locked on the mutex or <literal>nil</literal>
              if the mutex is not locked. This should only be used for
              debugging as it interferes with garbage collection of finished
              threads.
            </para>
          </listitem>
        </varlistentry>
       </variablelist>

      <para>
        NSE maintains a weak reference to the mutex so other calls to
        <literal>nmap.mutex</literal> with the same object will return the same
        mutex function. However, if you discard your reference to the mutex
        then it may be collected and subsequent calls to
        <literal>nmap.mutex</literal> with the object will return a different
        function. Therefore save your mutex to a (local) variable
        that persists as long as you need it.
      </para>

       <para>
         A simple example of using the API is provided in <xref
         linkend="nse-mutex-handling" xrefstyle="select: label nopage"/>.  For
         real-life examples, read the
         <filename>asn-query</filename><indexterm><primary><filename>asn-query</filename> script</primary></indexterm>
         and
         <filename>whois</filename><indexterm><primary><filename>whois</filename> script</primary></indexterm>
         scripts in the Nmap
         distribution.
       </para>

      <example id="nse-mutex-handling">
        <title>Mutex manipulation</title>
<programlisting>
local mutex = nmap.mutex("My Script's Unique ID");
function action(host, port)
  mutex "lock";
  -- Do critical section work - only one thread at a time executes this.
  mutex "done";
  return script_output;
end
</programlisting>
      </example>
    </sect2>
    <sect2 id="nse-parallelism-condvar">
      <title>Condition Variables</title>
      <para>
        Condition variables arose out of a need to coordinate with worker
        threads created by the <literal>stdnse.new_thread</literal>
        function.  A condition variable allows many threads to wait on
        one object, and one or all of them to be awakened when some condition
        is met. Said differently, multiple threads may unconditionally
        <literal>block</literal> on the condition variable by
        <emphasis>waiting</emphasis>. Other threads may use the condition
        variable to wake up the waiting threads.
      </para>

      <para>
        For example, consider the earlier <xref
        linkend="nse-worker-example"/>.  Until all
        the workers finish, the controller thread must sleep. Note that we cannot
        <literal>poll</literal> for results like in a traditional operating
        system thread because NSE does not preempt Lua threads. Instead,
        we use a condition variable that the controller thread
        <emphasis>waits</emphasis> on until awakened by a worker. The controller
        will continually wait until all workers have terminated.
      </para>

      <para>
        The first step in using a condition variable is to create one
        with a call to <literal>nmap.condvar</literal>.
      </para>

      <para><literal>condvarfn = nmap.condvar(object)</literal></para>

      <para>
        The semantics for condition variables are similar to those of mutexes.  The
        <literal>condvarfn</literal> returned is a function which works as a
        condition variable for the <literal>object</literal> passed in. This
        object can be any <ulink role="hidepdf"
        url="https://lua.org/manual/5.4/manual.html#2.1">Lua data
        type</ulink> except <literal>nil</literal>,
        Boolean, and number.  The
        returned function allows you to wait, signal, and broadcast on the
        condition variable.  Its sole argument must be one of the
        following:
      </para>

      <variablelist>
        <varlistentry>
          <term><literal>"wait"</literal></term>
          <listitem>
            <para>
              Wait on the condition variable. This adds the current thread to the
              waiting queue for the condition variable. It will resume
              execution when another thread signals or broadcasts on the
              condition variable.
            </para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term><literal>"signal"</literal></term>
          <listitem>
            <para>
              Signal the condition variable. One of the threads in the condition
              variable's waiting queue will be resumed.
            </para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term><literal>"broadcast"</literal></term>
          <listitem>
            <para>
              Resume all the threads in the condition variable's waiting queue.
            </para>
          </listitem>
        </varlistentry>
      </variablelist>

      <para>
        Like with mutexes, NSE maintains a weak reference to the condition
        variable so other calls to <literal>nmap.condvar</literal> with the
        same object will return the same function. However, if you discard your reference to the condition variable then
        it may be collected and subsequent calls to
        <literal>nmap.condvar</literal> with the object will return a different
        function. Therefore save your condition
        variable to a (local) variable that persists as long as you need it.
      </para>

      <para>
        When using condition variables, it is important to check the predicate
        before and after waiting. A predicate is a test on whether to continue
        doing work within a worker or controller thread. For worker
        threads, this will at the very least include a test to see if the
        controller is still alive. You do not want to continue doing work
        when there's no thread to use your results. A typical test before waiting
        may be: Check whether the controller is still running; if not, then quit.
        Check if is work to be done; if not, then wait.
      </para>

      <para>
        A thread waiting on a condition variable may be resumed without any
        other thread having called <literal>"signal"</literal> or
        <literal>"broadcast"</literal> on the condition variable (a spurious
        wakeup).
        The usual, but not only, reason that this may happen
        is the termination of one of the threads using the condition variable. This
        is an important guarantee NSE makes that allows you to avoid deadlock
        where a worker or controller waits for a thread to wake them up that ended
        without signaling the condition variable.
      </para>
    </sect2>
    <sect2 id="nse-parallelism-cm">
      <title>Collaborative Multithreading</title>
      <para>
        One of Lua's least-known features is collaborative multithreading
        through <emphasis>coroutines</emphasis>. A coroutine provides an
        independent execution stack that can be yielded and resumed.
        The standard <literal>coroutine</literal> table provides access to the
        creation and manipulation of coroutines.      Lua's online first
        edition of <ulink url="https://lua.org/pil/"><citetitle>Programming in
        Lua</citetitle></ulink> contains an excellent introduction to
        coroutines. What follows is an overview of the
        use of coroutines here for completeness, but this is no replacement for
        the definitive reference.
      </para>

      <para>
        We have mentioned coroutines throughout this section as
        <emphasis>threads</emphasis>. This is the <emphasis>type</emphasis>
        (<literal>"thread"</literal>) of a coroutine in Lua. They are not the
        preemptive threads that programmers may be expecting. Lua threads
        provide the basis for parallel scripting but only one thread is ever
        running at a time.
      </para>

      <para>
        A Lua function executes on top of a Lua
        thread. The thread maintains a stack of active
        functions, local variables, and the current instruction pointer. We can switch
        between coroutines by explicitly yielding the
        running thread. The coroutine which resumed the
        yielded thread resumes operation.
        <xref linkend="nse-cm-coroutines" xrefstyle="select: label nopage"/>
        shows a brief use of coroutines to print numbers.
      </para>
      <example id="nse-cm-coroutines">
        <title>Basic Coroutine Use</title>
<programlisting>
local function main ()
  coroutine.yield(1)
  coroutine.yield(2)
  coroutine.yield(3)
end
local co = coroutine.create(main)
for i = 1, 3 do
  print(coroutine.resume(co))
end
--> true    1
--> true    2
--> true    3
</programlisting>
      </example>

      <para>
        Coroutines are the facility
        that enables NSE to run scripts in parallel. All scripts are
        run as coroutines that yield whenever they make a blocking socket
        function call. This enables NSE to run other scripts and later resume
        the blocked script when its I/O operation has completed.
      </para>

      <para>
        Sometimes coroutines are the best
        tool for a job within a single script. One common use in socket programming is filtering
        data. You may write a function that generates all the links from an
        HTML document. An iterator using <literal>string.gmatch</literal>
        can catches only a single pattern. Because some complex matches may take
        many different Lua patterns, it is more appropriate to use a
        coroutine.
        <xref linkend="nse-cm-links" xrefstyle="select: label nopage"/>
        shows how to do this.
      </para>

      <example id="nse-cm-links">
        <title>Link Generator</title>
<programlisting>
function links (html_document)
  local function generate ()
    for m in string.gmatch(html_document, "url%((.-)%)") do
      coroutine.yield(m) -- css url
    end
    for m in string.gmatch(html_document, "href%s*=%s*\"(.-)\"") do
      coroutine.yield(m) -- anchor link
    end
    for m in string.gmatch(html_document, "src%s*=%s*\"(.-)\"") do
      coroutine.yield(m) -- img source
    end
  end
  return coroutine.wrap(generate)
end

function action (host, port)
  -- ... get HTML document and store in html_document local
  for link in links(html_document) do
    links[#links+1] = link; -- store it
  end
  -- ...
end
</programlisting>
      </example>

      <sect3 id="nse-parallelism-base">
        <title>The base thread</title>
        <para>
          Because scripts may use coroutines for their own multithreading,
          it is important to be able to identify the <emphasis>owner</emphasis>
          of a resource or to establish whether the script is still alive.
          NSE provides the function <literal>stdnse.base</literal> for this
          purpose.
        </para>
        <para>
          Particularly when writing a library that attributes
          ownership of a cache or socket to a script, you can use the
          base thread to establish whether the script is still running.
          <literal>coroutine.status</literal> on the base thread will give
          the current state of the script. In cases where the script is
          <literal>"dead"</literal>, you will want to release the resource.
          Be careful with keeping references to these threads; NSE may
          discard a script even though it has not finished executing. The
          thread will still report a status of <literal>"suspended"</literal>.
          You should keep a weak reference to the thread in these cases
          so that it may be collected.
        </para>
      </sect3>
    </sect2>
  </sect1>

  <sect1 id="nse-vscan">
    <title>Version Detection Using NSE</title>
    <indexterm class="startofrange" id="nse-sample-indexterm"><primary>Nmap Scripting Engine (NSE)</primary><secondary>sample scripts</secondary></indexterm>
    <indexterm><primary>version detection</primary><secondary>using NSE</secondary></indexterm>

    <para>
    The version detection system built into Nmap was designed to
    efficiently recognize the vast majority of protocols with a simple
    probe and pattern matching syntax.  Some protocols require more
    complex communication than version detection can handle.  A
    generalized scripting language as provided by NSE is perfect for
    these tough cases.
    </para>

    <para>
    NSE's <literal>version</literal><indexterm><primary><varname>version</varname> script category</primary></indexterm>
    category contains scripts that enhance standard version
    detection. Scripts in this category are run whenever you request
    version detection with <option>-sV</option>; you don't need to use
    <option>-sC</option> to run these. This cuts
    the other way too: if you use <option>-sC</option>, you won't get
    <literal>version</literal> scripts unless you also use
    <option>-sV</option>.
    </para>

    <para>
    One protocol which we were unable to detect with normal version
    detection is Skype<indexterm><primary>Skype</primary></indexterm>
    version 2.  The protocol was likely designed to
    frustrate detection out of a fear that telecom-affiliated Internet
    service providers might consider Skype competition and interfere
    with the traffic.  Yet we did find one way to detect it.  If Skype
    receives an HTTP GET request, it pretends to be a web server and
    returns a 404 error. But for other requests, it sends back
    a chunk of random-looking data. Proper identification requires
    sending two probes and comparing the two responses&mdash;an ideal
    task for NSE.  The simple NSE script which accomplishes this is
    shown in
    <xref linkend="nse-skypev2-script" xrefstyle="select: label nopage"/>.
    </para>

<example id="nse-skypev2-script">
      <title>A typical version detection script (Skype version 2 detection)</title>
<indexterm><primary><filename>skypev2-version</filename> script</primary></indexterm>
<programlisting>
description = [[
Detects the Skype version 2 service.
]]<indexterm><primary sortas="description script variable">&ldquo;<varname>description</varname>&rdquo; script variable</primary></indexterm>

---
-- @output
-- PORT   STATE SERVICE VERSION
-- 80/tcp open  skype2  Skype
 
author = "Brandon Enright"<indexterm><primary>Enright, Brandon</primary></indexterm><indexterm><primary sortas="author script variable">&ldquo;<varname>author</varname>&rdquo; script variable</primary></indexterm>
license = "Same as Nmap--See https://nmap.org/book/man-legal.html"<indexterm><primary sortas="license script variable">&ldquo;<varname>license</varname>&rdquo; script variable</primary></indexterm>
categories = {"version"}

require "comm"
require "shortport"

portrule = function(host, port)
        return (port.number == 80 or port.number == 443 or
                port.service == nil or port.service == "" or
                port.service == "unknown")
               and port.protocol == "tcp" and port.state == "open"
               and port.service ~= "http" and port.service ~= "ssl/http"
               and not(shortport.port_is_excluded(port.number,port.protocol))
end

action = function(host, port)
        local status, result = comm.exchange(host, port,
                "GET / HTTP/1.0\r\n\r\n", {bytes=26, proto=port.protocol})
        if (not status) then
                return
        end
        if (result ~= "HTTP/1.0 404 Not Found\r\n\r\n") then
                return
        end
        -- So far so good, now see if we get random data for another request
        status, result = comm.exchange(host, port,
                "random data\r\n\r\n", {bytes=15, proto=port.protocol})

        if (not status) then
                return
        end
        if string.match(result, "[^%s!-~].*[^%s!-~].*[^%s!-~]") then
                -- Detected
                port.version.name = "skype2"
                port.version.product = "Skype"
                nmap.set_port_version(host, port)
                return  
        end
        return
end
</programlisting>
</example>

    <para>
    If the script detects Skype, it augments its <varname>port</varname>
    table with now-known <varname>name</varname> and
    <varname>product</varname> fields. It then sends this new
    information to Nmap by calling
    <function>nmap.set_port_version</function>. Several other version
    fields are available to be set if they are known, but in this case
    we only have the name and product. For the full list of version
    fields, refer to the <ulink role="hidepdf" url="https://nmap.org/nsedoc/modules/nmap.html"><function>nmap.set_port_version</function> documentation</ulink>.
    </para>

    <para>
    Notice that this script does nothing unless it detects the protocol.
    A script shouldn't
    produce output (other than debug output) just to say it didn't learn
    anything.
    </para>
  </sect1>

  <sect1 id="nse-example-scripts">
    <title>Example Script: <filename>finger</filename></title>
      <indexterm><primary><literal>finger</literal> script</primary></indexterm>

	  <para>The <filename>finger</filename> script is a perfect
      example of a short and simple NSE script.
	  </para>

 <para>First the information fields are assigned.
A detailed description of what the script
actually does goes in the <literal>description</literal> field.</para>
<programlisting>
description = [[
Attempts to get a list of usernames via the finger service.
]]<indexterm><primary sortas="description script variable">&ldquo;<varname>description</varname>&rdquo; script variable</primary></indexterm>

author = "Eddie Bell"<indexterm><primary>Bell, Eddie</primary></indexterm><indexterm><primary sortas="author script variable">&ldquo;<varname>author</varname>&rdquo; script variable</primary></indexterm>

license = "Same as Nmap--See https://nmap.org/book/man-legal.html"<indexterm><primary sortas="license script variable">&ldquo;<varname>license</varname>&rdquo; script variable</primary></indexterm>
</programlisting>

<para>The <literal>categories</literal> field is a table
containing all the categories the script belongs to. These are used for
script selection with the <option>--script</option> option:</para>

<programlisting>
categories = {"default", "discovery", "safe"}<indexterm><primary sortas="categories script variable">&ldquo;<varname>categories</varname>&rdquo; script variable</primary></indexterm>
</programlisting>

<para>
Every good script comes with a sample of its output in an NSEDoc comment.
</para>

<programlisting>
---
-- @output
-- PORT   STATE SERVICE
-- 79/tcp open  finger
-- | finger:
-- | Welcome to Linux version 2.6.31.12-0.2-default at linux-pb94.site !
-- |  01:14am  up  18:54,  4 users,  load average: 0.14, 0.08, 0.01
-- |
-- | Login      Name                  Tty      Idle  Login Time   Where
-- | Gutek      Ange Gutek           *:0          -     Wed 06:19 console
-- | Gutek      Ange Gutek            pts/1   18:54     Wed 06:20
-- | Gutek      Ange Gutek           *pts/0       -     Thu 00:41
-- |_Gutek      Ange Gutek           *pts/4       3     Thu 01:06
</programlisting>

<para>You can use the facilities provided by the nselib (<xref
linkend="nse-library"/>) with <literal>require</literal>.  Here
we want to use common communication functions and shorter port rules:</para>

<programlisting>
require "comm"
require "shortport"
</programlisting>

<para>We want to run the script against the finger service.  So we
test whether it is using the well-known finger port (<literal>79/tcp</literal>), or
whether the service is named <quote>finger</quote> based on version
detection results or in the port number's listing
in <filename>nmap-services</filename>:</para>

<programlisting>
portrule = shortport.port_or_service(79, "finger")<indexterm><primary sortas="portrule script variable">&ldquo;<varname>portrule</varname>&rdquo; script variable</primary></indexterm>
</programlisting>

<para>First, the script uses <function>nmap.new_try</function> to
create an exception handler that will quit the script in case of an
error. Next, it passes control to <function>comm.exchange</function>,
which handles the network transaction. Here we have asked to wait in the communication exchange until we receive at least 100 lines, wait at least 5 seconds, or until the remote side closes the connection.  Any errors are handled by the
<function>try</function> exception handler. The script returns a string
if the call to <literal>comm.exchange()</literal> was successful.</para>

<programlisting>
action = function(host, port)
	local try = nmap.new_try()

	return try(comm.exchange(host, port, "\r\n",
        	{lines=100, proto=port.protocol, timeout=5000}))
end
</programlisting>
    <indexterm class="endofrange" startref="nse-sample-indexterm"/>
  </sect1>
  <sect1 id="nse-implementation">
    <title>Implementation Details</title>
    <indexterm><primary>Nmap Scripting Engine (NSE)</primary><secondary>implementation</secondary></indexterm>
	<para>
	Now it is time to explore the NSE implementation details in
	depth.  Understanding how NSE works is useful for designing
	efficient scripts and libraries.  The canonical reference to
	the NSE implementation is the source code, but this section
	provides an overview of key details.  It should be valuable to
	folks trying to understand and extend the NSE source code, as
	well as to script authors who want to better understand how
	their scripts are executed.
	</para>
    <sect2 id="nse-implementation-init">
      <title>Initialization Phase</title>
      <para>
      NSE is initialized before any scanning when Nmap first starts, by the
      <literal>open_nse</literal> function. <literal>open_nse</literal>
      creates a fresh Lua state that will persist across
      host groups,<indexterm><primary>host groups</primary><secondary>persistence of NSE through</secondary></indexterm>
      until the program exits.
      It then loads the standard Lua libraries and compiled NSE libraries.
      The standard Lua libraries are
      documented in the <ulink
      url="https://lua.org/manual/5.4/manual.html">Lua Reference
      Manual</ulink>. The standard Lua libraries available to NSE are
      <literal>debug</literal>,
      <literal>io</literal>,
      <literal>math</literal>,
      <literal>os</literal>,
      <literal>package</literal>,
      <literal>string</literal>, and
      <literal>table</literal>.
      Compiled NSE libraries are those that are defined in a C++ file instead
      of a Lua file. They include
      <literal>nmap</literal>,
      <literal>pcre</literal>,
      <literal>db</literal>,
      <literal>lpeg</literal>,
      <literal>debug</literal>,
      <literal>zlib</literal>,
      <literal>libssh2</literal>, and
      <literal>openssl</literal> (if available).
      </para>
 
      <para>
        After loading the basic libraries, <literal>open_nse</literal> loads
        the file
        <literal>nse_main.lua</literal>. The NSE core is in this
        file&mdash;Lua code manages scripts and sets up the appropriate
        environment. In this situation Lua really shines as a glue language.
        C++ is used to provide the network framework and low-level libraries.
        Lua is used to structure data, determine which scripts to load,
        and schedule and execute scripts.
      </para>

      <para>
        <literal>nse_main.lua</literal> sets up the Lua
        environment to be ready for script scanning later on. It
        loads all the scripts the user has chosen and returns a function
        that does the actual script scanning to
        <literal>open_nse</literal>.
      </para>

      <para>
        The <literal>nselib</literal> directory is added to the Lua path to
        give scripts access to the standard NSE library. NSE loads replacements
        for the standard coroutine functions so that yields initiated by NSE are
        caught and propagated back to the NSE scheduler.
      </para>

      <para>
        <literal>nse_main.lua</literal> next defines classes and functions
        to be used during setup. The script arguments
        (<literal>--script-args</literal>) are loaded into
        <literal>nmap.registry.args</literal>.
        A script database is created if one doesn't already exist or if this
        was requested with <option>--script-updatedb</option>.
      </para>

      <para>
        Finally, the scripts listed on the command line are loaded.
        The <literal>get_chosen_scripts</literal> function works to find
        chosen scripts by comparing categories, filenames, and directory names.
        The scripts are loaded into memory for later use.
        <literal>get_chosen_scripts</literal> works by transforming the
        argument to <literal>--script</literal> into a block of Lua code and
        then executing it. (This is how the <literal>and</literal>,
        <literal>or</literal>, and <literal>not</literal> operators are
        supported.) Any specifications that don't directly match a category or
        a filename from
        <filename>script.db</filename><indexterm><primary><filename>script.db</filename></primary></indexterm>
        are checked against file and directory names. If the specification is a
        regular file, it's loaded. If a directory, all the
        <literal>*.nse</literal> files within it are loaded. Otherwise, the
        engine raises an error.
      </para>

      <para>
        <literal>get_chosen_scripts</literal> finishes by arranging the
        selected scripts according to their dependencies (see
        <xref linkend="nse-format-dependencies"/>).
        Scripts that have no dependencies are in runlevel 1. Scripts that
        directly depend on these are in runlevel 2, and so on.
        When a script scan is run, each runlevel is run separately and in
        order.
      </para>

      <para>
        <literal>nse_main.lua</literal> defines two classes:
        <literal>Script</literal> and <literal>Thread</literal>. These classes
        are the objects that represent NSE scripts and their script threads.
        When a script is loaded, <literal>Script.new</literal>
        creates a new Script object.  The script file is loaded into Lua
        and saved for later use. These classes and their methods are intended
        to encapsulate the data needed for each script and its threads.
        <literal>Script.new</literal> also contains sanity checks to ensure that the
        script has required fields such as the <literal>action</literal>
        function.
      </para>
      </sect2>

      <sect2 id="nse-implementation-scan">
        <title>Script Scanning</title>
        <para>
          When NSE runs a script scan, <literal>script_scan</literal> is called
          in <literal>nse_main.cc</literal>.  Since there are three script scan
          phases, <literal>script_scan</literal> accepts two arguments, a 
          script scan type which can be one of these values:
          <literal>SCRIPT_PRE_SCAN</literal> (Script Pre-scanning phase) or
          <literal>SCRIPT_SCAN</literal> (Script scanning phase) or
          <literal>SCRIPT_POST_SCAN</literal> (Script Post-scanning phase),
          and a second argument which is a list of targets to scan if
          the script scan phase is <literal>SCRIPT_SCAN</literal>.
          These targets will be passed to the <literal>nse_main.lua</literal>
          main function for scanning.
        </para>

        <para>
          The main function for a script scan generates a number of script
          threads based on whether the <literal>rule</literal> function 
          returns true. The generated threads are stored in a list of runlevel
          lists. Each runlevel list of threads is passed separately to the
          <literal>run</literal> function. The <literal>run</literal> function
          is the main worker function for NSE where all the magic happens.
        </para>

        <para>
          The <literal>run</literal> function's purpose is run all the threads
          in a runlevel until they all finish. Before doing this however,
          <literal>run</literal> redefines some Lua registry values that
          help C code function. One such function,
          <literal>_R[WAITING_TO_RUNNING]</literal>, allows the network library
          binding written in C to move a thread from the waiting queue to the
          running queue. Scripts are run until the
          running and waiting queues are both empty. Threads that yield are
          moved to the waiting queue; threads that are ready to continue
          are moved back to the running queue. The cycle continues until
          the thread quits or ends in error. Along with the waiting and running
          queues, there is a pending queue. It serves as a temporary location
          for threads moving from the waiting queue to the running queue before
          a new iteration of the running queue begins.
        </para>

      </sect2>
  </sect1>
<indexterm class="endofrange" startref="nse-indexterm"/>
</chapter>