doc/source/rainerscript/queue_parameters.rst - main - githubmirror/rsyslog

doc/source/rainerscript/queue_parameters.rst
772 строки27 KB
Rainer Gerhards
doc: describe segmented disk-assisted queues
14 июл 2026, 21:57
14 июл 2026, 21:57e4559fa
.. meta::
   :description: Reference for rsyslog queue parameters, including memory, disk, and disk-assisted queue behavior.
   :keywords: rsyslog, queue parameters, disk queue, segmented disk, disk assisted, RainerScript, YAML

************************
General Queue Parameters
************************

===========================  ===========================================================================
**Authors:**                  `Rainer Gerhards <https://rainer.gerhards.net/>`_ <rgerhards@adiscon.com>;
===========================  ===========================================================================


Usage
=====

Queue parameters can be used together with the following statements:

- :doc:`action() <../configuration/actions>`
- ruleset()
- main\_queue()

Queues need to be configured in the action or ruleset it should affect.
If nothing is configured, default values will be used. Thus, the default
ruleset has only the default main queue. Specific Action queues are not
set up by default.

To fully understand queue parameters and how they interact, be sure to
read the :doc:`queues <../concepts/queues>` documentation.

.. summary-start

Queue parameters configure main, ruleset, and action queue capacity,
concurrency, persistence, disk-assisted behavior, and shutdown policy.

.. summary-end


Configuration Parameters
========================

.. note::

   As with other configuration objects, parameters for this
   object are case-insensitive.


queue.filename
--------------

.. csv-table::
   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
   :widths: auto
   :class: parameter-table

   "word", "none", "no", "``$ActionQueueFileName``"

File name to be used for the queue files. If specified, this parameter
enables disk-assisted queue functionality. If *not* specified,
the queue will operate without saving the queue to disk, either
during its operation or when shut down. See the separate
``queue.saveonshutdown`` parameter to configure that option.
Please note that this is actually just the file name. A directory
can NOT be specified in this parameter. If the files shall be
created in a specific directory, specify ``queue.spoolDirectory``
for this. The filename is used to build to complete path for queue
files.

For the experimental ``segmentedDisk`` queue type, ``queue.filename`` is
required. Configuration fails if it is omitted.


queue.diskQueueType
-------------------

.. csv-table::
   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
   :widths: auto
   :class: parameter-table

   "word", "auto", "no", "none"

Selects the disk engine used by a disk-assisted ``FixedArray`` or
``LinkedList`` queue. It does not change the memory-first behavior: no disk
store is created until the queue spills past its high watermark.

Accepted values are:

* ``auto``: use ``segmentedDisk`` for a fresh queue. Existing classic ``.qi``
  or numeric queue files select ``Disk`` and emit a warning. Existing
  segmented state selects ``segmentedDisk``.
* ``disk``: require the classic disk queue engine.
* ``segmentedDisk``: require the segmented engine.

The selection is recorded in a small versioned
``<queue.filename>.da-engine`` file. Malformed markers, mixed classic and
segmented stores, and an explicit engine that conflicts with existing data
fail queue initialization. ``auto`` selects classic ``Disk`` when encryption
or a classic-only corruption mode is configured. Explicit ``segmentedDisk``
rejects those unsupported options.

For a fresh segmented selection, neither the marker nor the ``.segq`` store is
created until the first spill. The marker is made durable before the store is
materialized. Idle cleanup removes the store but retains that marker.

This parameter has no legacy directive alias. Outside a disk-assisted memory
queue it is reported as dirty configuration and ignored.


queue.diskQueueAutoUpgrade
--------------------------

.. csv-table::
   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
   :widths: auto
   :class: parameter-table

   "binary", "off", "no", "none"

Applies only when ``queue.diskQueueType="auto"``. With the default ``off``, an
automatically selected classic engine remains sticky after its backlog drains.
With ``on``, the next clean restart after that drain selects an unmaterialized
segmented child. Records are never converted and engines never switch while a
process is running.


queue.diskQueueIdleTimeout
--------------------------

.. csv-table::
   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
   :widths: auto
   :class: parameter-table

   "integer", "60000", "no", "none"

Controls how long, in milliseconds, an empty segmented disk-assisted child
remains materialized. After the complete interval passes without new producer
or disk-transfer activity, rsyslog durably removes the empty segmented store
and lets its final disk-consumer worker terminate. The engine marker remains.
A later spill recreates both the store and workers transparently.

``0`` requests cleanup immediately after a safe drain. ``-1`` disables runtime
dematerialization. Recovery work, in-flight batches, retries, or uncertain
cleanup state postpone cleanup. This timeout is independent of
``queue.timeoutWorkerThreadShutdown`` and applies only to an effective
segmented disk-assisted child. A nondefault value with explicit classic
``disk`` is dirty configuration and is ignored.


queue.spoolDirectory
--------------------

.. csv-table::
   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
   :widths: auto
   :class: parameter-table

   "word", "none", "no", "none"

This is the directory into which queue files will be stored. Note
that the directory must exist, it is NOT automatically created by
rsyslog. If no spoolDirectory is specified, the work directory is
used.

The experimental ``segmentedDisk`` queue type requires either this parameter
or a configured global work directory. The selected directory must already
exist.


queue.size
----------

.. csv-table::
   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
   :widths: auto
   :class: parameter-table

   "integer", "1000/50000", "no", "``$ActionQueueSize``"

This is the maximum size of the queue in number of messages. Note
that setting the queue size to very small values (roughly below 100
messages) is not supported and can lead to unpredictable results.
For more information on the current status of this restriction see
the `rsyslog FAQ: "lower bound for queue
sizes" <http://www.rsyslog.com/lower-bound-for-queue-sizes/>`_.

The default depends on queue type and rsyslog version, if you need
a specific value, please specify it. Otherwise rsyslog selects what
it considers appropriate for the version in question. In rsyslog
rsyslog 8.30.0, for example, ruleset queues have a default size
of 50000 and action queues which are configured to be non-direct
have a size of 1000.


queue.dequeueBatchSize
----------------------

.. csv-table::
   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
   :widths: auto
   :class: parameter-table

   "integer", "128/1024", "no", "``$ActionQueueDequeueBatchSize``"

Specifies the maximum batch size for dequeue operations. This setting affects performance.
As a rule of thumb, larger batch sizes (up to a environment-induced upper limit)
provide better performance. For the average system, there usually should be no need
to adjust batch sizes as the defaults are sufficient. The default for ruleset queues
is 1024 and for action queues 128.

Note that this only specifies the **maximum** batch size. Batches will be slower if
rsyslog does not have as many messages inside the queue at time of dequeuing it.
If you want to set a minimum Batch size as well, you can use `queue.minDequeueBatchSize`.


queue.minDequeueBatchSize
-------------------------

.. csv-table::
   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
   :widths: auto
   :class: parameter-table

   "integer", "0", "no", "none"

Specifies the **minimum** batch size for dequeue operations. This setting is especially
useful with outputs like ElasticSearch or ClickHouse, where you want to limit the
number of HTTP requests. With this setting, the queue engine waits up to
`queue.minDequeueBatchSize.timeout` milliseconds when there are fewer messages
currently queued. Note that the minimum batch size cannot be larger than the
configured maximum batch size. If so, it is automatically adjusted to
match the maximum. So if in doubt, you need to specify both parameters.


queue.minDequeueBatchSize.timeout
---------------------------------

.. csv-table::
   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
   :widths: auto
   :class: parameter-table

   "integer", "1000", "no", "none"

This parameter is only meaningful if use together with `queue.minDequeueBatchSize`,
otherwise it is ignored. It specifies the amount of time (in milliseconds) rsyslogs
waits for new
messages so that the minimum batch size can be reached. After this period, the
batch is processed, *even if it is below minimum size*. This capability exists to
prevent having messages stalled in an incomplete batch just because no new
messages arrive. We would expect that it usually makes little sense to set it
to higher than 60.000 (60 seconds), but this is permitted. Just be warned that
this potentially delays log processing for that long.


queue.maxDiskSpace
------------------

.. csv-table::
   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
   :widths: auto
   :class: parameter-table

   "integer", "0", "no", "``$ActionQueueMaxDiskSpace``"

The maximum size that all queue files together will use on disk. Note
that the actual size may be slightly larger than the configured max,
as rsyslog never writes partial queue records.


queue.highWatermark
-------------------

.. csv-table::
   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
   :widths: auto
   :class: parameter-table

   "integer", "90% of queue.size", "no", "``$ActionQueueHighWaterMark``"

This applies to disk-assisted queues, only. When the queue fills up
to this number of messages, the queue begins to spool messages to
disk. Please note that this should not happen as part of usual
processing, because disk queue mode is very considerably slower than
in-memory queue mode. Going to disk should be reserved for cases
where an output action destination is offline for some period.


queue.lowWatermark
------------------

.. csv-table::
   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
   :widths: auto
   :class: parameter-table

   "integer", "70% of queue.size", "no", "``$ActionQueueLowWaterMark``"

This applies to disk-assisted queues, only. When the high watermark is
reached, the queue will write data to disk. It does so until the low
watermark is reached, then the queue reverts back to in-memory mode.


queue.fullDelaymark
-------------------

.. csv-table::
   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
   :widths: auto
   :class: parameter-table

   "integer", "97% of queue.size", "no", "none"

Number of messages when the queue should block delayable messages.
Messages are NO LONGER PROCESSED until the queue has sufficient space
again. If a message is delayable depends on the input. For example,
messages received via imtcp are delayable (because TCP can push back),
but those received via imudp are not (as UDP does not permit a push back).
The intent behind this setting is to leave some space in an almost-full
queue for non-delayable messages, which would be lost if the queue runs
out of space. Please note that if you use a DA queue, setting the
fulldelaymark BELOW the highwatermark makes the queue never activate
disk mode for delayable inputs. So this is probably not what you want.


queue.lightDelayMark
--------------------

.. csv-table::
   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
   :widths: auto
   :class: parameter-table

   "integer", "70% of queue.size", "no", "none"

If this mark is reached the sender will be throttled if possible. The
main idea to do this is leave some space inside the queue for inputs
like UDP which cannot be throttled - and so any data arriving at
"queue full" would be discarded.

If the special value `0` is used, `queue.LightDelayMark` will be set
to the value of `queue.size`. This effectively **disables** light delay
functionality. This is useful if a queue is not used by non-delayable
inputs like UDP. The special value was introduced in rsyslog 8.1904.0
and is **not** available in earlier versions. There, you can achieve the
same result by setting `queue.LightDelayMark` to a very large value.


queue.discardMark
-----------------

.. csv-table::
   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
   :widths: auto
   :class: parameter-table

   "integer", "98% of queue.size", "no", "``$ActionQueueDiscardMark``"

Specifies the threshold at which rsyslog begins to discard less important
messages. To define which messages should be discarded use the
queue.discardseverity parameter.


queue.discardSeverity
---------------------

.. csv-table::
   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
   :widths: auto
   :class: parameter-table

   "integer", "8", "no", "``$ActionQueueDiscardSeverity``"

As soon as the threshold of the parameter queue.discardMark is reached
incoming as well as queued messages with a priority equal or lower than
specified will be erased. With the default no messages will be erased.
You have to specify a numeric severity value for this parameter.


queue.checkpointInterval
------------------------

.. csv-table::
   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
   :widths: auto
   :class: parameter-table

   "integer", "0", "no", "``$ActionQueueCheckpointInterval``"

Disk queues by default do not update housekeeping structures every time
the queue writes to disk. This is for performance reasons. In the event of failure,
data will still be lost (except when data is mangled via the file structures).
However, disk queues can be set to write bookkeeping information on checkpoints
(every n records), so that this can be made ultra-reliable, too. If the
checkpoint interval is set to one, no data can be lost, but the queue is
exceptionally slow.

For ``queue.type="segmentedDisk"``, the same default of ``0`` applies. A
positive interval counts records by which the contiguous commit frontier
advanced, not batches and not out-of-order worker completions. A larger value
reduces state-file writes and SSD wear but can replay more already-processed
records after a crash. Topology transitions, segment-ID reservation, deletion,
and clean shutdown can force additional state writes regardless of this
interval.


queue.syncqueuefiles
--------------------

.. csv-table::
   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
   :widths: auto
   :class: parameter-table

   "binary", "off", "no", "``$ActionQueueSyncQueueFiles``"

Disk-based queues can be made very reliable by issuing a (f)sync after each
write operation. This happens when you set the parameter to "on".
Activating this option has a performance penalty, so it should not
be turned on without a good reason. Note that the penalty also depends on
*queue.checkpointInterval* frequency.


queue.onCorruption
------------------

.. csv-table::
   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
   :widths: auto
   :class: parameter-table

   "word", "safe", "no", "none"

Controls how rsyslog handles disk queue corruption. Classic disk queues can
detect corruption during startup; segmented disk queues defer payload checks
until dequeue.

``segmentedDisk`` currently accepts only ``safe``. For that backend, safe mode
performs record-level salvage: corrupt framed records are skipped and scanning
continues at the next valid record. A crash-left active segment becomes a lazy
recovery segment while a newly reserved active segment accepts new messages.
The classic disk queue has separate corruption policies and behavior.

For the classic disk queue, supported values are:

* ``safe``: detect corruption, move queue files to a timestamped
  ``.bad`` directory, and start with a fresh disk queue.
* ``inMemory``: switch to a non-persistent in-memory queue for the current
  process lifetime.
* ``ignore``: skip the corruption verification logic and keep legacy startup
  behavior.

If rsyslog cannot safely quarantine corrupted files in ``safe`` mode (for
example because the recovery directory cannot be created or files cannot be
moved), it logs alert-level errors and switches to pure in-memory emergency
mode for safety.

The startup verification checks queue structure consistency (file sequence and
pointer continuity). It does **not** parse and validate each payload record
inside queue segment files.


queue.samplingInterval
----------------------

.. csv-table::
   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
   :widths: auto
   :class: parameter-table

   "integer", "0", "no", "none"

.. versionadded:: 8.23.0

This option allows queues to be populated by events produced at a specific interval.
It provides a way to sample data each N events, instead of processing all, in order
to reduce resources usage (disk, bandwidth...)


queue.type
----------

.. csv-table::
   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
   :widths: auto
   :class: parameter-table

   "word", "Direct", "no", "``$ActionQueueType``"

Specifies the type of queue that will be used. Possible options are
``FixedArray``, ``LinkedList``, ``Direct``, ``Disk``, or the experimental
``segmentedDisk`` pure-disk backend. For more information read the
documentation for :doc:`queues <../concepts/queues>`.

``segmentedDisk`` must be configured through modern object parameters; the
legacy ``$ActionQueueType`` directive does not accept it. It requires
``queue.filename`` plus either ``queue.spoolDirectory`` or a global work
directory.


queue.workerThreads
-------------------

.. csv-table::
   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
   :widths: auto
   :class: parameter-table

   "integer", "1", "no", "``$ActionQueueWorkerThreads``"

Specifies the maximum number of worker threads that can be run parallel.


queue.workerThreadMinimumMessages
---------------------------------

.. csv-table::
   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
   :widths: auto
   :class: parameter-table

   "integer", "queue.size/queue.workerthreads", "no", "``$ActionQueueWorkerThreadMinimumMessages``"

Specify the number of messages a worker thread is processing before another
worker thread is created. This number is limited by parameter queue.workerThreads.
For example if this parameter is set to 200 and in the queue are 201 messages a
second worker thread will be created.


queue.timeoutWorkerthreadShutdown
---------------------------------

.. csv-table::
   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
   :widths: auto
   :class: parameter-table

   "integer", "60000", "no", "``$ActionQueueTimeoutWorkerthreadShutdown``"

After starting a worker thread, it will process messages until there are no
messages for him to process. This parameter specifies the time the worker
thread has to be inactive before it times out.
The parameter must be specified in milliseconds. Which means the default of
60000 is 1 minute.


queue.timeoutshutdown
---------------------

.. csv-table::
   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
   :widths: auto
   :class: parameter-table

   "integer", "10/1500", "no", "``$ActionQueueTimeoutShutdown``"

If a queue that still contains messages is terminated it will wait the
specified time interval for the worker thread to finish.
The time is specified in milliseconds (1000ms is 1sec).
Default for action queues is 10, for ruleset queues it is 1500.


queue.timeoutActionCompletion
-----------------------------

.. csv-table::
   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
   :widths: auto
   :class: parameter-table

   "integer", "1000", "no", "``$ActionQueueTimeoutActionCompletion``"

When a queue is terminated, the timeout shutdown is over and there is
still data in the queue, the queue will finish the current data element
and then terminate. This parameter specifies the timeout for processing
this last element.
Parameter is specified in milliseconds (1000ms is 1sec).


queue.timeoutEnqueue
--------------------

.. csv-table::
   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
   :widths: auto
   :class: parameter-table

   "integer", "2000", "no", "``$ActionQueueTimeoutEnqueue``"

This timeout value is used when the queue is full. If rsyslog cannot
enqueue a message within the timeout period, the message is discarded.
Note that this is setting of last resort (assuming defaults are used
for the queue settings or proper parameters are set): all delayable
inputs (like imtcp or imfile) have already been pushed back at this
stage. Also, discarding of lower priority messages (if configured) has
already happened. So we run into one of these situations if we do not
timeout quickly enough:

* if using imuxsock and no systemd journal is involved, the system
  would become unresponsive and most probably a hard reset would be
  required.
* if using imuxsock with imjournal forwarding is active, messages are
  lost because the journal discards them (more aggressive than rsyslog does)
* if using imjournal, the journal will buffer messages. If journal
  runs out of configured space, messages will be discarded. So in this
  mode discarding is moved to a bit later place.
* other non-delayable sources like imudp will also loose messages

So this setting is provided in order to guard against problematic situations,
which always will result either in message loss or system hang. For
action queues, one may debate if it would be better to overflow rapidly
to the main queue. If so desired, this is easy to accomplish by setting
a very large timeout value. The same, of course, is true for the main
queue, but you have been warned if you do so!

In some other words, you can consider this scenario, using default values.
With all progress blocked (unable to deliver a message):

* all delayable inputs (tcp, relp, imfile, imjournal, etc) will block
  indefinitely (assuming queue.lightdelaymark and queue.fulldelaymark
  are set sensible, which they are by default).
* imudp will be losing messages because the OS will be dropping them
* messages arriving via UDP or imuxsock that do make it to rsyslog,
  and that are a severity high enough to not be filtered by
  discardseverity, will block for 2 seconds trying to put the message in
  the queue (in the hope that something happens to make space in the
  queue) and then be dropped to avoid blocking the machine permanently.

  Then the next message to be processed will also be tried for 2 seconds, etc.

* If this is going into an action queue, the log message will remain
  in the main queue during these 2 seconds, and additional logs that
  arrive will accumulate behind this in the main queue.


queue.maxFileSize
-----------------

.. csv-table::
   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
   :widths: auto
   :class: parameter-table

   "integer", "1m/16m", "no", "``$ActionQueueMaxFileSize``"

Specifies the maximum size for a disk queue file. For ``segmentedDisk`` it is
the target size of each segment; a single oversized record is stored in a
dedicated segment and may exceed this target.
Parameter can be specified in Mebibyte or Gibibyte, default for action
queues is 1m and for ruleset queues 16m (1m = 1024*1024).


queue.saveOnShutdown
--------------------

.. csv-table::
   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
   :widths: auto
   :class: parameter-table

   "binary", "off", "no", "``$ActionQueueSaveOnShutdown``"

This parameter specifies if queued data should be saved during an
orderly rsyslog shutdown. It applies to disk-assisted queues, letting
rsyslog persist messages that are still in the memory portion of the
queue before the process exits.

The setting does not protect messages that are still in memory when the
process is killed without graceful shutdown handling, for example by
``SIGKILL``, the kernel OOM killer, a system crash, or power loss. Use a
pure disk queue plus appropriate ``queue.checkpointInterval`` and
``queue.syncqueuefiles`` settings when crash durability is more important
than throughput.


queue.dequeueSlowDown
---------------------

.. csv-table::
   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
   :widths: auto
   :class: parameter-table

   "integer", "0", "no", "``$ActionQueueDequeueSlowDown``"

Regulates how long dequeueing should be delayed. This value must be specified
in microseconds (1000000us is 1sec). It can be used to slow down rsyslog so
it won't send things to fast.
For example if this parameter is set to 10000 on a UDP send action, the action
won't be able to put out more than 100 messages per second.


queue.dequeueTimeBegin
----------------------

.. csv-table::
   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
   :widths: auto
   :class: parameter-table

   "integer", "0", "no", "``$ActionQueueDequeueTimeBegin``"

With this parameter you can specify rsyslog to process queues during specific
time periods. To define a time frame use the 24-hour format without minutes.
This parameter specifies the begin and "queue.dequeuetimeend" the end of the
time frame.


queue.dequeueTimeEnd
--------------------

.. csv-table::
   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
   :widths: auto
   :class: parameter-table

   "integer", "25", "no", "``$ActionQueueDequeueTimeEnd``"

With this parameter you can specify rsyslog to process queues during specific
time periods. To define a time frame use the 24-hour format without minutes.
This parameter specifies the end and "queue.dequeuetimebegin" the begin of the
time frame. The default 25 disables the time-window.


queue.takeFlowCtlFromMsg
------------------------

.. csv-table::
   :header: "type", "default", "mandatory", "|FmtObsoleteName| directive"
   :widths: auto
   :class: parameter-table

   "boolean", "off", "no", "none"

.. versionadded:: 8.1911.0

This is a fine-tuning parameter which permits to control whether or not
rsyslog shall always take the flow control setting from the message. If
so, non-primary queues may also **block** when reaching high water mark.

This permits to add some synchronous processing to rsyslog core engine.
However, **this involves some risk**:  Improper use may make the core engine
stall. As such, **enabling this parameter requires very careful planning
of the rsyslog configuration and deep understanding of the consequences**.

Note that the parameter is applied to individual queues, so a configuration
with a large number of queues can (and must if used) be fine-tuned to
the exact use case.

**The rsyslog team strongly recommends to let this parameter turned off.**



Examples
========

Example 1
---------

The following is a sample of a TCP forwarding action with its own queue.

.. code-block:: none

   action(type="omfwd" target="192.168.2.11" port="10514" protocol="tcp"
          queue.filename="forwarding" queue.size="1000000" queue.type="LinkedList"
         )

See also
========

- :ref:`dev-action-threads`