build/0000775000175100017510000000000012651410764010647 5ustar rgerrgerbuild/development/0000775000175100017510000000000012651410761013166 5ustar rgerrgerbuild/development/index.html0000664000175100017510000002023412651410761015164 0ustar rgerrger Development — rsyslog 8.16.0 documentation

Previous topic

GELF forwarding in rsyslog

Next topic

The rsyslog config data model

This Page

build/development/dev_queue.html0000664000175100017510000005666612651410760016060 0ustar rgerrger The rsyslog queue object — rsyslog 8.16.0 documentation

The rsyslog queue object

This page reflects the status as of 2008-01-17. The documentation is still incomplete. Target audience is developers and users who would like to get an in-depth understanding of queues as used in rsyslog.

Please note that this document is outdated and does not longer reflect the specifics of the queue object. However, I have decided to leave it in the doc set, as the overall picture provided still is quite OK. I intend to update this document somewhat later when I have reached the “store-and-forward” milestone.

Some definitions

A queue is DA-enabled if it is configured to use disk-assisted mode when there is need to. A queue is in DA mode (or DA run mode), when it actually runs disk assisted.

Implementation Details

Disk-Assisted Mode

Memory-Type queues may utilize disk-assisted (DA) mode. DA mode is enabled whenever a queue file name prefix is provided. This is called DA-enabled mode. If DA-enabled, the queue operates as a regular memory queue until a high water mark is reached. If that happens, the queue activates disk assistance (called “runs disk assisted” or “runs DA” - you can find that often in source file comments). To do so, it creates a helper queue instance (the DA queue). At that point, there are two queues running - the primary queue’s consumer changes to a shuffle-to-DA-queue consumer and the original primary consumer is assigned to the DA queue. Existing and new messages are spooled to the disk queue, where the DA worker takes them from and passes them for execution to the actual consumer. In essence, the primary queue has now become a memory buffer for the DA queue. The primary queue will be drained until a low water mark is reached. At that point, processing is held. New messages enqueued to the primary queue will not be processed but kept in memory. Processing resumes when either the high water mark is reached again or the DA queue indicates it is empty. If the DA queue is empty, it is shut down and processing of the primary queue continues as a regular in-memory queue (aka “DA mode is shut down”). The whole thing iterates once the high water mark is hit again.

There is one special case: if the primary queue is shut down and could not finish processing all messages within the configured timeout periods, the DA queue is instantiated to take up the remaining messages. These will be preserved and be processed during the next run. During that period, the DA queue runs in “enqueue-only” mode and does not execute any consumer. Draining the primary queue is typically very fast. If that behaviour is not desired, it can be turned of via parameters. In that case, any remaining in-memory messages are lost.

Due to the fact that when running DA two queues work closely together and worker threads (including the DA worker) may shut down at any time (due to timeout), processing synchronization and startup and shutdown is somewhat complex. I’ll outline the exact conditions and steps down here. I also do this so that I know clearly what to develop to, so please be patient if the information is a bit too in-depth ;)

DA Run Mode Initialization

Three cases:

  1. any time during queueEnqObj() when the high water mark is hit
  2. at queue startup if there is an on-disk queue present (presence of QI file indicates presence of queue data)
  3. at queue shutdown if remaining in-memory data needs to be persisted to disk

In case 1, the worker pool is running. When switching to DA mode, all regular workers are sent termination commands. The DA worker is initiated. Regular workers may run in parallel to the DA worker until they terminate. Regular workers shall terminate as soon as their current consumer has completed. They shall not execute the DA consumer.

In case 2, the worker pool is not yet running and is NOT started. The DA worker is initiated.

In case 3, the worker pool is already shut down. The DA worker is initiated. The DA queue runs in enqueue-only mode.

In all cases, the DA worker starts up and checks if DA mode is already fully initialized. If not, it initializes it, what most importantly means construction of the queue.

Then, regular worker processing is carried out. That is, the queue worker will wait on empty queue and terminate after an timeout. However, If any message is received, the DA consumer is executed. That consumer checks the low water mark. If the low water mark is reached, it stops processing until either the high water mark is reached again or the DA queue indicates it is empty (there is a pthread_cond_t for this synchronization).

In theory, a case-2 startup could lead to the worker becoming inactive and terminating while waiting on the primary queue to fill. In practice, this is highly unlikely (but only for the main message queue) because rsyslog issues a startup message. HOWEVER, we can not rely on that, it would introduce a race. If the primary rsyslog thread (the one that issues the message) is scheduled very late and there is a low inactivity timeout for queue workers, the queue worker may terminate before the startup message is issued. And if the on-disk queue holds only a few messages, it may become empty before the DA worker is re-initiated again. So it is possible that the DA run mode termination criteria occurs while no DA worker is running on the primary queue.

In cases 1 and 3, the DA worker can never become inactive without hitting the DA shutdown criteria. In case 1, it either shuffles messages from the primary to the DA queue or it waits because it has the hit low water mark.

In case 3, it always shuffles messages between the queues (because, that’s the sole purpose of that run). In order for this to happen, the high water mark has been set to the value of 1 when DA run mode has been initialized. This ensures that the regular logic can be applied to drain the primary queue. To prevent a hold due to reaching the low water mark, that mark must be changed to 0 before the DA worker starts.

DA Run Mode Shutdown

In essence, DA run mode is terminated when the DA queue is empty and the primary worker queue size is below the high water mark. It is also terminated when the primary queue is shut down. The decision to switch back to regular (non-DA) run mode is typically made by the DA worker. If it switches, the DA queue is destructed and the regular worker pool is restarted. In some cases, the queue shutdown process may initiate the “switch” (in this case more or less a clean shutdown of the DA queue).

One might think that it would be more natural for the DA queue to detect being idle and shut down itself. However, there are some issues associated with that. Most importantly, all queue worker threads need to be shut down during queue destruction. Only after that has happened, final destruction steps can happen (else we would have a myriad of races). However, it is the DA queues worker thread that detects it is empty (empty queue detection always happens at the consumer side and must so). That would lead to the DA queue worker thread to initiate DA queue destruction which in turn would lead to that very same thread being canceled (because workers must shut down before the queue can be destructed). Obviously, this does not work out (and I didn’t even mention the other issues - so let’s forget about it). As such, the thread that enqueues messages must destruct the queue - and that is the primary queue’s DA worker thread.

There are some subtleties due to thread synchronization and the fact that the DA consumer may not be running (in a case-2 startup). So it is not trivial to reliably change the queue back from DA run mode to regular run mode. The priority is a clean switch. We accept the fact that there may be situations where we cleanly shut down DA run mode, just to re-enable it with the very next message being enqueued. While unlikely, this will happen from time to time and is considered perfectly legal. We can’t predict the future and it would introduce too great complexity to try to do something against that (that would most probably even lead to worse performance under regular conditions).

The primary queue’s DA worker thread may wait at two different places:

  1. after reaching the low water mark and waiting for either high water or DA queue empty
  2. at the regular pthread_cond_wait() on an empty primary queue

Case 2 is unlikely, but may happen (see info above on a case 2 startup).

The DA worker may also not wait at all, because it is actively executing and shuffling messages between the queues. In that case, however, the program flow passes both of the two wait conditions but simply does not wait.

Finally, the DA worker may be inactive(again, with a case-2 startup). In that case no work(er) at all is executed. Most importantly, without the DA worker being active, nobody will ever detect the need to change back to regular mode. If we have this situation, the very next message enqueued will cause the switch, because then the DA run mode shutdown criteria is met. However, it may take close to eternal for this message to arrive. During that time, disk and memory resources for the DA queue remain allocated. This also leaves processing in a sub-optimal state and it may take longer than necessary to switch back to regular queue mode when a message burst happens. In extreme cases, this could even lead to shutdown of DA run mode, which takes so long that the high water mark is passed and DA run mode is immediately re-initialized - while with an immediate switch, the message burst may have been able to be processed by the in-memory queue without DA support.

So in short, it is desirable switch to regular run mode as soon as possible. To do this, we need an active DA worker. The easy solution is to initiate DA worker startup from the DA queue’s worker once it detects empty condition. To do so, the DA queue’s worker must call into a “DA worker startup initiation” routine inside the main queue. As a reminder, the DA worker will most probably not receive the “DA queue empty” signal in that case, because it will be long sent (in most cases) before the DA worker even waits for it. So it is vital that DA run mode termination checks be done in the DA worker before it goes into any wait condition.

Please note that the “DA worker startup initiation” routine may be called concurrently from multiple initiators. To prevent a race, it must be guarded by the queue mutexand return without any action (and no error code!) if the DA worker is already initiated.

All other cases can be handled by checking the termination criteria immediately at the start of the worker and then once again for each run. The logic follows this simplified flow diagram:

Some of the more subtle aspects of worker processing (e.g. enqueue thread signaling and other fine things) have been left out in order to get the big picture. What is called “check DA mode switchback...” right after “worker init” is actually a check for the worker’s termination criteria. Typically, the worker termination criteria is a shutdown request. However, for a DA worker, termination is also requested if the queue size is below the high water mark AND the DA queue is empty. There is also a third termination criteria and it is not even on the chart: that is the inactivity timeout, which exists in all modes. Note that while the inactivity timeout shuts down a thread, it logically does not terminate the worker pool (or DA worker): workers are restarted on an as-needed basis. However, inactivity timeouts are very important because they require us to restart workers in some situations where we may expect a running one. So always keep them on your mind.

Queue Destruction

Now let’s consider the case of destruction of the primary queue.During destruction, our focus is on loosing as few messages as possible. If the queue is not DA-enabled, there is nothing but the configured timeouts to handle that situation. However, with a DA-enabled queue there are more options.

If the queue is DA-enabled, it may be configured to persist messages to disk before it is terminated. In that case, loss of messages never occurs (at the price of a potentially lengthy shutdown). Even if that setting is not applied, the queue should drain as many messages as possible to the disk. For that reason, it makes no sense to wait on a low water mark. Also, if the queue is already in DA run mode, it does not make any sense to switch back to regular run mode during termination and then try to process some messages via the regular consumer. It is much more appropriate the try completely drain the queue during the remaining timeout period. For the same reason, it is preferred that no new consumers be activated (via the DA queue’s worker), as they only cost valuable CPU cycles and, more importantly, would potentially be long(er)-running and possibly be needed to be cancelled. To prevent all of that, queue parameters are changed for DA-enabled queues: the high water mark is to 1 and the low water mark to 0 on the primary queue. The DA queue is commanded to run in enqueue-only mode. If the primary queue is configured to persist messages to disk before it is terminated, its SHUTDOWN timeout is changed to to eternal. These parameters will cause the queue to drain as much as possible to disk (and they may cause a case 3 DA run mode initiation). Please note that once the primary queue has been drained, the DA queue’s worker will automatically switch back to regular (non-DA) run mode. It must be ensured that no worker cancellation occurs during that switchback. Please note that the queue may not switch back to regular run mode if it is not configured to persist messages to disk before it is terminated. In order to apply the new parameters, worker threads must be awakened. Remember we may not be in DA run mode at this stage. In that case, the regular workers must be awakened, which then will switch to DA run mode. No worker may be active, in that case one must be initiated. If in DA run mode and the DA worker is inactive, the  “DA worker startup initiation” must be called to activate it. That routine ensures only one DA worker is started even with multiple concurrent callers - this may be the case here. The DA queue’s worker may have requested DA worker startup in order to terminate on empty queue (which will probably not be honored as we have changed the low water mark).

After all this is done, the queue destructor requests termination of the queue’s worker threads. It will use the normal timeouts and potentially cancel too-long running worker threads. The shutdown process must ensure that all workers reach running state before they are commanded to terminate. Otherwise it may run into a race condition that could lead to a false shutdown with workers running asynchronously. As a few workers may have just been started to initialize (to apply new parameter settings), the probability for this race condition is extremely high, especially on single-CPU systems.

After all workers have been shut down (or cancelled), the queue may still be in DA run mode. If so, this must be terminated, which now can simply be done by destructing the DA queue object. This is not a real switchback to regular run mode, but that doesn’t matter because the queue object will soon be gone away.

Finally, the queue is mostly shut down and ready to be actually destructed. As a last try, the queuePersists() entry point is called. It is used to persists a non-DA-enabled queue in whatever way is possible for that queue. There may be no implementation for the specific queue type. Please note that this is not just a theoretical construct. This is an extremely important code path when the DA queue itself is destructed. Remember that it is a queue object in its own right. The DA queue is obviously not DA-enabled, so it calls into queuePersists() during its destruction - this is what enables us to persist the disk queue!

After that point, left over queue resources (mutexes, dynamic memory, ...) are freed and the queue object is actually destructed.

build/development/dev_oplugins.html0000664000175100017510000006766012651410760016570 0ustar rgerrger Writing Rsyslog Output Plugins — rsyslog 8.16.0 documentation

Writing Rsyslog Output Plugins

This page is the begin of some developer documentation for writing output plugins. Doing so is quite easy (and that was a design goal), but there currently is only sparse documentation on the process available. I was tempted NOT to write this guide here because I know I will most probably not be able to write a complete guide.

However, I finally concluded that it may be better to have some information and pointers than to have nothing.

Getting Started and Samples

The best to get started with rsyslog plugin development is by looking at existing plugins. All that start with “om” are output modules. That means they are primarily thought of being message sinks. In theory, however, output plugins may aggergate other functionality, too. Nobody has taken this route so far so if you would like to do that, it is highly suggested to post your plan on the rsyslog mailing list, first (so that we can offer advise).

The rsyslog distribution tarball contains the omstdout plugin which is extremely well targeted for getting started. Just note that this plugin itself is not meant for production use. But it is very simplistic and so a really good starting point to grasp the core ideas.

In any case, you should also read the comments in ./runtime/module-template.h. Output plugins are build based on a large set of code-generating macros. These macros handle most of the plumbing needed by the interface. As long as no special callback to rsyslog is needed (it typically is not), an output plugin does not really need to be aware that it is executed by rsyslog. As a plug-in programmer, you can (in most cases) “code as usual”. However, all macros and entry points need to be provided and thus reading the code comments in the files mentioned is highly suggested.

For testing, you need rsyslog’s debugging support. Some useful information is given in “troubleshooting rsyslog from the doc set.

Special Topics

Threading

Rsyslog uses massive parallel processing and multithreading. However, a plugin’s entry points are guaranteed to be never called concurrently for the same action. That means your plugin must be able to be called concurrently by two or more threads, but you can be sure that for the same instance no concurrent calls happen. This is guaranteed by the interface specification and the rsyslog core guards against multiple concurrent calls. An instance, in simple words, is one that shares a single instanceData structure.

So as long as you do not mess around with global data, you do not need to think about multithreading (and can apply a purely sequential programming methodology).

Please note that during the configuration parsing stage of execution, access to global variables for the configuration system is safe. In that stage, the core will only call sequentially into the plugin.

Getting Message Data

The doAction() entry point of your plugin is provided with messages to be processed. It will only be activated after filtering and all other conditions, so you do not need to apply any other conditional but can simply process the message.

Note that you do NOT receive the full internal representation of the message object. There are various (including historical) reasons for this and, among others, this is a design decision based on security.

Your plugin will only receive what the end user has configured in a $template statement. However, starting with 4.1.6, there are two ways of receiving the template content. The default mode, and in most cases sufficient and optimal, is to receive a single string with the expanded template. As I said, this is usually optimal, think about writing things to files, emailing content or forwarding it.

The important philosophy is that a plugin should never reformat any of such strings - that would either remove the user’s ability to fully control message formats or it would lead to duplicating code that is already present in the core. If you need some formatting that is not yet present in the core, suggest it to the rsyslog project, best done by sending a patch ;), and we will try hard to get it into the core (so far, we could accept all such suggestions - no promise, though).

If a single string seems not suitable for your application, the plugin can also request access to the template components. The typical use case seems to be databases, where you would like to access properties via specific fields. With that mode, you receive a char ** array, where each array element points to one field from the template (from left to right). Fields start at array index 0 and a NULL pointer means you have reached the end of the array (the typical Unix “poor man’s linked list in an array” design). Note, however, that each of the individual components is a string. It is not a date stamp, number or whatever, but a string. This is because rsyslog processes strings (from a high-level design look at it) and so this is the natural data type. Feel free to convert to whatever you need, but keep in mind that malformed packets may have lead to field contents you’d never expected...

If you like to use the array-based parameter passing method, think that it is only available in rsyslog 4.1.6 and above. If you can accept that your plugin will not be working with previous versions, you do not need to handle pre 4.1.6 cases. However, it would be “nice” if you shut down yourself in these cases - otherwise the older rsyslog core engine will pass you a string where you expect the array of pointers, what most probably results in a segfault. To check whether or not the core supports the functionality, you can use this code sequence:

BEGINmodInit()
    rsRetVal localRet;
    rsRetVal (*pomsrGetSupportedTplOpts)(unsigned long *pOpts);
    unsigned long opts;
    int bArrayPassingSupported;     /* does core support template passing as an array? */
CODESTARTmodInit
    *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */
CODEmodInit_QueryRegCFSLineHdlr
    /* check if the rsyslog core supports parameter passing code */
    bArrayPassingSupported = 0;
    localRet = pHostQueryEtryPt((uchar*)"OMSRgetSupportedTplOpts", &pomsrGetSupportedTplOpts);
    if(localRet == RS_RET_OK) {
        /* found entry point, so let's see if core supports array passing */
        CHKiRet((*pomsrGetSupportedTplOpts)(&opts));
        if(opts & OMSR_TPL_AS_ARRAY)
            bArrayPassingSupported = 1;
    } else if(localRet != RS_RET_ENTRY_POINT_NOT_FOUND) {
        ABORT_FINALIZE(localRet); /* Something else went wrong, what is not acceptable */
    }
    DBGPRINTF("omstdout: array-passing is %ssupported by rsyslog core.\n", bArrayPassingSupported ? "" : "not ");

    if(!bArrayPassingSupported) {
        DBGPRINTF("rsyslog core too old, shutting down this plug-in\n");
        ABORT_FINALIZE(RS_RET_ERR);
    }

The code first checks if the core supports the OMSRgetSupportedTplOpts() API (which is also not present in all versions!) and, if so, queries the core if the OMSR_TPL_AS_ARRAY mode is supported. If either does not exits, the core is too old for this functionality. The sample snippet above then shuts down, but a plugin may instead just do things different. In omstdout, you can see how a plugin may deal with the situation.

In any case, it is recommended that at least a graceful shutdown is made and the array-passing capability not blindly be used. In such cases, we can not guard the plugin from segfaulting and if the plugin (as currently always) is run within rsyslog’s process space, that results in a segfault for rsyslog. So do not do this.

Another possible mode is OMSR_TPL_AS_JSON, where instead of the template a json-c memory object tree is passed to the module. The module can extract data via json-c API calls. It MUST NOT modify the provided structure. This mode is primarily aimed at plugins that need to process tree-like data, as found for example in MongoDB or ElasticSearch.

Batching of Messages

Starting with rsyslog 4.3.x, batching of output messages is supported. Previously, only a single-message interface was supported.

With the single message plugin interface, each message is passed via a separate call to the plugin. Most importantly, the rsyslog engine assumes that each call to the plugin is a complete transaction and as such assumes that messages be properly committed after the plugin returns to the engine.

With the batching interface, rsyslog employs something along the line of “transactions”. Obviously, the rsyslog core can not make non-transactional outputs to be fully transactional. But what it can is support that the output tells the core which messages have been committed by the output and which not yet. The core can than take care of those uncommitted messages when problems occur. For example, if a plugin has received 50 messages but not yet told the core that it committed them, and then returns an error state, the core assumes that all these 50 messages were not written to the output. The core then requeues all 50 messages and does the usual retry processing. Once the output plugin tells the core that it is ready again to accept messages, the rsyslog core will provide it with these 50 not yet committed messages again (actually, at this point, the rsyslog core no longer knows that it is re-submitting the messages). If, in contrary, the plugin had told rsyslog that 40 of these 50 messages were committed (before it failed), then only 10 would have been requeued and resubmitted.

In order to provide an efficient implementation, there are some (mild) constraints in that transactional model: first of all, rsyslog itself specifies the ultimate transaction boundaries. That is, it tells the plugin when a transaction begins and when it must finish. The plugin is free to commit messages in between, but it must commit all work done when the core tells it that the transaction ends. All messages passed in between a begin and end transaction notification are called a batch of messages. They are passed in one by one, just as without transaction support. Note that batch sizes are variable within the range of 1 to a user configured maximum limit. Most importantly, that means that plugins may receive batches of single messages, so they are required to commit each message individually. If the plugin tries to be “smarter” than the rsyslog engine and does not commit messages in those cases (for example), the plugin puts message stream integrity at risk: once rsyslog has notified the plugin of transaction end, it discards all messages as it considers them committed and save. If now something goes wrong, the rsyslog core does not try to recover lost messages (and keep in mind that “goes wrong” includes such uncontrollable things like connection loss to a database server). So it is highly recommended to fully abide to the plugin interface details, even though you may think you can do it better. The second reason for that is that the core engine will have configuration settings that enable the user to tune commit rate to their use-case specific needs. And, as a relief: why would rsyslog ever decide to use batches of one? There is a trivial case and that is when we have very low activity so that no queue of messages builds up, in which case it makes sense to commit work as it arrives. (As a side-note, there are some valid cases where a timeout-based commit feature makes sense. This is also under evaluation and, once decided, the core will offer an interface plus a way to preserve message stream integrity for properly-crafted plugins).

The second restriction is that if a plugin makes commits in between (what is perfectly legal) those commits must be in-order. So if a commit is made for message ten out of 50, this means that messages one to nine are also committed. It would be possible to remove this restriction, but we have decided to deliberately introduce it to simplify things.

Output Plugin Transaction Interface

In order to keep compatible with existing output plugins (and because it introduces no complexity), the transactional plugin interface is build on the traditional non-transactional one. Well... actually the traditional interface was transactional since its introduction, in the sense that each message was processed in its own transaction.

So the current doAction() entry point can be considered to have this structure (from the transactional interface point of view):

doAction()
    {
    beginTransaction()
    ProcessMessage()
    endTransaction()
    }

For the transactional interface, we now move these implicit beginTransaction() and endTransaction(() call out of the message processing body, resulting is such a structure:

beginTransaction()
    {
    /* prepare for transaction */
    }

doAction()
    {
    ProcessMessage()
    /* maybe do partial commits */
    }

endTransaction()
    {
    /* commit (rest of) batch */
    }

And this calling structure actually is the transactional interface! It is as simple as this. For the new interface, the core calls a beginTransaction() entry point inside the plugin at the start of the batch. Similarly, the core call endTransaction() at the end of the batch. The plugin must implement these entry points according to its needs.

But how does the core know when to use the old or the new calling interface? This is rather easy: when loading a plugin, the core queries the plugin for the beginTransaction() and endTransaction() entry points. If the plugin supports these, the new interface is used. If the plugin does not support them, the old interface is used and rsyslog implies that a commit is done after each message. Note that there is no special “downlevel” handling necessary to support this. In the case of the non-transactional interface, rsyslog considers each completed call to doAction as partial commit up to the current message. So implementation inside the core is very straightforward.

Actually, we recommend that the transactional entry points only be defined by those plugins that actually need them. All others should not define them in which case the default commit behaviour inside rsyslog will apply (thus removing complexity from the plugin).

In order to support partial commits, special return codes must be defined for doAction. All those return codes mean that processing completed successfully. But they convey additional information about the commit status as follows:

RS_RET_OK The record and all previous inside the batch has been commited. Note: this definition is what makes integrating plugins without the transaction being/end calls so easy - this is the traditional “success” return state and if every call returns it, there is no need for actually calling endTransaction(), because there is no transaction open).
RS_RET_DEFER_COMMIT The record has been processed, but is not yet commited. This is the expected state for transactional-aware plugins.
RS_RET_PREVIOUS_COMMITTED The previous record inside the batch has been committed, but the current one not yet. This state is introduced to support sources that fill up buffers and commit once a buffer is completely filled. That may occur halfway in the next record, so it may be important to be able to tell the engine the everything up to the previouos record is commited

Note that the typical calling cycle is beginTransaction(), followed by n times doAction() followed by endTransaction(). However, if either beginTransaction() or doAction() return back an error state (including RS_RET_SUSPENDED), then the transaction is considered aborted. In result, the remaining calls in this cycle (e.g. endTransaction()) are never made and a new cycle (starting with beginTransaction() is begun when processing resumes. So an output plugin must expect and handle those partial cycles gracefully.

The question remains how can a plugin know if the core supports batching? First of all, even if the engine would not know it, the plugin would return with RS_RET_DEFER_COMMIT, what then would be treated as an error by the engine. This would effectively disable the output, but cause no further harm (but may be harm enough in itself).

The real solution is to enable the plugin to query the rsyslog core if this feature is supported or not. At the time of the introduction of batching, no such query-interface exists. So we introduce it with that release. What the means is if a rsyslog core can not provide this query interface, it is a core that was build before batching support was available. So the absence of a query interface indicates that the transactional interface is not available. One might now be tempted to think there is no need to do the actual check, but is is recommended to ask the rsyslog engine explicitly if the transactional interface is present and will be honored. This enables us to create versions in the future which have, for whatever reason we do not yet know, no support for this interface.

The logic to do these checks is contained in the INITChkCoreFeature macro, which can be used as follows:

INITChkCoreFeature(bCoreSupportsBatching, CORE_FEATURE_BATCHING);

Here, bCoreSupportsBatching is a plugin-defined integer which after execution is 1 if batches (and thus the transactional interface) is supported and 0 otherwise. CORE_FEATURE_BATCHING is the feature we are interested in. Future versions of rsyslog may contain additional feature-test-macros (you can see all of them in ./runtime/rsyslog.h).

Note that the ompsql output plugin supports transactional mode in a hybrid way and thus can be considered good example code.

Open Issues

  • Processing errors handling
  • reliable re-queue during error handling and queue termination

Licensing

From the rsyslog point of view, plugins constitute separate projects. As such, we think plugins are not required to be compatible with GPLv3. However, this is no legal advise. If you intend to release something under a non-GPLV3 compatible license it is probably best to consult with your lawyer.

Most importantly, and this is definite, the rsyslog team does not expect or require you to contribute your plugin to the rsyslog project (but of course we are happy if you do).

build/development/config_data_model.html0000664000175100017510000003364012651410760017477 0ustar rgerrger The rsyslog config data model — rsyslog 8.16.0 documentation

The rsyslog config data model

This document describes the config data model on a high layer. For details, it is suggested to review the actual source code. The aim of this document is to provide general understanding for both rsyslog developers as well as developers writing config management systems.

Objects

Most config objects life in a flat space and are global to rsyslog. However, actual rule processing is done via a script-like language. These config scripts need to be represented via a tree structure.

Note that the language as currently implemented is Turing-complete if the user makes use of very tricky constructs. It was never our intention to provide a Turing-complete language and we will probably try to disable these tricks in the future. However, this is not a priority for us, as these users get what they deserve. For someone involved with the config, it probably is sufficient to know that loops are not supported by the config language (even though you can create loop-like structures). Thus, a tree is fully sufficient to represent any configuration.

In the following sections, we’ll quickly describe variables/properties, flat structure elements and the execution tree.

Variables/Properties

Rsyslog supports

  • traditional syslog (RFC-based) message properties
  • structured data content, including any non-syslog properties
  • Variables
    • global
    • local
    • message-enhancing (like message properties)

A description of these properties and variables is available elsewhere. As far as a config processor is concerned, the important thing to know is that they be used during template definitions and script operations.

Flat Elements

Global Parameters

This element must contain all global parameters settable by rsyslog. This includes elements from the global() as well as main_queue() config statements. As of this writing, some global parameter can only be set by legacy statements.

Note that main_queue() actually is a full queue defintion.

Modules

This contains all loaded modules, among others:

  • input modules
  • output modules
  • message modification modules
  • message parsers

Note that for historical reasons some output modules are directly linked into rsyslog and must not be specified.

Each module must be given only once. The data object must contain all module-global parameters.

Inputs

Describes all defined inputs with their parameters. Is build from the input() statement or its legacy equivalent (ugly). Contains links to

  • module used for input
  • ruleset used for processing

Rulesets

They contain the tree-like execution structure. However, rulesets itself are flat and cannot be nested. Note that there exists statements that permit rulesets to call into each other, but all rulesets are in the same flat top-level space.

Note that a ruleset has an associated queue object. In most cases, it needs not to be configured as a real queue (not one in direct mode) is only required in special cases.

Hierarchical Elements

These are used for rule exection. They are somewhat hard to fit into a traditional config scheme, as they provide full tree-like branching structure.

Basically, a tree consists of statements and evaluations. Consider the ruleset to be the root of the execution tree. It is rather common that the tree’s main level is a long linked list, with only actions being branched out. This, for example, happens with a traditional rsyslog.conf setting, which only contains files to be written based on some priority filters. However, one must not be tricked into thinking that this basic case is sufficient to support as enterprise users typically create far more complex cases.

In essence, rsyslog walks the tree, and executes statements while it does so. Usually, a filter needs to be evaluated and execution branches based on the filter outcome. The tree actually is an AST.

Execution Statements

These are most easy to implement as they are end nodes (and as such nothing can be nested under them). They are most importantly created by the action() config object, but also with statements like “set” and “unset”. Note that “call” is also considered a terminal node, even though it executes another ruleset.

Note that actions have associated queues, so a queue object and its parameter need to be present. When building configurations interactively, it is suggested that the default is either not to configure queue parameters by default or to do this only for actions where it makes sense (e.g. connection to remote systems which may go offline).

Expression Evaluation

A full expression evaluation engine is available who does the typical programming-language type of expression processing. The usual mathematical, boolean and string operations are supported, as well as functions. As of this writing, functions are hardcoded into rsyslog but may in the future be part of a loadable module. Evaluations can access all rsyslog properties and variables. They may be nested arbitrarily deep.

Control-of-Flow Statements

Remember that rsyslog does intentionally not support loop statements. So control-of-flow boils down to

  • conditional statements
    • “if ... then ... else ...”
    • syslog PRI-based filters
    • property-based filters
  • stop

Where “stop” terminates processing of this message. The conditional statements contain subbranches, where “if” contains both “then” and “else” subbranches and the other two only the “then” subranch (Note: inside the execution engine, the others may also have “else” branches, but these are result of the rsyslog config optimizer run and cannot configured by the user).

When executing a config script, rsyslog executes the subbranch in question and then continues to evaluate the next statement in the currently executing branch that contained the conditional statement. If there is no next statement, it goes up one layer. This is continued until the last statement of the root statement list is reached. At that point execution of the message is terminated and the message object destructed. Again, think AST, as this is exactly what it is.

Note on Queue Objects

Queue objects are not named objects inside the rsyslog configuration. So their data is always contained with the object that uses the queue (action(), ruleset(), main_queue()). From a UI perspective, this unfortunately tends to complicate a config builder a bit.

build/development/generic_design.html0000664000175100017510000003156312651410761017031 0ustar rgerrger Generic design of a syslogd — rsyslog 8.16.0 documentation

Generic design of a syslogd

Written 2007-04-10 by Rainer Gerhards

The text below describes a generic approach on how a syslogd can be implemented. I created this description for some other project, where it was not used. Instead of throwing it away, I thought it would be a good addition to the rsyslog documentation. While rsyslog differs in details from the description below, it is sufficiently close to it. Further development of rsyslog will probably match it even closer to the description.

If you intend to read the rsyslog source code, I recommend reading this document here first. You will not find the same names and not all of the concepts inside rsyslog. However, I think your understanding will benefit from knowing the generic architecture.

+-----------------+
| "remote" PLOrig |
+-----------------+
    |
    I  +--------+-----+-----+          +-----+-------+------+-----+
    P  | PLOrig | GWI | ... |          | GWO | Store | Disc | ... |
    C  +--------+-----+-----+          +-----+-------+------+-----+
    |        |                                          ^
    v        v                                          |
   +--------------+        +------------+          +--------------+
   | PLGenerator  |        |  RelayEng  |          | CollectorEng |
   +--------------+        +------------+          +--------------+
          |                      ^                       ^
          |                      |                       |
          v                      v                       |
   +-------------+         +------------+          +--------------+
   |   PLG Ext   |         | RelEng Ext |          | CollcEng Ext |
   +-------------+         +------------+          +--------------+
          |                      ^                       ^
          |                      |                       |
          v                      v                       |
   +--------------------------------------------------------------+
   |                      Message Router                          |
   +--------------------------------------------------------------+
                      |                            ^
                      v                            |
   +--------------------------------------------------------------+
   |           Message CoDec (e.g. RFC 3164, RFCYYYY)             |
   +--------------------------------------------------------------+
                      |                            ^
                      v                            |
   +---------------------+-----------------------+----------------+
   |    transport UDP    |    transport TLS      |      ...       |
   +---------------------+-----------------------+----------------+

             Generic Syslog Application Architecture
  • A “syslog application” is an application whose purpose is the processing of syslog messages. It may be part of a larger application with a broader purpose. An example: a database application might come with its own syslog send subsystem and not go through a central syslog application. In the sense of this document, that application is called a “syslog application” even though a casual observer might correctly call it a database application and may not even know that it supports sending of syslog messages.
  • Payload is the information that is to be conveyed. Payload by itself may have any format and is totally independent from to format specified in this document. The “Message CoDec” of the syslog application will bring it into the required format.
  • Payload Originators (“PLOrig”) are the original creators of payload. Typically, these are application programs.
  • A “Remote PLOrig” is a payload originator residing in a different application than the syslog application itself. That application may reside on a different machine and may talk to the syslog application via RPC.
  • A “PLOrig” is a payload originator residing within the syslog application itself. Typically, this PLOrig emits syslog application startup, shutdown, error and status log messages.
  • A “GWI” is a inbound gateway. For example, a SNMP-to-syslog gateway may receive SNMP messages and translate them into syslog.
  • The ellipsis after “GWI” indicates that there are potentially a variety of different other ways to originally generate payload.
  • A “PLGenerator” is a payload generator. It takes the information from the payload-generating source and integrates it into the syslog subsystem of the application. This is a highly theoretical concept. In practice, there may not actually be any such component. Instead, the payload generators (or other parts like the GWI) may talk directly to the syslog subsystem. Conceptually, the “PLGenerator” is the first component where the information is actually syslog content.
  • A “PLG Ext” is a payload generator extension. It is used to modify the syslog information. An example of a “PLG Ext” might be the addition of cryptographic signatures to the syslog information.
  • A “Message Router” is a component that accepts in- and outbound syslog information and routes it to the proper next destination inside the syslog application. The routing information itself is expected to be learnt by operator configuration.
  • A “Message CoDec” is the message encoder/decoder. The encoder takes syslog information and encodes them into the required format for a syslog message. The decoder takes a syslog message and decodes it into syslog information. Codecs for multiple syslog formats may be present inside a single syslog application.
  • A transport (UDP, TLS, yet-to-be-defined ones) sends and receives syslog messages. Multiple transports may be used by a single syslog application at the same time. A single transport instance may be used for both sending and receiving. Alternatively, a single instance might be used for sending and receiving exclusively. Multiple instances may be used for different listener ports and receivers.
  • A “RelayEng” is the relaying engine. It provides functionality necessary for receiving syslog information and sending it to another syslog application.
  • A “RelEng Ext” is an extension that processes syslog information as it enters or exits a RelayEng. An example of such a component might be a relay cryptographically signing received syslog messages. Such a function might be useful to guarantee authenticity starting from a given point inside a relay chain.
  • A “CollectorEng” is a collector engine. At this component, syslog information leaves the syslog system and is translated into some other form. After the CollectorEng, the information is no longer defined to be of native syslog type.
  • A “CollcEng Ext” is a collector engine extension. It modifies syslog information before it is passed on to the CollectorEng. An example for this might be the verification of cryptographically signed syslog message information. Please note that another implementation approach would be to do the verification outside of the syslog application or in a stage after “CollectorEng”.
  • A “GWO” is an outbound gateway. An example of this might be the forwarding of syslog information via SNMP or SMTP. Please note that when a GWO directly connects to a GWI on a different syslog application, no native exchange of syslog information takes place. Instead, the native protocol of these gateways (e.g. SNMP) is used. The syslog information is embedded inside that protocol. Depending on protocol and gateway implementation, some of the native syslog information might be lost.
  • A “Store” is any way to persistently store the extracted syslog information, e.g. to the file system or to a data base.
  • “Disc” means the discarding of messages. Operators often find it useful to discard noise messages and so most syslog applications contain a way to do that.
  • The ellipsis after “Disc” indicates that there are potentially a variety of different other ways to consume syslog information.
  • There may be multiple instances of each of the described components in a single syslog application.
  • A syslog application is made up of all or some of the above mentioned components.

Previous topic

The rsyslog queue object

Next topic

Historical Documents

This Page

build/index.html0000664000175100017510000002365312651410761012652 0ustar rgerrger Welcome to Rsyslog — rsyslog 8.16.0 documentation

Welcome to Rsyslog

Rsyslog is a rocket-fast system for log processing. It offers high-performance, great security features and a modular design. While it started as a regular syslogd, rsyslog has evolved into a kind of swiss army knife of logging, being able to

  • accept inputs from a wide variety of sources,
  • transform them,
  • and output the results to diverse destinations.

Rsyslog has a strong enterprise focus but also scales down to small systems. It supports, among others, MySQL, PostgreSQL, failover log destinations, ElasticSearch, syslog/tcp transport, fine grain output format control, high precision timestamps, queued operations and the ability to filter on any message part.

Sponsors and Community

Please visit the rsyslog Sponsor’s Page[4] to honor the project sponsors or become one yourself! We are very grateful for any help towards the project goals.

Visit the Rsyslog Status Page[2] to obtain current version information and project status.

If you like rsyslog, you might want to lend us a helping hand. It doesn’t require a lot of time - even a single mouse click helps. Learn how to help the rsyslog project.

Table Of Contents

Next topic

Configuration

This Page

build/_sources/0000775000175100017510000000000012651410763012470 5ustar rgerrgerbuild/_sources/development/0000775000175100017510000000000012651410761015010 5ustar rgerrgerbuild/_sources/development/index.txt0000664000175100017510000000006612411216133016651 0ustar rgerrgerDevelopment =========== .. toctree:: :glob: * build/_sources/development/config_data_model.txt0000664000175100017510000001470212651410660021171 0ustar rgerrgerThe rsyslog config data model ============================= This document describes the config data model on a high layer. For details, it is suggested to review the actual source code. The aim of this document is to provide general understanding for both rsyslog developers as well as developers writing config management systems. Objects ======= Most config objects life in a flat space and are global to rsyslog. However, actual rule processing is done via a script-like language. These config scripts need to be represented via a tree structure. Note that the language as currently implemented is Turing-complete if the user makes use of very tricky constructs. It was never our intention to provide a Turing-complete language and we will probably try to disable these tricks in the future. However, this is not a priority for us, as these users get what they deserve. For someone involved with the config, it probably is sufficient to know that loops are **not** supported by the config language (even though you can create loop-like structures). Thus, a tree is fully sufficient to represent any configuration. In the following sections, we'll quickly describe variables/properties, flat structure elements and the execution tree. Variables/Properties -------------------- Rsyslog supports * traditional syslog (RFC-based) message properties * structured data content, including any non-syslog properties * Variables - global - local - message-enhancing (like message properties) A description of these properties and variables is available elsewhere. As far as a config processor is concerned, the important thing to know is that they be used during template definitions and script operations. Flat Elements ------------- Global Parameters ^^^^^^^^^^^^^^^^^ This element must contain all global parameters settable by rsyslog. This includes elements from the global() as well as main_queue() config statements. As of this writing, some global parameter can only be set by legacy statements. Note that main_queue() actually is a full queue defintion. Modules ^^^^^^^ This contains all loaded modules, among others: * input modules * output modules * message modification modules * message parsers Note that for historical reasons some output modules are directly linked into rsyslog and must not be specified. Each module must be given only once. The data object must contain all module-global parameters. Inputs ^^^^^^ Describes all defined inputs with their parameters. Is build from the input() statement or its legacy equivalent (ugly). Contains links to * module used for input * ruleset used for processing Rulesets ^^^^^^^^ They contain the tree-like execution structure. However, rulesets itself are flat and cannot be nested. Note that there exists statements that permit rulesets to call into each other, but all rulesets are in the same flat top-level space. Note that a ruleset has an associated queue object. In most cases, it needs not to be configured as a real queue (not one in direct mode) is only required in special cases. Hierarchical Elements --------------------- These are used for rule exection. They are somewhat hard to fit into a traditional config scheme, as they provide full tree-like branching structure. Basically, a tree consists of statements and evaluations. Consider the ruleset to be the root of the execution tree. It is rather common that the tree's main level is a long linked list, with only actions being branched out. This, for example, happens with a traditional rsyslog.conf setting, which only contains files to be written based on some priority filters. However, one must not be tricked into thinking that this basic case is sufficient to support as enterprise users typically create far more complex cases. In essence, rsyslog walks the tree, and executes statements while it does so. Usually, a filter needs to be evaluated and execution branches based on the filter outcome. The tree actually **is** an AST. Execution Statements ^^^^^^^^^^^^^^^^^^^^ These are most easy to implement as they are end nodes (and as such nothing can be nested under them). They are most importantly created by the action() config object, but also with statements like "set" and "unset". Note that "call" is also considered a terminal node, even though it executes *another* ruleset. Note that actions have associated queues, so a queue object and its parameter need to be present. When building configurations interactively, it is suggested that the default is either not to configure queue parameters by default or to do this only for actions where it makes sense (e.g. connection to remote systems which may go offline). Expression Evaluation ^^^^^^^^^^^^^^^^^^^^^ A full expression evaluation engine is available who does the typical programming-language type of expression processing. The usual mathematical, boolean and string operations are supported, as well as functions. As of this writing, functions are hardcoded into rsyslog but may in the future be part of a loadable module. Evaluations can access all rsyslog properties and variables. They may be nested arbitrarily deep. Control-of-Flow Statements ^^^^^^^^^^^^^^^^^^^^^^^^^^ Remember that rsyslog does intentionally not support loop statements. So control-of-flow boils down to * conditional statements - "if ... then ... else ..." - syslog PRI-based filters - property-based filters * stop Where "stop" terminates processing of this message. The conditional statements contain subbranches, where "if" contains both "then" and "else" subbranches and the other two only the "then" subranch (Note: inside the execution engine, the others may also have "else" branches, but these are result of the rsyslog config optimizer run and cannot configured by the user). When executing a config script, rsyslog executes the subbranch in question and then continues to evaluate the next statement in the currently executing branch that contained the conditional statement. If there is no next statement, it goes up one layer. This is continued until the last statement of the root statement list is reached. At that point execution of the message is terminated and the message object destructed. Again, think AST, as this is exactly what it is. Note on Queue Objects --------------------- Queue objects are **not** named objects inside the rsyslog configuration. So their data is always contained with the object that uses the queue (action(), ruleset(), main_queue()). From a UI perspective, this unfortunately tends to complicate a config builder a bit. build/_sources/development/dev_queue.txt0000664000175100017510000004106012651410660017532 0ustar rgerrgerThe rsyslog queue object ======================== This page reflects the status as of 2008-01-17. The documentation is still incomplete. Target audience is developers and users who would like to get an in-depth understanding of queues as used in `rsyslog `_. **Please note that this document is outdated and does not longer reflect the specifics of the queue object. However, I have decided to leave it in the doc set, as the overall picture provided still is quite OK. I intend to update this document somewhat later when I have reached the "store-and-forward" milestone.** Some definitions ---------------- A queue is DA-enabled if it is configured to use disk-assisted mode when there is need to. A queue is in DA mode (or DA run mode), when it actually runs disk assisted. Implementation Details ---------------------- Disk-Assisted Mode ~~~~~~~~~~~~~~~~~~ Memory-Type queues may utilize disk-assisted (DA) mode. DA mode is enabled whenever a queue file name prefix is provided. This is called DA-enabled mode. If DA-enabled, the queue operates as a regular memory queue until a high water mark is reached. If that happens, the queue activates disk assistance (called "runs disk assisted" or "runs DA" - you can find that often in source file comments). To do so, it creates a helper queue instance (the DA queue). At that point, there are two queues running - the primary queue's consumer changes to a shuffle-to-DA-queue consumer and the original primary consumer is assigned to the DA queue. Existing and new messages are spooled to the disk queue, where the DA worker takes them from and passes them for execution to the actual consumer. In essence, the primary queue has now become a memory buffer for the DA queue. The primary queue will be drained until a low water mark is reached. At that point, processing is held. New messages enqueued to the primary queue will not be processed but kept in memory. Processing resumes when either the high water mark is reached again or the DA queue indicates it is empty. If the DA queue is empty, it is shut down and processing of the primary queue continues as a regular in-memory queue (aka "DA mode is shut down"). The whole thing iterates once the high water mark is hit again. There is one special case: if the primary queue is shut down and could not finish processing all messages within the configured timeout periods, the DA queue is instantiated to take up the remaining messages. These will be preserved and be processed during the next run. During that period, the DA queue runs in "enqueue-only" mode and does not execute any consumer. Draining the primary queue is typically very fast. If that behaviour is not desired, it can be turned of via parameters. In that case, any remaining in-memory messages are lost. Due to the fact that when running DA two queues work closely together and worker threads (including the DA worker) may shut down at any time (due to timeout), processing synchronization and startup and shutdown is somewhat complex. I'll outline the exact conditions and steps down here. I also do this so that I know clearly what to develop to, so please be patient if the information is a bit too in-depth ;) DA Run Mode Initialization ~~~~~~~~~~~~~~~~~~~~~~~~~~ Three cases: #. any time during queueEnqObj() when the high water mark is hit #. at queue startup if there is an on-disk queue present (presence of QI file indicates presence of queue data) #. at queue shutdown if remaining in-memory data needs to be persisted to disk In **case 1**, the worker pool is running. When switching to DA mode, all regular workers are sent termination commands. The DA worker is initiated. Regular workers may run in parallel to the DA worker until they terminate. Regular workers shall terminate as soon as their current consumer has completed. They shall not execute the DA consumer. In **case 2**, the worker pool is not yet running and is NOT started. The DA worker is initiated. In **case 3**, the worker pool is already shut down. The DA worker is initiated. The DA queue runs in enqueue-only mode. In all cases, the DA worker starts up and checks if DA mode is already fully initialized. If not, it initializes it, what most importantly means construction of the queue. Then, regular worker processing is carried out. That is, the queue worker will wait on empty queue and terminate after an timeout. However, If any message is received, the DA consumer is executed. That consumer checks the low water mark. If the low water mark is reached, it stops processing until either the high water mark is reached again or the DA queue indicates it is empty (there is a pthread\_cond\_t for this synchronization). In theory, a **case-2** startup could lead to the worker becoming inactive and terminating while waiting on the primary queue to fill. In practice, this is highly unlikely (but only for the main message queue) because rsyslog issues a startup message. HOWEVER, we can not rely on that, it would introduce a race. If the primary rsyslog thread (the one that issues the message) is scheduled very late and there is a low inactivity timeout for queue workers, the queue worker may terminate before the startup message is issued. And if the on-disk queue holds only a few messages, it may become empty before the DA worker is re-initiated again. So it is possible that the DA run mode termination criteria occurs while no DA worker is running on the primary queue. In cases 1 and 3, the DA worker can never become inactive without hitting the DA shutdown criteria. In **case 1**, it either shuffles messages from the primary to the DA queue or it waits because it has the hit low water mark. In **case 3**, it always shuffles messages between the queues (because, that's the sole purpose of that run). In order for this to happen, the high water mark has been set to the value of 1 when DA run mode has been initialized. This ensures that the regular logic can be applied to drain the primary queue. To prevent a hold due to reaching the low water mark, that mark must be changed to 0 before the DA worker starts. DA Run Mode Shutdown ~~~~~~~~~~~~~~~~~~~~ In essence, DA run mode is terminated when the DA queue is empty and the primary worker queue size is below the high water mark. It is also terminated when the primary queue is shut down. The decision to switch back to regular (non-DA) run mode is typically made by the DA worker. If it switches, the DA queue is destructed and the regular worker pool is restarted. In some cases, the queue shutdown process may initiate the "switch" (in this case more or less a clean shutdown of the DA queue). One might think that it would be more natural for the DA queue to detect being idle and shut down itself. However, there are some issues associated with that. Most importantly, all queue worker threads need to be shut down during queue destruction. Only after that has happened, final destruction steps can happen (else we would have a myriad of races). However, it is the DA queues worker thread that detects it is empty (empty queue detection always happens at the consumer side and must so). That would lead to the DA queue worker thread to initiate DA queue destruction which in turn would lead to that very same thread being canceled (because workers must shut down before the queue can be destructed). Obviously, this does not work out (and I didn't even mention the other issues - so let's forget about it). As such, the thread that enqueues messages must destruct the queue - and that is the primary queue's DA worker thread. There are some subtleties due to thread synchronization and the fact that the DA consumer may not be running (in a **case-2 startup**). So it is not trivial to reliably change the queue back from DA run mode to regular run mode. The priority is a clean switch. We accept the fact that there may be situations where we cleanly shut down DA run mode, just to re-enable it with the very next message being enqueued. While unlikely, this will happen from time to time and is considered perfectly legal. We can't predict the future and it would introduce too great complexity to try to do something against that (that would most probably even lead to worse performance under regular conditions). The primary queue's DA worker thread may wait at two different places: #. after reaching the low water mark and waiting for either high water or DA queue empty #. at the regular pthread\_cond\_wait() on an empty primary queue Case 2 is unlikely, but may happen (see info above on a case 2 startup). **The DA worker may also not wait at all,** because it is actively executing and shuffling messages between the queues. In that case, however, the program flow passes both of the two wait conditions but simply does not wait. **Finally, the DA worker may be inactive**\ (again, with a case-2 startup). In that case no work(er) at all is executed. Most importantly, without the DA worker being active, nobody will ever detect the need to change back to regular mode. If we have this situation, the very next message enqueued will cause the switch, because then the DA run mode shutdown criteria is met. However, it may take close to eternal for this message to arrive. During that time, disk and memory resources for the DA queue remain allocated. This also leaves processing in a sub-optimal state and it may take longer than necessary to switch back to regular queue mode when a message burst happens. In extreme cases, this could even lead to shutdown of DA run mode, which takes so long that the high water mark is passed and DA run mode is immediately re-initialized - while with an immediate switch, the message burst may have been able to be processed by the in-memory queue without DA support. So in short, it is desirable switch to regular run mode as soon as possible. To do this, we need an active DA worker. The easy solution is to initiate DA worker startup from the DA queue's worker once it detects empty condition. To do so, the DA queue's worker must call into a "*DA worker startup initiation*\ " routine inside the main queue. As a reminder, the DA worker will most probably not receive the "DA queue empty" signal in that case, because it will be long sent (in most cases) before the DA worker even waits for it. So **it is vital that DA run mode termination checks be done in the DA worker before it goes into any wait condition**. Please note that the "*DA worker startup initiation*\ " routine may be called concurrently from multiple initiators. **To prevent a race, it must be guarded by the queue mutex**\ and return without any action (and no error code!) if the DA worker is already initiated. All other cases can be handled by checking the termination criteria immediately at the start of the worker and then once again for each run. The logic follows this simplified flow diagram: .. |image0| image:: queueWorkerLogic.jpg Some of the more subtle aspects of worker processing (e.g. enqueue thread signaling and other fine things) have been left out in order to get the big picture. What is called "check DA mode switchback..." right after "worker init" is actually a check for the worker's termination criteria. Typically, **the worker termination criteria is a shutdown request**. However, **for a DA worker, termination is also requested if the queue size is below the high water mark AND the DA queue is empty**. There is also a third termination criteria and it is not even on the chart: that is the inactivity timeout, which exists in all modes. Note that while the inactivity timeout shuts down a thread, it logically does not terminate the worker pool (or DA worker): workers are restarted on an as-needed basis. However, inactivity timeouts are very important because they require us to restart workers in some situations where we may expect a running one. So always keep them on your mind. Queue Destruction ~~~~~~~~~~~~~~~~~ Now let's consider **the case of destruction of the primary queue.**\ During destruction, our focus is on loosing as few messages as possible. If the queue is not DA-enabled, there is nothing but the configured timeouts to handle that situation. However, with a DA-enabled queue there are more options. If the queue is DA-enabled, it may be *configured to persist messages to disk before it is terminated*. In that case, loss of messages never occurs (at the price of a potentially lengthy shutdown). Even if that setting is not applied, the queue should drain as many messages as possible to the disk. For that reason, it makes no sense to wait on a low water mark. Also, if the queue is already in DA run mode, it does not make any sense to switch back to regular run mode during termination and then try to process some messages via the regular consumer. It is much more appropriate the try completely drain the queue during the remaining timeout period. For the same reason, it is preferred that no new consumers be activated (via the DA queue's worker), as they only cost valuable CPU cycles and, more importantly, would potentially be long(er)-running and possibly be needed to be cancelled. To prevent all of that, **queue parameters are changed for DA-enabled queues:** the high water mark is to 1 and the low water mark to 0 on the primary queue. The DA queue is commanded to run in enqueue-only mode. If the primary queue is *configured to persist messages to disk before it is terminated*, its SHUTDOWN timeout is changed to to eternal. These parameters will cause the queue to drain as much as possible to disk (and they may cause a case 3 DA run mode initiation). Please note that once the primary queue has been drained, the DA queue's worker will automatically switch back to regular (non-DA) run mode. **It must be ensured that no worker cancellation occurs during that switchback**. Please note that the queue may not switch back to regular run mode if it is not *configured to persist messages to disk before it is terminated*. In order to apply the new parameters, **worker threads must be awakened.** Remember we may not be in DA run mode at this stage. In that case, the regular workers must be awakened, which then will switch to DA run mode. No worker may be active, in that case one must be initiated. If in DA run mode and the DA worker is inactive, the  "*DA worker startup initiation*\ " must be called to activate it. That routine ensures only one DA worker is started even with multiple concurrent callers - this may be the case here. The DA queue's worker may have requested DA worker startup in order to terminate on empty queue (which will probably not be honored as we have changed the low water mark). After all this is done, the queue destructor requests termination of the queue's worker threads. It will use the normal timeouts and potentially cancel too-long running worker threads. **The shutdown process must ensure that all workers reach running state before they are commanded to terminate**. Otherwise it may run into a race condition that could lead to a false shutdown with workers running asynchronously. As a few workers may have just been started to initialize (to apply new parameter settings), the probability for this race condition is extremely high, especially on single-CPU systems. After all workers have been shut down (or cancelled), the queue may still be in DA run mode. If so, this must be terminated, which now can simply be done by destructing the DA queue object. This is not a real switchback to regular run mode, but that doesn't matter because the queue object will soon be gone away. Finally, the queue is mostly shut down and ready to be actually destructed. As a last try, the queuePersists() entry point is called. It is used to persists a non-DA-enabled queue in whatever way is possible for that queue. There may be no implementation for the specific queue type. Please note that this is not just a theoretical construct. This is an extremely important code path when the DA queue itself is destructed. Remember that it is a queue object in its own right. The DA queue is obviously not DA-enabled, so it calls into queuePersists() during its destruction - this is what enables us to persist the disk queue! After that point, left over queue resources (mutexes, dynamic memory, ...) are freed and the queue object is actually destructed. Copyright ~~~~~~~~~ Copyright (c) 2008-2014 `Rainer Gerhards `_ and `Adiscon `_. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license can be viewed at `http://www.gnu.org/copyleft/fdl.html `_. build/_sources/development/generic_design.txt0000664000175100017510000002072512411216133020513 0ustar rgerrgerGeneric design of a syslogd --------------------------- Written 2007-04-10 by `Rainer Gerhards `_ The text below describes a generic approach on how a syslogd can be implemented. I created this description for some other project, where it was not used. Instead of throwing it away, I thought it would be a good addition to the rsyslog documentation. While rsyslog differs in details from the description below, it is sufficiently close to it. Further development of rsyslog will probably match it even closer to the description. If you intend to read the rsyslog source code, I recommend reading this document here first. You will not find the same names and not all of the concepts inside rsyslog. However, I think your understanding will benefit from knowing the generic architecture. :: +-----------------+ | "remote" PLOrig | +-----------------+ | I +--------+-----+-----+ +-----+-------+------+-----+ P | PLOrig | GWI | ... | | GWO | Store | Disc | ... | C +--------+-----+-----+ +-----+-------+------+-----+ | | ^ v v | +--------------+ +------------+ +--------------+ | PLGenerator | | RelayEng | | CollectorEng | +--------------+ +------------+ +--------------+ | ^ ^ | | | v v | +-------------+ +------------+ +--------------+ | PLG Ext | | RelEng Ext | | CollcEng Ext | +-------------+ +------------+ +--------------+ | ^ ^ | | | v v | +--------------------------------------------------------------+ | Message Router | +--------------------------------------------------------------+ | ^ v | +--------------------------------------------------------------+ | Message CoDec (e.g. RFC 3164, RFCYYYY) | +--------------------------------------------------------------+ | ^ v | +---------------------+-----------------------+----------------+ | transport UDP | transport TLS | ... | +---------------------+-----------------------+----------------+ Generic Syslog Application Architecture - A "syslog application" is an application whose purpose is the processing of syslog messages. It may be part of a larger application with a broader purpose. An example: a database application might come with its own syslog send subsystem and not go through a central syslog application. In the sense of this document, that application is called a "syslog application" even though a casual observer might correctly call it a database application and may not even know that it supports sending of syslog messages. - Payload is the information that is to be conveyed. Payload by itself may have any format and is totally independent from to format specified in this document. The "Message CoDec" of the syslog application will bring it into the required format. - Payload Originators ("PLOrig") are the original creators of payload. Typically, these are application programs. - A "Remote PLOrig" is a payload originator residing in a different application than the syslog application itself. That application may reside on a different machine and may talk to the syslog application via RPC. - A "PLOrig" is a payload originator residing within the syslog application itself. Typically, this PLOrig emits syslog application startup, shutdown, error and status log messages. - A "GWI" is a inbound gateway. For example, a SNMP-to-syslog gateway may receive SNMP messages and translate them into syslog. - The ellipsis after "GWI" indicates that there are potentially a variety of different other ways to originally generate payload. - A "PLGenerator" is a payload generator. It takes the information from the payload-generating source and integrates it into the syslog subsystem of the application. This is a highly theoretical concept. In practice, there may not actually be any such component. Instead, the payload generators (or other parts like the GWI) may talk directly to the syslog subsystem. Conceptually, the "PLGenerator" is the first component where the information is actually syslog content. - A "PLG Ext" is a payload generator extension. It is used to modify the syslog information. An example of a "PLG Ext" might be the addition of cryptographic signatures to the syslog information. - A "Message Router" is a component that accepts in- and outbound syslog information and routes it to the proper next destination inside the syslog application. The routing information itself is expected to be learnt by operator configuration. - A "Message CoDec" is the message encoder/decoder. The encoder takes syslog information and encodes them into the required format for a syslog message. The decoder takes a syslog message and decodes it into syslog information. Codecs for multiple syslog formats may be present inside a single syslog application. - A transport (UDP, TLS, yet-to-be-defined ones) sends and receives syslog messages. Multiple transports may be used by a single syslog application at the same time. A single transport instance may be used for both sending and receiving. Alternatively, a single instance might be used for sending and receiving exclusively. Multiple instances may be used for different listener ports and receivers. - A "RelayEng" is the relaying engine. It provides functionality necessary for receiving syslog information and sending it to another syslog application. - A "RelEng Ext" is an extension that processes syslog information as it enters or exits a RelayEng. An example of such a component might be a relay cryptographically signing received syslog messages. Such a function might be useful to guarantee authenticity starting from a given point inside a relay chain. - A "CollectorEng" is a collector engine. At this component, syslog information leaves the syslog system and is translated into some other form. After the CollectorEng, the information is no longer defined to be of native syslog type. - A "CollcEng Ext" is a collector engine extension. It modifies syslog information before it is passed on to the CollectorEng. An example for this might be the verification of cryptographically signed syslog message information. Please note that another implementation approach would be to do the verification outside of the syslog application or in a stage after "CollectorEng". - A "GWO" is an outbound gateway. An example of this might be the forwarding of syslog information via SNMP or SMTP. Please note that when a GWO directly connects to a GWI on a different syslog application, no native exchange of syslog information takes place. Instead, the native protocol of these gateways (e.g. SNMP) is used. The syslog information is embedded inside that protocol. Depending on protocol and gateway implementation, some of the native syslog information might be lost. - A "Store" is any way to persistently store the extracted syslog information, e.g. to the file system or to a data base. - "Disc" means the discarding of messages. Operators often find it useful to discard noise messages and so most syslog applications contain a way to do that. - The ellipsis after "Disc" indicates that there are potentially a variety of different other ways to consume syslog information. - There may be multiple instances of each of the described components in a single syslog application. - A syslog application is made up of all or some of the above mentioned components. build/_sources/development/dev_oplugins.txt0000664000175100017510000005124512651410660020254 0ustar rgerrgerWriting Rsyslog Output Plugins ============================== This page is the begin of some developer documentation for writing output plugins. Doing so is quite easy (and that was a design goal), but there currently is only sparse documentation on the process available. I was tempted NOT to write this guide here because I know I will most probably not be able to write a complete guide. However, I finally concluded that it may be better to have some information and pointers than to have nothing. Getting Started and Samples --------------------------- The best to get started with rsyslog plugin development is by looking at existing plugins. All that start with "om" are **o**\ utput **m**\ odules. That means they are primarily thought of being message sinks. In theory, however, output plugins may aggergate other functionality, too. Nobody has taken this route so far so if you would like to do that, it is highly suggested to post your plan on the rsyslog mailing list, first (so that we can offer advise). The rsyslog distribution tarball contains the omstdout plugin which is extremely well targeted for getting started. Just note that this plugin itself is not meant for production use. But it is very simplistic and so a really good starting point to grasp the core ideas. In any case, you should also read the comments in ./runtime/module-template.h. Output plugins are build based on a large set of code-generating macros. These macros handle most of the plumbing needed by the interface. As long as no special callback to rsyslog is needed (it typically is not), an output plugin does not really need to be aware that it is executed by rsyslog. As a plug-in programmer, you can (in most cases) "code as usual". However, all macros and entry points need to be provided and thus reading the code comments in the files mentioned is highly suggested. For testing, you need rsyslog's debugging support. Some useful information is given in "`troubleshooting rsyslog `_ from the doc set. Special Topics -------------- Threading ~~~~~~~~~ Rsyslog uses massive parallel processing and multithreading. However, a plugin's entry points are guaranteed to be never called concurrently **for the same action**. That means your plugin must be able to be called concurrently by two or more threads, but you can be sure that for the same instance no concurrent calls happen. This is guaranteed by the interface specification and the rsyslog core guards against multiple concurrent calls. An instance, in simple words, is one that shares a single instanceData structure. So as long as you do not mess around with global data, you do not need to think about multithreading (and can apply a purely sequential programming methodology). Please note that during the configuration parsing stage of execution, access to global variables for the configuration system is safe. In that stage, the core will only call sequentially into the plugin. Getting Message Data ~~~~~~~~~~~~~~~~~~~~ The doAction() entry point of your plugin is provided with messages to be processed. It will only be activated after filtering and all other conditions, so you do not need to apply any other conditional but can simply process the message. Note that you do NOT receive the full internal representation of the message object. There are various (including historical) reasons for this and, among others, this is a design decision based on security. Your plugin will only receive what the end user has configured in a $template statement. However, starting with 4.1.6, there are two ways of receiving the template content. The default mode, and in most cases sufficient and optimal, is to receive a single string with the expanded template. As I said, this is usually optimal, think about writing things to files, emailing content or forwarding it. The important philosophy is that a plugin should **never** reformat any of such strings - that would either remove the user's ability to fully control message formats or it would lead to duplicating code that is already present in the core. If you need some formatting that is not yet present in the core, suggest it to the rsyslog project, best done by sending a patch ;), and we will try hard to get it into the core (so far, we could accept all such suggestions - no promise, though). If a single string seems not suitable for your application, the plugin can also request access to the template components. The typical use case seems to be databases, where you would like to access properties via specific fields. With that mode, you receive a char \*\* array, where each array element points to one field from the template (from left to right). Fields start at array index 0 and a NULL pointer means you have reached the end of the array (the typical Unix "poor man's linked list in an array" design). Note, however, that each of the individual components is a string. It is not a date stamp, number or whatever, but a string. This is because rsyslog processes strings (from a high-level design look at it) and so this is the natural data type. Feel free to convert to whatever you need, but keep in mind that malformed packets may have lead to field contents you'd never expected... If you like to use the array-based parameter passing method, think that it is only available in rsyslog 4.1.6 and above. If you can accept that your plugin will not be working with previous versions, you do not need to handle pre 4.1.6 cases. However, it would be "nice" if you shut down yourself in these cases - otherwise the older rsyslog core engine will pass you a string where you expect the array of pointers, what most probably results in a segfault. To check whether or not the core supports the functionality, you can use this code sequence: :: BEGINmodInit() rsRetVal localRet; rsRetVal (*pomsrGetSupportedTplOpts)(unsigned long *pOpts); unsigned long opts; int bArrayPassingSupported; /* does core support template passing as an array? */ CODESTARTmodInit *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the current interface specification */ CODEmodInit_QueryRegCFSLineHdlr /* check if the rsyslog core supports parameter passing code */ bArrayPassingSupported = 0; localRet = pHostQueryEtryPt((uchar*)"OMSRgetSupportedTplOpts", &pomsrGetSupportedTplOpts); if(localRet == RS_RET_OK) { /* found entry point, so let's see if core supports array passing */ CHKiRet((*pomsrGetSupportedTplOpts)(&opts)); if(opts & OMSR_TPL_AS_ARRAY) bArrayPassingSupported = 1; } else if(localRet != RS_RET_ENTRY_POINT_NOT_FOUND) { ABORT_FINALIZE(localRet); /* Something else went wrong, what is not acceptable */ } DBGPRINTF("omstdout: array-passing is %ssupported by rsyslog core.\n", bArrayPassingSupported ? "" : "not "); if(!bArrayPassingSupported) { DBGPRINTF("rsyslog core too old, shutting down this plug-in\n"); ABORT_FINALIZE(RS_RET_ERR); } The code first checks if the core supports the OMSRgetSupportedTplOpts() API (which is also not present in all versions!) and, if so, queries the core if the OMSR\_TPL\_AS\_ARRAY mode is supported. If either does not exits, the core is too old for this functionality. The sample snippet above then shuts down, but a plugin may instead just do things different. In omstdout, you can see how a plugin may deal with the situation. **In any case, it is recommended that at least a graceful shutdown is made and the array-passing capability not blindly be used.** In such cases, we can not guard the plugin from segfaulting and if the plugin (as currently always) is run within rsyslog's process space, that results in a segfault for rsyslog. So do not do this. Another possible mode is OMSR\_TPL\_AS\_JSON, where instead of the template a json-c memory object tree is passed to the module. The module can extract data via json-c API calls. It MUST NOT modify the provided structure. This mode is primarily aimed at plugins that need to process tree-like data, as found for example in MongoDB or ElasticSearch. Batching of Messages ~~~~~~~~~~~~~~~~~~~~ Starting with rsyslog 4.3.x, batching of output messages is supported. Previously, only a single-message interface was supported. With the **single message** plugin interface, each message is passed via a separate call to the plugin. Most importantly, the rsyslog engine assumes that each call to the plugin is a complete transaction and as such assumes that messages be properly committed after the plugin returns to the engine. With the **batching** interface, rsyslog employs something along the line of "transactions". Obviously, the rsyslog core can not make non-transactional outputs to be fully transactional. But what it can is support that the output tells the core which messages have been committed by the output and which not yet. The core can than take care of those uncommitted messages when problems occur. For example, if a plugin has received 50 messages but not yet told the core that it committed them, and then returns an error state, the core assumes that all these 50 messages were **not** written to the output. The core then requeues all 50 messages and does the usual retry processing. Once the output plugin tells the core that it is ready again to accept messages, the rsyslog core will provide it with these 50 not yet committed messages again (actually, at this point, the rsyslog core no longer knows that it is re-submitting the messages). If, in contrary, the plugin had told rsyslog that 40 of these 50 messages were committed (before it failed), then only 10 would have been requeued and resubmitted. In order to provide an efficient implementation, there are some (mild) constraints in that transactional model: first of all, rsyslog itself specifies the ultimate transaction boundaries. That is, it tells the plugin when a transaction begins and when it must finish. The plugin is free to commit messages in between, but it **must** commit all work done when the core tells it that the transaction ends. All messages passed in between a begin and end transaction notification are called a batch of messages. They are passed in one by one, just as without transaction support. Note that batch sizes are variable within the range of 1 to a user configured maximum limit. Most importantly, that means that plugins may receive batches of single messages, so they are required to commit each message individually. If the plugin tries to be "smarter" than the rsyslog engine and does not commit messages in those cases (for example), the plugin puts message stream integrity at risk: once rsyslog has notified the plugin of transaction end, it discards all messages as it considers them committed and save. If now something goes wrong, the rsyslog core does not try to recover lost messages (and keep in mind that "goes wrong" includes such uncontrollable things like connection loss to a database server). So it is highly recommended to fully abide to the plugin interface details, even though you may think you can do it better. The second reason for that is that the core engine will have configuration settings that enable the user to tune commit rate to their use-case specific needs. And, as a relief: why would rsyslog ever decide to use batches of one? There is a trivial case and that is when we have very low activity so that no queue of messages builds up, in which case it makes sense to commit work as it arrives. (As a side-note, there are some valid cases where a timeout-based commit feature makes sense. This is also under evaluation and, once decided, the core will offer an interface plus a way to preserve message stream integrity for properly-crafted plugins). The second restriction is that if a plugin makes commits in between (what is perfectly legal) those commits must be in-order. So if a commit is made for message ten out of 50, this means that messages one to nine are also committed. It would be possible to remove this restriction, but we have decided to deliberately introduce it to simplify things. Output Plugin Transaction Interface ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ In order to keep compatible with existing output plugins (and because it introduces no complexity), the transactional plugin interface is build on the traditional non-transactional one. Well... actually the traditional interface was transactional since its introduction, in the sense that each message was processed in its own transaction. So the current ``doAction() entry point can be considered to have this structure (from the transactional interface point of view):`` :: doAction() { beginTransaction() ProcessMessage() endTransaction() } For the **transactional interface**, we now move these implicit ``beginTransaction()`` and ``endTransaction(()`` call out of the message processing body, resulting is such a structure: :: beginTransaction() { /* prepare for transaction */ } doAction() { ProcessMessage() /* maybe do partial commits */ } endTransaction() { /* commit (rest of) batch */ } And this calling structure actually is the transactional interface! It is as simple as this. For the new interface, the core calls a ``beginTransaction()`` entry point inside the plugin at the start of the batch. Similarly, the core call ``endTransaction()`` at the end of the batch. The plugin must implement these entry points according to its needs. But how does the core know when to use the old or the new calling interface? This is rather easy: when loading a plugin, the core queries the plugin for the ``beginTransaction()`` and ``endTransaction()`` entry points. If the plugin supports these, the new interface is used. If the plugin does not support them, the old interface is used and rsyslog implies that a commit is done after each message. Note that there is no special "downlevel" handling necessary to support this. In the case of the non-transactional interface, rsyslog considers each completed call to ``doAction`` as partial commit up to the current message. So implementation inside the core is very straightforward. Actually, **we recommend that the transactional entry points only be defined by those plugins that actually need them**. All others should not define them in which case the default commit behaviour inside rsyslog will apply (thus removing complexity from the plugin). In order to support partial commits, special return codes must be defined for ``doAction``. All those return codes mean that processing completed successfully. But they convey additional information about the commit status as follows: +----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | *RS\_RET\_OK* | The record and all previous inside the batch has been commited. *Note:* this definition is what makes integrating plugins without the transaction being/end calls so easy - this is the traditional "success" return state and if every call returns it, there is no need for actually calling ``endTransaction()``, because there is no transaction open). | +----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | *RS\_RET\_DEFER\_COMMIT* | The record has been processed, but is not yet commited. This is the expected state for transactional-aware plugins. | +----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | *RS\_RET\_PREVIOUS\_COMMITTED* | The **previous** record inside the batch has been committed, but the current one not yet. This state is introduced to support sources that fill up buffers and commit once a buffer is completely filled. That may occur halfway in the next record, so it may be important to be able to tell the engine the everything up to the previouos record is commited | +----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ Note that the typical **calling cycle** is ``beginTransaction()``, followed by *n* times ``doAction()`` followed by ``endTransaction()``. However, if either ``beginTransaction()`` or ``doAction()`` return back an error state (including RS\_RET\_SUSPENDED), then the transaction is considered aborted. In result, the remaining calls in this cycle (e.g. ``endTransaction()``) are never made and a new cycle (starting with ``beginTransaction()`` is begun when processing resumes. So an output plugin must expect and handle those partial cycles gracefully. **The question remains how can a plugin know if the core supports batching?** First of all, even if the engine would not know it, the plugin would return with RS\_RET\_DEFER\_COMMIT, what then would be treated as an error by the engine. This would effectively disable the output, but cause no further harm (but may be harm enough in itself). The real solution is to enable the plugin to query the rsyslog core if this feature is supported or not. At the time of the introduction of batching, no such query-interface exists. So we introduce it with that release. What the means is if a rsyslog core can not provide this query interface, it is a core that was build before batching support was available. So the absence of a query interface indicates that the transactional interface is not available. One might now be tempted to think there is no need to do the actual check, but is is recommended to ask the rsyslog engine explicitly if the transactional interface is present and will be honored. This enables us to create versions in the future which have, for whatever reason we do not yet know, no support for this interface. The logic to do these checks is contained in the ``INITChkCoreFeature`` macro, which can be used as follows: :: INITChkCoreFeature(bCoreSupportsBatching, CORE_FEATURE_BATCHING); Here, bCoreSupportsBatching is a plugin-defined integer which after execution is 1 if batches (and thus the transactional interface) is supported and 0 otherwise. CORE\_FEATURE\_BATCHING is the feature we are interested in. Future versions of rsyslog may contain additional feature-test-macros (you can see all of them in ./runtime/rsyslog.h). Note that the ompsql output plugin supports transactional mode in a hybrid way and thus can be considered good example code. Open Issues ----------- - Processing errors handling - reliable re-queue during error handling and queue termination Licensing ~~~~~~~~~ From the rsyslog point of view, plugins constitute separate projects. As such, we think plugins are not required to be compatible with GPLv3. However, this is no legal advise. If you intend to release something under a non-GPLV3 compatible license it is probably best to consult with your lawyer. Most importantly, and this is definite, the rsyslog team does not expect or require you to contribute your plugin to the rsyslog project (but of course we are happy if you do). Copyright --------- Copyright (c) 2009 `Rainer Gerhards `_ and `Adiscon `_. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license can be viewed at `http://www.gnu.org/copyleft/fdl.html `_. build/_sources/features.txt0000664000175100017510000001623212651410660015047 0ustar rgerrgerRSyslog - Features ================== **This page lists both current features as well as those being considered for future versions of rsyslog.** If you think a feature is missing, drop `Rainer `_ a note. Rsyslog is a vital project. Features are added each few days. If you would like to keep up of what is going on, you can also subscribe to the `rsyslog mailing list `_. A better structured feature list is now contained in our `rsyslog vs. syslog-ng comparison `_. Probably that page will replace this one in the future. Current Features ---------------- - native support for `writing to MySQL databases `_ - native support for writing to Postgres databases - direct support for Firebird/Interbase, OpenTDS (MS SQL, Sybase), SQLLite, Ingres, Oracle, and mSQL via libdbi, a database abstraction layer (almost as good as native) - native support for `sending mail messages `_ (first seen in 3.17.0) - support for (plain) tcp based syslog - much better reliability - support for sending and receiving compressed syslog messages - support for on-demand on-disk spooling of messages that can not be processed fast enough (a great feature for `writing massive amounts of syslog messages to a database `_) - support for selectively `processing messages only during specific timeframes `_ and spooling them to disk otherwise - ability to monitor text files and convert their contents into syslog messages (one per line) - ability to configure backup syslog/database servers - if the primary fails, control is switched to a prioritized list of backups - support for receiving messages via reliable `RFC 3195 `_ delivery (a bit clumsy to build right now...) - ability to generate file names and directories (log targets) dynamically, based on many different properties - control of log output format, including ability to present channel and priority as visible log data - good timestamp format control; at a minimum, ISO 8601/RFC 3339 second-resolution UTC zone - ability to reformat message contents and work with substrings - support for log files larger than 2gb - support for file size limitation and automatic rollover command execution - support for running multiple rsyslogd instances on a single machine - support for `TLS-protected syslog `_ (both `natively `_ and via `stunnel `_) - ability to filter on any part of the message, not just facility and severity - ability to use regular expressions in filters - support for discarding messages based on filters - ability to execute shell scripts on received messages - control of whether the local hostname or the hostname of the origin of the data is shown as the hostname in the output - ability to preserve the original hostname in NAT environments and relay chains - ability to limit the allowed network senders - powerful BSD-style hostname and program name blocks for easy multi-host support - massively multi-threaded with dynamic work thread pools that start up and shut themselves down on an as-needed basis (great for high log volume on multicore machines) - very experimental and volatile support for `syslog-protocol `_ compliant messages (it is volatile because standardization is currently underway and this is a proof-of-concept implementation to aid this effort) - world's first implementation of syslog-transport-tls - the sysklogd's klogd functionality is implemented as the *imklog* input plug-in. So rsyslog is a full replacement for the sysklogd package - support for IPv6 - ability to control repeated line reduction ("last message repeated n times") on a per selector-line basis - supports sub-configuration files, which can be automatically read from directories. Includes are specified in the main configuration file - supports multiple actions per selector/filter condition - MySQL and Postgres SQL functionality as a dynamically loadable plug-in - modular design for inputs and outputs - easily extensible via custom plugins - an easy-to-write to plugin interface - ability to send SNMP trap messages - ability to filter out messages based on sequence of arrival - support for comma-separated-values (CSV) output generation (via the "csv" property replace option). The CSV format supported is that from RFC 4180. - support for arbitrary complex boolean, string and arithmetic expressions in message filters World's first ------------- Rsyslog has an interesting number of "world's firsts" - things that were implemented for the first time ever in rsyslog. Some of them are still features not available elsewhere. - world's first implementation of IETF I-D syslog-protocol (February 2006, version 1.12.2 and above), now RFC5424 - world's first implementation of dynamic syslog on-the-wire compression (December 2006, version 1.13.0 and above) - world's first open-source implementation of a disk-queueing syslogd (January 2008, version 3.11.0 and above) - world's first implementation of IETF I-D syslog-transport-tls (May 2008, version 3.19.0 and above) Upcoming Features ----------------- The list below is something like a repository of ideas we'd like to implement. Features on this list are typically NOT scheduled for immediate inclusion. **Note that we also maintain a `list of features that are looking for sponsors `_. If you are interested in any of these features, or any other feature, you may consider sponsoring the implementation. This is also a great way to show your commitment to the open source community. Plus, it can be financially attractive: just think about how much less it may be to sponsor a feature instead of purchasing a commercial implementation. Also, the benefit of being recognised as a sponsor may even drive new customers to your business!** - port it to more \*nix variants (eg AIX and HP UX) - this needs volunteers with access to those machines and knowledge - pcre filtering - maybe (depending on feedback)  - simple regex already partly added. So far, this seems sufficient so that there is no urgent need to do pcre. If done, it will be a loadable RainerScript function. - support for `RFC 3195 `_ as a sender - this is currently unlikely to happen, because there is no real demand for it. Any work on RFC 3195 has been suspend until we see some real interest in it.  It is probably much better to use TCP-based syslog, which is interoperable with a large number of applications. You may also read my blog post on the future of liblogging, which contains interesting information about the `future of RFC 3195 in rsyslog `_. To see when each feature was added, see the `rsyslog change log `_ (online only). build/_sources/historical/0000775000175100017510000000000012651410761014627 5ustar rgerrgerbuild/_sources/historical/php_syslog_ng.txt0000664000175100017510000002023112411216133020230 0ustar rgerrgerUsing php-syslog-ng with rsyslog ================================ *Written by* `Rainer Gerhards `_ *(2005-08-04)* Note: it has been reported that this guide is somewhat outdated. Most importantly, this guide is for the **original** php-syslog-ng and **cannot** be used for it successor logzilla. Please use the guide with care. Also, please note that **rsyslog's "native" web frontend is** `Adiscon LogAnalyzer `_, which provides best integration and a lot of extra functionality. Abstract -------- **In this paper, I describe how to use** `php-syslog-ng `_ **with** `rsyslogd `_. Php-syslog-ng is a popular web interface to syslog data. Its name stem from the fact that it usually picks up its data from a database created by `syslog-ng `_ and some helper scripts. However, there is nothing syslog-ng specific in the database. With rsyslogd's high customizability, it is easy to write to a syslog-ng like schema. I will tell you how to do this, enabling you to use php-syslog-ng as a front-end for rsyslogd - or save the hassle with syslog-ng database configuration and simply go ahead and use rsyslogd instead.* Overall System Setup -------------------- The setup is pretty straightforward. Basically, php-syslog-ng's interface to the syslogd is the database. We use the schema that php-syslog-ng expects and make rsyslogd write to it in its format. Because of this, php-syslog-ng does not even know there is no syslog-ng present. Setting up the system --------------------- For php-syslog-ng, you can follow its usual setup instructions. Just skip any steps refering to configure syslog-ng. Make sure you create the database schema in `MySQL `_. As of this writing, the expected schema can be created via this script: :: CREATE DATABASE syslog USE syslog CREATE TABLE logs(host varchar(32) default NULL, facility varchar(10) default NULL, priority varchar(10) default NULL, level varchar(10) default NULL, tag varchar(10) default NULL, date date default NULL, time time default NULL, program varchar(15) default NULL, msg text, seq int(10) unsigned NOT NULL auto_increment, PRIMARY KEY (seq), KEY host (host), KEY seq (seq), KEY program (program), KEY time (time), KEY date (date), KEY priority (priority), KEY facility (facility ) TYPE=MyISAM;`` Please note that at the time you are reading this paper, the schema might have changed. Check for any differences. As we customize rsyslogd to the schema, it is vital to have the correct one. If this paper is outdated, `let me know `_ so that I can fix it. Once this schema is created, we simply instruct rsyslogd to store received data in it. I wont go into too much detail here. If you are interested in some more details, you might find my paper "`Writing syslog messages to MySQL `_\ " worth reading. For this article, we simply modify `rsyslog.conf `_\ so that it writes to the database. That is easy. Just these two lines are needed: :: $template syslog-ng,"insert into logs(host, facility, priority, tag, date, time, msg) values ('%HOSTNAME%', %syslogfacility%, %syslogpriority%, '%syslogtag%', '%timereported:::date-mysql%', '%timereported:::date-mysql%', '%msg%')", SQL *.*,           mysql-server,syslog,user,pass;syslog-ng These are just **two** lines. I have color-coded them so that you see what belongs together (the colors have no other meaning). The green line is the actual SQL statement being used to take care of the syslog-ng schema. Rsyslogd allows you to fully control the statement sent to the database. This allows you to write to any database format, including your homegrown one (if you so desire). Please note that there is a small inefficiency in our current usage: the ``'%timereported:::date-mysql%'`` property is used for both the time and the date (if you wonder about what all these funny characters mean, see the `rsyslogd property replacer manual `_) . We could have extracted just the date and time parts of the respective properties. However, this is more complicated and also adds processing time to rsyslogd's processing (substrings must be extracted). So we take a full mysql-formatted timestamp and supply it to MySQL. The sql engine in turn discards the unneeded part. It works pretty well. As of my understanding, the inefficiency of discarding the unneeded part in MySQL is lower than the effciency gain from using the full timestamp in rsyslogd. So it is most probably the best solution. Please note that rsyslogd knows two different timestamp properties: one is timereported, used here. It is the timestamp from the message itself. Sometimes that is a good choice, in other cases not. It depends on your environment. The other one is the timegenerated property. This is the time when rsyslogd received the message. For obvious reasons, that timestamp is consistent, even when your devices are in multiple time zones or their clocks are off. However, it is not "the real thing". It's your choice which one you prefer. If you prefer timegenerated ... simply use it ;) The line in red tells rsyslogd which messages to log and where to store it. The "\*.\*" selects all messages. You can use standard syslog selector line filters here if you do not like to see everything in your database. The ">" tells rsyslogd that a MySQL connection must be established. Then, "mysql-server" is the name or IP address of the server machine, "syslog" is the database name (default from the schema) and "user" and "pass" are the logon credentials. Use a user with low privileges, insert into the logs table is sufficient. "syslog-ng" is the template name and tells rsyslogd to use the SQL statement shown above. Once you have made the changes, all you need to do is restart rsyslogd. Then, you should see syslog messages flow into your database - and show up in php-syslog-ng. Conclusion ---------- With minumal effort, you can use php-syslog-ng together with rsyslogd. For those unfamiliar with syslog-ng, this configuration is probably easier to set up then switching to syslog-ng. For existing rsyslogd users, php-syslog-ng might be a nice add-on to their logging infrastructure. Please note that the `MonitorWare family `_ (to which rsyslog belongs) also offers a web-interface: `Adiscon LogAnalyzer`_. From my point of view, obviously, **phpLogCon is the more natural choice for a web interface to be used together with rsyslog**. It also offers superb functionality and provides, for example,native display of Windows event log entries. I have set up a `demo server `_., You can have a peek at it without installing anything. Feedback Requested ------------------ I would appreciate feedback on this paper. If you have additional ideas, comments or find bugs, please `let me know `_. References and Additional Material ---------------------------------- - `php-syslog-ng `_ Revision History ---------------- - 2005-08-04 \* `Rainer Gerhards `_ \* initial version created Copyright --------- Copyright (c) 2005 `Rainer Gerhards `_ and `Adiscon `_. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license can be viewed at `http://www.gnu.org/copyleft/fdl.html `_. build/_sources/historical/index.txt0000664000175100017510000000045712651410660016503 0ustar rgerrgerHistorical Documents -------------------- This part of the documentation set contains historical documents which are still of interest or may be useful in some more exotic environments. .. toctree:: :maxdepth: 2 php_syslog_ng stunnel multi_ruleset_legacy_format_samples module_devel build/_sources/historical/multi_ruleset_legacy_format_samples.txt0000664000175100017510000001566012651410660024713 0ustar rgerrgerLegacy Format Samples for Multiple Rulesets =========================================== This chapter complements rsyslog's documentation of :doc:`rulesets <../concepts/multi_ruleset>`. While the base document focusses on RainerScript format, it does not provide samples in legacy format. These are included in this document. **Important:** do **not** use legacy ruleset defintions for new configurations. Especially with rulesets, legacy format is extremely hard to get right. The information in this page is included in order to help you understand already existing configurations using the ruleset feature. We even recommend to convert any such configs to RainerScript format because of its increased robustness and simplicity. Legacy ruleset support was available starting with version 4.5.0 and 5.1.1. Split local and remote logging ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Let's say you have a pretty standard system that logs its local messages to the usual bunch of files that are specified in the default rsyslog.conf. As an example, your rsyslog.conf might look like this: :: # ... module loading ... # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* /var/log/maillog # Log cron stuff cron.* /var/log/cron # Everybody gets emergency messages *.emerg * ... more ... Now, you want to add receive messages from a remote system and log these to a special file, but you do not want to have these messages written to the files specified above. The traditional approach is to add a rule in front of all others that filters on the message, processes it and then discards it: :: # ... module loading ... # process remote messages :fromhost-ip, isequal, "192.0.2.1" /var/log/remotefile & ~ # only messages not from 192.0.21 make it past this point # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* /var/log/maillog # Log cron stuff cron.* /var/log/cron # Everybody gets emergency messages *.emerg * ... more ... Note the tilde character, which is the discard action!. Also note that we assume that 192.0.2.1 is the sole remote sender (to keep it simple). With multiple rulesets, we can simply define a dedicated ruleset for the remote reception case and bind it to the receiver. This may be written as follows: :: # ... module loading ... # process remote messages # define new ruleset and add rules to it: $RuleSet remote *.* /var/log/remotefile # only messages not from 192.0.21 make it past this point # bind ruleset to tcp listener $InputTCPServerBindRuleset remote # and activate it: $InputTCPServerRun 10514 # switch back to the default ruleset: $RuleSet RSYSLOG_DefaultRuleset # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* /var/log/maillog # Log cron stuff cron.* /var/log/cron # Everybody gets emergency messages *.emerg * ... more ... Here, we need to switch back to the default ruleset after we have defined our custom one. This is why I recommend a different ordering, which I find more intuitive. The sample below has it, and it leads to the same results: :: # ... module loading ... # at first, this is a copy of the unmodified rsyslog.conf # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* /var/log/maillog # Log cron stuff cron.* /var/log/cron # Everybody gets emergency messages *.emerg * ... more ... # end of the "regular" rsyslog.conf. Now come the new definitions: # process remote messages # define new ruleset and add rules to it: $RuleSet remote *.* /var/log/remotefile # bind ruleset to tcp listener $InputTCPServerBindRuleset remote # and activate it: $InputTCPServerRun 10514 Here, we do not switch back to the default ruleset, because this is not needed as it is completely defined when we begin the "remote" ruleset. Now look at the examples and compare them to the single-ruleset solution. You will notice that we do **not** need a real filter in the multi-ruleset case: we can simply use "\*.\*" as all messages now means all messages that are being processed by this rule set and all of them come in via the TCP receiver! This is what makes using multiple rulesets so much easier. Split local and remote logging for three different ports ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This example is almost like the first one, but it extends it a little bit. While it is very similar, I hope it is different enough to provide a useful example why you may want to have more than two rulesets. Again, we would like to use the "regular" log files for local logging, only. But this time we set up three syslog/tcp listeners, each one listening to a different port (in this example 10514, 10515, and 10516). Logs received from these receivers shall go into different files. Also, logs received from 10516 (and only from that port!) with "mail.\*" priority, shall be written into a specif file and **not** be written to 10516's general log file. This is the config: :: # ... module loading ... # at first, this is a copy of the unmodified rsyslog.conf # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* /var/log/maillog # Log cron stuff cron.* /var/log/cron # Everybody gets emergency messages *.emerg * ... more ... # end of the "regular" rsyslog.conf. Now come the new definitions: # process remote messages #define rulesets first $RuleSet remote10514 *.* /var/log/remote10514 $RuleSet remote10515 *.* /var/log/remote10515 $RuleSet remote10516 mail.* /var/log/mail10516 & ~ # note that the discard-action will prevent this messag from # being written to the remote10516 file - as usual... *.* /var/log/remote10516 # and now define listeners bound to the relevant ruleset $InputTCPServerBindRuleset remote10514 $InputTCPServerRun 10514 $InputTCPServerBindRuleset remote10515 $InputTCPServerRun 10515 $InputTCPServerBindRuleset remote10516 $InputTCPServerRun 10516 Note that the "mail.\*" rule inside the "remote10516" ruleset does not affect processing inside any other rule set, including the default rule set. This documentation is part of the `rsyslog `_ project. Copyright © 2009-2014 by `Rainer Gerhards `_ and `Adiscon `_. Released under the GNU GPL version 3 or higher. build/_sources/historical/module_devel.txt0000664000175100017510000001327612411216133020034 0ustar rgerrgerDeveloping rsyslog modules (outdated) ===================================== *Written by `Rainer Gerhards* `_ *(2007-07-28)* **This document is outdated and primarily contains historical information. Do not trust it to build code. It currently is under review.** This document is incomplete. The module interface is also quite incomplete and under development. Do not currently use it! You may want to visit `Rainer's blog `_ to learn what's going on. Overview -------- In theory, modules provide input and output, among other functions, in rsyslog. In practice, modules are only utilized for output in the current release. The module interface is not yet completed and a moving target. We do not recommend to write a module based on the current specification. If you do, please be prepared that future released of rsyslog will probably break your module. A goal of modularization is to provide an easy to use plug-in interface. However, this goal is not yet reached and all modules must be statically linked. Module "generation" ------------------- There is a lot of plumbing that is always the same in all modules. For example, the interface definitions, answering function pointer queries and such. To get rid of these laborious things, I generate most of them automatically from a single file. This file is named module-template.h. It also contains the current best description of the interface "specification". One thing that can also be achieved with it is the capability to cope with a rapidly changing interface specification. The module interface is evolving. Currently, it is far from being finished. As I moved the monolithic code to modules, I needed (and still need) to make many "non-clean" code hacks, just to get it working. These things are now gradually being removed. However, this requires frequent changes to the interfaces, as things move in and out while working towards a clean interface. All the interim is necessary to reach the goal. This volatility of specifications is the number one reasons I currently advise against implementing your own modules (hint: if you do, be sure to use module-template.h and be prepared to fix newly appearing and disappearing data elements). Naming Conventions ------------------ Source ~~~~~~ Output modules, and only output modules, should start with a file name of "om" (e.g. "omfile.c", "omshell.c"). Similarly, input modules will use "im" and filter modules "fm". The third character shall not be a hyphen. Module Security --------------- Modules are directly loaded into rsyslog's address space. As such, any module is provided a big level of trust. Please note that further module interfaces might provide a way to load a module into an isolated address space. This, however, is far from being completed. So the number one rule about module security is to run only code that you know you can trust. To minimize the security risks associated with modules, rsyslog provides only the most minimalistic access to data structures to its modules. For that reason, the output modules do not receive any direct pointers to the selector\_t structure, the syslogd action structures and - most importantly - the msg structure itself. Access to these structures would enable modules to access data that is none of their business, creating a potential security weakness. Not having access to these structures also simplifies further queueing and error handling cases. As we do not need to provide e.g. full access to the msg object itself, we do not need to serialize and cache it. Instead, strings needed by the module are created by syslogd and then the final result is provided to the module. That, for example, means that in a queued case $NOW is the actual timestamp of when the message was processed, which may be even days before it being dequeued. Think about it: If we wouldn't cache the resulting string, $NOW would be the actual date if the action were suspended and messages queued for some time. That could potentially result in big confusion. It is thought that if an output module actually needs access to the while msg object, we will (then) introduce a way to serialize it (e.g. to XML) in the property replacer. Then, the output module can work with this serialized object. The key point is that output modules never deal directly with msg objects (and other internal structures). Besides security, this also greatly simplifies the job of the output module developer. Action Selectors ---------------- Modules (and rsyslog) need to know when they are called. For this, there must a an action identification in selector lines. There are two syntaxes: the single-character syntax, where a single characters identifies a module (e.g. "\*" for a wall message) and the modules designator syntax, where the module name is given between colons (e.g. ":ommysql:"). The single character syntax is depreciated and should not be used for new plugins. An in-depth discussion of module designation in action selectors can be found in this forum thread: `http://www.rsyslog.com/index.php?name=PNphpBB2&file=viewtopic&p=678#678 `_ Copyright --------- Copyright (c) 2007 `Rainer Gerhards `_ and `Adiscon `_. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license can be viewed at `http://www.gnu.org/copyleft/fdl.html `_. build/_sources/historical/stunnel.txt0000664000175100017510000003350112411216133017051 0ustar rgerrgerSSL Encrypting Syslog with Stunnel ================================== *Written by* `Rainer Gerhards `_ *(2005-07-22)* **HISTORICAL DOCUMENT** **Note: this is an outdated HISTORICAL document.** A much better description on `securing syslog with TLS `_ is available. Abstract -------- **In this paper, I describe how to encrypt** `syslog `_ **messages on the network.** Encryption is vital to keep the confidiental content of syslog messages secure. I describe the overall approach and provide an HOWTO do it with the help of `rsyslogd `_ and `stunnel `_.* Please note that starting with rsyslog 3.19.0, `rsyslog provides native TLS/SSL encryption `_ without the need of stunnel. I strongly recomend to use that feature instead of stunnel. The stunnel documentation here is mostly provided for backwards compatibility. New deployments are advised to use native TLS mode.\ ** Background ---------- **Syslog is a clear-text protocol. That means anyone with a sniffer can have a peek at your data.** In some environments, this is no problem at all. In others, it is a huge setback, probably even preventing deployment of syslog solutions. Thankfully, there is an easy way to encrypt syslog communication. I will describe one approach in this paper. The most straightforward solution would be that the syslogd itself encrypts messages. Unfortuantely, encryption is only standardized in `RFC 3195 `_. But there is currently no syslogd that implements RFC 3195's encryption features, so this route leads to nothing. Another approach would be to use vendor- or project-specific syslog extensions. There are a few around, but the problem here is that they have compatibility issues. However, there is one surprisingly easy and interoperable solution: though not standardized, many vendors and projects implement plain tcp syslog. In a nutshell, plain tcp syslog is a mode where standard syslog messages are transmitted via tcp and records are separated by newline characters. This mode is supported by all major syslogd's (both on Linux/Unix and Windows) as well as log sources (for example, `EventReporter `_ for Windows Event Log forwarding). Plain tcp syslog offers reliability, but it does not offer encryption in itself. However, since it operates on a tcp stream, it is now easy to add encryption. There are various ways to do that. In this paper, I will describe how it is done with stunnel (an other alternative would be `IPSec `_, for example). Stunnel is open source and it is available both for Unix/Linux and Windows. It provides a way to use ssl communication for any non-ssl aware client and server - in this case, our syslogd. Stunnel works much like a wrapper. Both on the client and on the server machine, tunnel portals are created. The non-ssl aware client and server software is configured to not directly talk to the remote partner, but to the local (s)tunnel portal instead. Stunnel, in turn, takes the data received from the client, encrypts it via ssl, sends it to the remote tunnel portal and that remote portal sends it to the recipient process on the remote machine. The transfer to the portals is done via unencrypted communication. As such, it is vital that the portal and the respective program that is talking to it are on the same machine, otherwise data would travel partly unencrypted. Tunneling, as done by stunnel, requires connection oriented communication. This is why you need to use tcp-based syslog. As a side-note, you can also encrypt a plain-text RFC 3195 session via stunnel, though this definitely is not what the protocol designers had on their mind ;) In the rest of this document, I assume that you use rsyslog on both the client and the server. For the samples, I use `Debian `_. Interestingly, there are some annoying differences between stunnel implementations. For example, on Debian a comment line starts with a semicolon (';'). On `Red Hat `_, it starts with a hash sign ('#'). So you need to watch out for subtle issues when setting up your system. Overall System Setup -------------------- In ths paper, I assume two machines, one named "client" and the other named "server". It is obvious that, in practice, you will probably have multiple clients but only one server. Syslog traffic shall be transmitted via stunnel over the network. Port 60514 is to be used for that purpose. The machines are set up as follows: **Client** - rsyslog forwards  message to stunnel local portal at port 61514 - local stunnel forwards data via the network to port 60514 to its remote peer **Server** - stunnel listens on port 60514 to connections from its client peers - all connections are forwarded to the locally-running rsyslog listening at port 61514 Setting up the system --------------------- For Debian, you need the "stunnel4" package. The "stunnel" package is the older 3.x release, which will not support the configuration I describe below. Other distributions might have other names. For example, on Red Hat it is just "stunnel". Make sure that you install the appropriate package on both the client and the server. It is also a good idea to check if there are updates for either stunnel or openssl (which stunnel uses) - there are often security fixes available and often the latest fixes are not included in the default package. In my sample setup, I use only the bare minimum of options. For example, I do not make the server check client cerficiates. Also, I do not talk much about certificates at all. If you intend to really secure your system, you should probably learn about certificates and how to manage and deploy them. This is beyond the scope of this paper. For additional information, `http://www.stunnel.org/faq/certs.html `_ is a good starting point. You also need to install rsyslogd on both machines. Do this before starting with the configuration. You should also familarize yourself with its configuration file syntax, so that you know which actions you can trigger with it. Rsyslogd can work as a drop-in replacement for stock `sysklogd `_. So if you know the standard syslog.conf syntax, you do not need to learn any more to follow this paper. Server Setup ~~~~~~~~~~~~ At the server, you need to have a digital certificate. That certificate enables SSL operation, as it provides the necessary crypto keys being used to secure the connection. Many versions of stunnel come with a default certificate, often found in /etc/stunnel/stunnel.pem. If you have it, it is good for testing only. If you use it in production, it is very easy to break into your secure channel as everybody is able to get hold of your private key. I didn't find an stunnel.pem on my Debian machine. I guess the Debian folks removed it because of its insecurity. You can create your own certificate with a simple openssl tool - you need to do it if you have none and I highly recommend to create one in any case. To create it, cd to /etc/stunnel and type: ``openssl req -new -x509 -days 3650 -nodes -out stunnel.pem -keyout stunnel.pem`` That command will ask you a number of questions. Provide some answer for them. If you are unsure, read `http://www.stunnel.org/faq/certs.html `_. After the command has finished, you should have a usable stunnel.pem in your working directory. Next is to create a configuration file for stunnel. It will direct stunnel what to do. You can use the following basic file: :: ; Certificate/key is needed in server modecert = /etc/stunnel/stunnel.pem; Some debugging stuff useful for troubleshootingdebug = 7foreground=yes [ssyslog] accept = 60514 connect = 61514 Save this file to e.g. /etc/stunnel/syslog-server.conf. Please note that the settings in *italics* are for debugging only. They run stunnel with a lot of debug information in the foreground. This is very valuable while you setup the system - and very useless once everything works well. So be sure to remove these lines when going to production. Finally, you need to start the stunnel daemon. Under Debian, this is done via "stunnel /etc/stunnel/syslog.server.conf". If you have enabled the debug settings, you will immediately see a lot of nice messages. Now you have stunnel running, but it obviously unable to talk to rsyslog - because it is not yet running. If not already done, configure it so that it does everything you want. If in doubt, you can simply copy /etc/syslog.conf to /etc/rsyslog.conf and you probably have what you want. The really important thing in rsyslogd configuration is that you must make it listen to tcp port 61514 (remember: this is where stunnel send the messages to). Thankfully, this is easy to achive: just add "-t 61514" to the rsyslogd startup options in your system startup script. After done so, start (or restart) rsyslogd. The server should now be fully operational. Client Setup ~~~~~~~~~~~~ The client setup is simpler. Most importantly, you do not need a certificate (of course, you can use one if you would like to authenticate the client, but this is beyond the scope of this paper). So the basic thing you need to do is create the stunnel configuration file. :: ; Some debugging stuff useful for troubleshootingdebug = 7foreground=yes client=yes [ssyslog] accept = 127.0.0.1:61514 connect = 192.0.2.1:60514 Again, the text in *italics* is for debugging purposes only. I suggest you leave it in during your initial testing and then remove it. The most important difference to the server configuration outlined above is the "client=yes" directive. It is what makes this stunnel behave like a client. The accept directive binds stunnel only to the local host, so that it is protected from receiving messages from the network (somebody might fake to be the local sender). The address "192.0.2.1" is the address of the server machine. You must change it to match your configuration. Save this file to /etc/stunnel/syslog-client.conf. Then, start stunnel via "stunnel4 /etc/stunnel/syslog-client.conf".  Now you should see some startup messages. If no errors appear, you have a running client stunnel instance. Finally, you need to tell rsyslogd to send data to the remote host. In stock syslogd, you do this via the "@host" forwarding directive. The same works with rsyslog, but it suppports extensions to use tcp. Add the following line to your /etc/rsyslog.conf: :: *.* @@127.0.0.1:61514 Please note the double at-sign (@@). This is no typo. It tells rsyslog to use tcp instead of udp delivery. In this sample, all messages are forwarded to the remote host. Obviously, you may want to limit this via the usual rsyslog.conf settings (if in doubt, use man rsyslog.con). You do not need to add any special startup settings to rsyslog on the client. Start or restart rsyslog so that the new configuration setting takes place. Done ~~~~ After following these steps, you should have a working secure syslog forwarding system. To verify, you can type "logger test" or a similar smart command on the client. It should show up in the respective server log file. If you dig out you sniffer, you should see that the traffic on the wire is actually protected. In the configuration use above, the two stunnel endpoints should be quite chatty, so that you can follow the action going on on your system. If you have only basic security needs, you can probably just remove the debug settings and take the rest of the configuration to production. If you are security-sensitve, you should have a look at the various stunnel settings that help you further secure the system. Preventing Systems from talking directly to the rsyslog Server -------------------------------------------------------------- It is possible that remote systems (or attackers) talk to the rsyslog server by directly connecting to its port 61514. Currently (July of 2005), rsyslog does not offer the ability to bind to the local host, only. This feature is planned, but as long as it is missing, rsyslog must be protected via a firewall. This can easily be done via e.g iptables. Just be sure not to forget it. Conclusion ---------- With minumal effort, you can set up a secure logging infrastructure employing ssl encrypted syslog message transmission. As a side note, you also have the benefit of reliable tcp delivery which is far less prone to message loss than udp. Feedback requested ~~~~~~~~~~~~~~~~~~ I would appreciate feedback on this tutorial. If you have additional ideas, comments or find bugs (I \*do\* bugs - no way... ;)), please `let me know `_. Revision History ---------------- - 2005-07-22 \* `Rainer Gerhards`_ \* Initial Version created - 2005-07-26 \* `Rainer Gerhards`_ \* Some text brush-up, hyperlinks added - 2005-08-03 \* `Rainer Gerhards`_ \* license added - 2008-05-05 \* `Rainer Gerhards`_ \* updated to reflect native TLS capability of rsyslog 3.19.0 and above Copyright --------- Copyright (c) 2008 `Rainer Gerhards`_ and `Adiscon`_. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license can be viewed at `http://www.gnu.org/copyleft/fdl.html `_. This documentation is part of the `rsyslog `_ project. Copyright © 2008 by `Rainer Gerhards`_ and `Adiscon `_. Released under the GNU GPL version 2 or higher. build/_sources/tutorials/0000775000175100017510000000000012651410763014516 5ustar rgerrgerbuild/_sources/tutorials/tls.txt0000664000175100017510000003510112651410660016055 0ustar rgerrgerEncrypting Syslog Traffic with TLS (SSL) [short version] ======================================================== *Written by* `Rainer Gerhards `_ *(2008-05-06)* Abstract -------- **In this paper, I describe how to encrypt** `syslog `_ **messages on the network.** Encryption is vital to keep the confidiental content of syslog messages secure. I describe the overall approach and provide an HOWTO do it with `rsyslog's `_ TLS features.  Please note that TLS is the more secure successor of SSL. While people often talk about "SSL encryption" they actually mean "TLS encryption". So don't look any further if you look for how to SSL-encrypt syslog. You have found the right spot. This is a quick guide. There is a more elaborate guide currently under construction which provides a much more secure environment. It is highly recommended to `at least have a look at it `_. Background ---------- **Traditional syslog is a clear-text protocol. That means anyone with a sniffer can have a peek at your data.** In some environments, this is no problem at all. In others, it is a huge setback, probably even preventing deployment of syslog solutions. Thankfully, there are easy ways to encrypt syslog communication.  The traditional approach involves `running a wrapper like stunnel around the syslog session `_. This works quite well and is in widespread use. However, it is not tightly coupled with the main syslogd and some, even severe, problems can result from this (follow a mailing list thread that describes `total loss of syslog messages due to stunnel mode `_ and the `unreliability of TCP syslog `_). `Rsyslog supports syslog via GSSAP `_\ I since long to overcome these limitatinos. However, syslog via GSSAPI is a rsyslog-exclusive transfer mode and it requires a proper Kerberos environment. As such, it isn't a really universal solution. The `IETF `_ has begun standardizing syslog over plain tcp over TLS for a while now. While I am not fully satisfied with the results so far, this obviously has the  potential to become the long-term solution. The Internet Draft in question, syslog-transport-tls has been dormant for some time but is now (May of 2008) again being worked on. I expect it to turn into a RFC within the next 12 month (but don't take this for granted ;)). I didn't want to wait for it, because there obviously is need for TLS syslog right now (and, honestly, I have waited long enough...). Consequently, I have implemented the current draft, with some interpretations I made (there will be a compliance doc soon). So in essence, a TLS-protected syslog transfer mode is available right now. As a side-note, Rsyslog is the world's first implementation of syslog-transport-tls. Please note that in theory it should be compatible with other, non IETF syslog-transport-tls implementations. If you would like to run it with something else, please let us know so that we can create a compatibility list (and implement compatbility where it doesn't yet exist).  Overall System Setup -------------------- Encryption requires a reliable stream. So It will not work over UDP syslog. In rsyslog, network transports utilize a so-called "network stream layer" (netstream for short). This layer provides a unified view of the transport to the application layer. The plain TCP syslog sender and receiver are the upper layer. The driver layer currently consists of the "ptcp" and "gtls" library plugins. "ptcp" stands for "plain tcp" and is used for unencrypted message transfer. It is also used internally by the gtls driver, so it must always be present on a system. The "gtls" driver is for GnutTLS, a TLS library. It is used for encrypted message transfer. In the future, additional drivers will become available (most importantly, we would like to include a driver for NSS). What you need to do to build an encrypted syslog channel is to simply use the proper netstream drivers on both the client and the server. Client, in the sense of this document, is the rsyslog system that is sending syslog messages to a remote (central) loghost, which is called the server. In short, the setup is as follows: **Client** - forwards messages via plain tcp syslog using gtls netstream driver to central server on port 10514 **Server** - accept incoming messages via plain tcp syslog using gtls netstream driver on port 10514 Setting up the system --------------------- Server Setup ~~~~~~~~~~~~ At the server, you need to have a digital certificate. That certificate enables SSL operation, as it provides the necessary crypto keys being used to secure the connection. There is a set of default certificates in ./contrib/gnutls. These are key.pem and cert.pem. These are good for testing. If you use it in production, it is very easy to break into your secure channel as everybody is able to get hold of your private key. So it is a good idea to generate the key and certificate yourself. You also need a root CA certificate. Again, there is a sample CA certificate in ./contrib/gnutls, named ca.cert. It is suggested to generate your own. To configure the server, you need to tell it where are its certificate files, to use the gtls driver and start up a listener. This is done as follows: :: # make gtls driver the default $DefaultNetstreamDriver gtls # certificate files $DefaultNetstreamDriverCAFile /path/to/contrib/gnutls/ca.pem $DefaultNetstreamDriverCertFile /path/to/contrib/gnutls/cert.pem $DefaultNetstreamDriverKeyFile /path/to/contrib/gnutls/key.pem $ModLoad imtcp # load TCP listener $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode $InputTCPServerStreamDriverAuthMode anon # client is NOT authenticated $InputTCPServerRun 10514 # start up listener at port 10514 This is all you need to do. You can use the rest of your rsyslog.conf together with this configuration. The way messages are received does not interfer with any other option, so you are able to do anything else you like without any restrictions. Restart rsyslogd. The server should now be fully operational. Client Setup ~~~~~~~~~~~~ The client setup is equally simple. You need less certificates, just the CA cert.  :: # certificate files - just CA for a client $DefaultNetstreamDriverCAFile /path/to/contrib/gnutls/ca.pem # set up the action $DefaultNetstreamDriver gtls # use gtls netstream driver $ActionSendStreamDriverMode 1 # require TLS for the connection $ActionSendStreamDriverAuthMode anon # server is NOT authenticated *.* @@(o)server.example.net:10514 # send (all) messages Note that we use the regular TCP forwarding syntax (@@) here. There is nothing special, because the encryption is handled by the netstream driver. So I have just forwarded every message (\*.\*) for simplicity - you can use any of rsyslog's filtering capabilities (like epxression-based filters or regular expressions). Note that the "(o)" part is not strictly necessary. It selects octet-based framing, which provides compatiblity to IETF's syslog-transport-tls draft. Besides compatibility, this is also a more reliable transfer mode, so I suggest to always use it. Done ~~~~ After following these steps, you should have a working secure syslog forwarding system. To verify, you can type "logger test" or a similar "smart" command on the client. It should show up in the respective server log file. If you dig out your sniffer, you should see that the traffic on the wire is actually protected. Limitations ~~~~~~~~~~~ The RELP transport can currently not be protected by TLS. A work-around is to use stunnel. TLS support for RELP will be added once plain TCP syslog has sufficiently matured and there either is some time left to do this or we find a sponsor ;). Certificates ------------ In order to be really secure, certificates are needed. This is a short summary on how to generate the necessary certificates with GnuTLS' certtool. You can also generate certificates via other tools, but as we currently support GnuTLS as the only TLS library, we thought it is a good idea to use their tools. Note that this section aims at people who are not involved with PKI at all. The main goal is to get them going in a reasonable secure way.  CA Certificate ~~~~~~~~~~~~~~ This is used to sign all of your other certificates. The CA cert must be trusted by all clients and servers. The private key must be well-protected and not given to any third parties. The certificate itself can (and must) be distributed. To generate it, do the following: #. generate the private key: :: certtool --generate-privkey --outfile ca-key.pem This takes a short while. Be sure to do some work on your workstation, it waits for radom input. Switching between windows is sufficient ;) #. now create the (self-signed) CA certificate itself: :: certtool --generate-self-signed --load-privkey ca-key.pem --outfile ca.pem This generates the CA certificate. This command queries you for a number of things. Use appropriate responses. When it comes to certificate validity, keep in mind that you need to recreate all certificates when this one expires. So it may be a good idea to use a long period, eg. 3650 days (roughly 10 years). You need to specify that the certificates belongs to an authrity. The certificate is used to sign other certificates. #. You need to distribute this certificate to all peers and you need to point to it via the $DefaultNetstreamDriverCAFile config directive. All other certificates will be issued by this CA. Important: do only distribute the ca.pem, NOT ca-key.pem (the private key). Distributing the CA private key would totally breach security as everybody could issue new certificates on the behalf of this CA. Individual Peer Certificate ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Each peer (be it client, server or both), needs a certificate that conveys its identity. Access control is based on these certificates. You can, for example, configure a server to accept connections only from configured clients. The client ID is taken from the client instances certificate. So as a general rule of thumb, you need to create a certificate for each instance of rsyslogd that you run. That instance also needs the private key, so that it can properly decrypt the traffic. Safeguard the peer's private key file. If somebody gets hold of it, it can malicously pretend to be the compromised host. If such happens, regenerate the certificate and make sure you use a different name instead of the compromised one (if you use name-based authentication).  These are the steps to generate the indivudual certificates (repeat: you need to do this for every instance, do NOT share the certificates created in this step): #. generate a private key (do NOT mistake this with the CA's private key - this one is different): :: certtool --generate-privkey --outfile key.pem Again, this takes a short while. #. generate a certificate request: :: certtool --generate-request --load-privkey key.pem --outfile request.pem If you do not have the CA's private key (because you are not authorized for this), you can send the certificate request to the responsible person. If you do this, you can skip the remaining steps, as the CA will provide you with the final certificate. If you submit the request to the CA, you need to tell the CA the answers that you would normally provide in step 3 below. #. Sign (validate, authorize) the certificate request and generate the instances certificate. You need to have the CA's certificate and private key for this: :: certtool --generate-certificate --load-request request.pem --outfile cert.pem \ --load-ca-certificate ca.pem --load-ca-privkey ca-key.pem Answer questions as follows: Cert does not belogn to an authority; it is a TLS web server and client certificate; the dnsName MUST be the name of the peer in question (e.g. centralserver.example.net) - this is the name used for authenticating the peers. Please note that you may use an IP address in dnsName. This is a good idea if you would like to use default server authentication and you use selector lines with IP addresses (e.g. "\*.\* @@192.168.0.1") - in that case you need to select a dnsName of 192.168.0.1. But, of course, changing the server IP then requires generating a new certificate. After you have generated the certificate, you need to place it onto the local machine running rsyslogd. Specify the certificate and key via the $DefaultNetstreamDriverCertFile /path/to/cert.pem and $DefaultNetstreamDriverKeyFile /path/to/key.pem configuration directives. Make sure that nobody has access to key.pem, as that would breach security. And, once again: do NOT use these files on more than one instance. Doing so would prevent you from distinguising between the instances and thus would disable useful authentication. Troubleshooting Certificates ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If you experience trouble with your certificate setup, it may be useful to get some information on what is contained in a specific certificate (file). To obtain that information, do  :: $ certtool --certificate-info --infile cert.pem where "cert.pem" can be replaced by the various certificate pem files (but it does not work with the key files). Conclusion ---------- With minumal effort, you can set up a secure logging infrastructure employing TLS encrypted syslog message transmission. Feedback requested ~~~~~~~~~~~~~~~~~~ I would appreciate feedback on this tutorial. If you have additional ideas, comments or find bugs (I \*do\* bugs - no way... ;)), please `let me know `_. Revision History ---------------- - 2008-05-06 \* `Rainer Gerhards`_ \* Initial Version created - 2008-05-26 \* `Rainer Gerhards`_ \* added information about certificates Copyright --------- Copyright (c) 2008-2014 `Rainer Gerhards`_ and `Adiscon `_. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license can be viewed at `http://www.gnu.org/copyleft/fdl.html `_. [`rsyslog site `_\ ] build/_sources/tutorials/log_rotation_fix_size.txt0000664000175100017510000000450712651410660021661 0ustar rgerrgerLog rotation with rsyslog ========================= *Written by Michael Meckelein* Situation --------- Your environment does not allow you to store tons of logs? You have limited disc space available for logging, for example you want to log to a 124 MB RAM usb stick? Or you do not want to keep all the logs for months, logs from the last days is sufficient? Think about log rotation. Log rotation based on a fixed log size -------------------------------------- This small but hopefully useful article will show you the way to keep your logs at a given size. The following sample is based on rsyslog illustrating a simple but effective log rotation with a maximum size condition. Use Output Channels for fixed-length syslog files ------------------------------------------------- Lets assume you do not want to spend more than 100 MB hard disc space for you logs. With rsyslog you can configure Output Channels to achieve this. Putting the following directive :: # start log rotation via outchannel # outchannel definition $outchannel log_rotation,/var/log/log_rotation.log, 52428800,/home/me/./log_rotation_script # activate the channel and log everything to it *.* :omfile:$log_rotation # end log rotation via outchannel to ryslog.conf instruct rsyslog to log everything to the destination file '/var/log/log\_rotation.log' until the give file size of 50 MB is reached. If the max file size is reached it will perform an action. In our case it executes the script /home/me/log\_rotation\_script which contains a single command: :: mv -f /var/log/log_rotation.log /var/log/log_rotation.log.1 This moves the original log to a kind of backup log file. After the action was successfully performed rsyslog creates a new /var/log/log\_rotation.log file and fill it up with new logs. So the latest logs are always in log\_rotation.log. Conclusion ---------- With this approach two files for logging are used, each with a maximum size of 50 MB. So we can say we have successfully configured a log rotation which satisfies our requirement. We keep the logs at a fixed-size level of 100 MB. This documentation is part of the `rsyslog `_ project. Copyright © 2008 by `Rainer Gerhards `_ and `Adiscon `_. Released under the GNU GPL version 2 or higher. build/_sources/tutorials/database.txt0000664000175100017510000003717412411216133017024 0ustar rgerrgerWriting syslog messages to MySQL, PostgreSQL or any other supported Database ============================================================================ *Written by* \ `Rainer Gerhards `_\ *with some additions by Marc Schiffbauer (2008-02-28)* Abstract -------- ***In this paper, I describe how to write***\ `syslog `_\ ***messages to a***\ `MySQL `_\ ***or***\ `PostgreSQL `_\ ***database.***\ *Having syslog messages in a database is often handy, especially when you intend to set up a front-end for viewing them. This paper describes an approach with*\ `rsyslogd `_\ *, an alternative enhanced syslog daemon natively supporting MySQL and PostgreSQL. I describe the components needed to be installed and how to configure them. Please note that as of this writing, rsyslog supports a variety of databases. While this guide is still MySQL- and PostgreSQL-focused, you can probably use it together with other ones too. You just need to modify a few settings.* Background ---------- In many cases, syslog data is simply written to text files. This approach has some advantages, most notably it is very fast and efficient. However, data stored in text files is not readily accessible for real-time viewing and analysis. To do that, the messages need to be in a database. There are various ways to store syslog messages in a database. For example, some have the syslogd write text files which are later feed via a separate script into the database. Others have written scripts taking the data (via a pipe) from a non-database-aware syslogd and store them as they appear. Some others use database-aware syslogds and make them write the data directly to the database. In this paper, I use that "direct write" approach. I think it is superior, because the syslogd itself knows the status of the database connection and thus can handle it intelligently (well ... hopefully ;)). I use rsyslogd to acomplish this, simply because I have initiated the rsyslog project with database-awareness as one goal. **One word of caution:** while message storage in the database provides an excellent foundation for interactive analysis, it comes at a cost. Database i/o is considerably slower than text file i/o. As such, directly writing to the database makes sense only if your message volume is low enough to allow a) the syslogd, b) the network, and c) the database server to catch up with it. Some time ago, I have written a paper on `optimizing syslog server performance `_. While this paper talks about Window-based solutions, the ideas in it are generic enough to apply here, too. So it might be worth reading if you anticipate medium high to high traffic. If you anticipate really high traffic (or very large traffic spikes), you should seriously consider forgetting about direct database writes - in my opinion, such a situation needs either a very specialized system or a different approach (the text-file-to-database approach might work better for you in this case). Overall System Setup -------------------- In this paper, I concentrate on the server side. If you are thinking about interactive syslog message review, you probably want to centralize syslog. In such a scenario, you have multiple machines (the so-called clients) send their data to a central machine (called server in this context). While I expect such a setup to be typical when you are interested in storing messages in the database, I do not describe how to set it up. This is beyond the scope of this paper. If you search a little, you will probably find many good descriptions on how to centralize syslog. If you do that, it might be a good idea to do it securely, so you might also be interested in my paper on `ssl-encrypting syslog message transfer `_. No matter how the messages arrive at the server, their processing is always the same. So you can use this paper in combination with any description for centralized syslog reporting. As I already said, I use rsyslogd on the server. It has intrinsic support for talking to the supported databases. For obvious reasons, we also need an instance of MySQL or PostgreSQL running. To keep us focused, the setup of the database itself is also beyond the scope of this paper. I assume that you have successfully installed the database and also have a front-end at hand to work with it (for example, `phpMyAdmin `_ or `phpPgAdmin `_. Please make sure that this is installed, actually working and you have a basic understanding of how to handle it. Setting up the system --------------------- You need to download and install rsyslogd first. Obtain it from the `rsyslog site `_. Make sure that you disable stock syslogd, otherwise you will experience some difficulties. On some distributions  (Fedora 8 and above, for example), rsyslog may already by the default syslogd, in which case you obviously do not need to do anything specific. For many others, there are prebuild packages available. If you use either, please make sure that you have the required database plugins for your database available. It usually is a separate package and typically **not** installed by default. It is important to understand how rsyslogd talks to the database. In rsyslogd, there is the concept of "templates". Basically, a template is a string that includes some replacement characters, which are called "properties" in rsyslog. Properties are accessed via the "`Property Replacer `_\ ". Simply said, you access properties by including their name between percent signs inside the template. For example, if the syslog message is "Test", the template "%msg%" would be expanded to "Test". Rsyslogd supports sending template text as a SQL statement to the database. As such, the template must be a valid SQL statement. There is no limit in what the statement might be, but there are some obvious and not so obvious choices. For example, a template "drop table xxx" is possible, but does not make an awful lot of sense. In practice, you will always use an "insert" statement inside the template. An example: if you would just like to store the msg part of the full syslog message, you have probably created a table "syslog" with a single column "message". In such a case, a good template would be "insert into syslog(message) values ('%msg%')". With the example above, that would be expanded to "insert into syslog(message) values('Test')". This expanded string is then sent to the database. It's that easy, no special magic. The only thing you must ensure is that your template expands to a proper SQL statement and that this statement matches your database design. Does that mean you need to create database schema yourself and also must fully understand rsyslogd's properties? No, that's not needed. Because we anticipated that folks are probably more interested in getting things going instead of designing them from scratch. So we have provided a default schema as well as build-in support for it. This schema also offers an additional benefit: rsyslog is part of `Adiscon `_'s `MonitorWare product line `_ (which includes open source and closed source members). All of these tools share the same default schema and know how to operate on it. For this reason, the default schema is also called the "MonitorWare Schema". If you use it, you can simply add `phpLogCon, a GPLed syslog web interface `_, to your system and have instant interactive access to your database. So there are some benefits in using the provided schema. The schema definition is contained in the file "createDB.sql". It comes with the rsyslog package and one can be found for each supported database type (in the plugins directory). Review it to check that the database name is acceptable for you. Be sure to leave the table and field names unmodified, because otherwise you need to customize rsyslogd's default sql template, which we do not do in this paper. Then, run the script with your favorite SQL client. Double-check that the table was successfully created. It is important to note that the correct database encoding must be used so that the database will accept strings independend of the string encoding. This is an important part because it can not be guarantied that all syslog messages will have a defined character encoding. This is especially true if the rsyslog-Server will collect messages from different clients and different products. For example PostgreSQL may refuse to accept messages if you would set the database encoding to “UTF8” while a client is sending invalid byte sequences for that encoding. Database support in rsyslog is integrated via loadable plugin modules. To use the database functionality, the database plugin must be enabled in the config file BEFORE the first database table action is used. This is done by placing the ``$ModLoad ommysql`` directive at the begining of /etc/rsyslog.conf for MySQL and ``$ModLoad ompgsql`` ``for PostgreSQL.`` For other databases, use their plugin name (e.g. omoracle). Next, we need to tell rsyslogd to write data to the database. As we use the default schema, we do NOT need to define a template for this. We can use the hardcoded one (rsyslogd handles the proper template linking). So all we need to do e.g. for MySQL is add a simple selector line to /etc/rsyslog.conf: ``*.*       :ommysql:database-server,database-name,database-userid,database-password`` Again, other databases have other selector names, e.g. ":ompgsql:" instead of ":ommysql:". See the output plugin's documentation for details. In many cases, the database will run on the local machine. In this case, you can simply use "127.0.0.1" for *database-server*. This can be especially advisable, if you do not need to expose the database to any process outside of the local machine. In this case, you can simply bind it to 127.0.0.1, which provides a quite secure setup. Of course, rsyslog also supports remote database instances. In that case, use the remote server name (e.g. mydb.example.com) or IP-address. The *database-name* by default is "Syslog". If you have modified the default, use your name here. *Database-userid* and *-password* are the credentials used to connect to the database. As they are stored in clear text in rsyslog.conf, that user should have only the least possible privileges. It is sufficient to grant it INSERT privileges to the systemevents table, only. As a side note, it is strongly advisable to make the rsyslog.conf file readable by root only - if you make it world-readable, everybody could obtain the password (and eventually other vital information from it). In our example, let's assume you have created a database user named "syslogwriter" with a password of "topsecret" (just to say it bluntly: such a password is NOT a good idea...). If your database is on the local machine, your rsyslog.conf line might look like in this sample: ``*.*       :ommysql:127.0.0.1,Syslog,syslogwriter,topsecret`` Save rsyslog.conf, restart rsyslogd - and you should see syslog messages being stored in the "systemevents" table! The example line stores every message to the database. Especially if you have a high traffic volume, you will probably limit the amount of messages being logged. This is easy to accomplish: the "write database" action is just a regular selector line. As such, you can apply normal selector-line filtering. If, for example, you are only interested in messages from the mail subsystem, you can use the following selector line: ``mail.*       :ommysql:127.0.0.1,syslog,syslogwriter,topsecret`` Review the `rsyslog.conf `_ documentation for details on selector lines and their filtering. **You have now completed everything necessary to store syslog messages to the a database.** If you would like to try out a front-end, you might want to look at `phpLogCon `_, which displays syslog data in a browser. As of this writing, phpLogCon is not yet a powerful tool, but it's open source, so it might be a starting point for your own solution. On Reliability... ----------------- Rsyslogd writes syslog messages directly to the database. This implies that the database must be available at the time of message arrival. If the database is offline, no space is left or something else goes wrong - rsyslogd can not write the database record. If rsyslogd is unable to store a message, it performs one retry. This is helpful if the database server was restarted. In this case, the previous connection was broken but a reconnect immediately succeeds. However, if the database is down for an extended period of time, an immediate retry does not help. Message loss in this scenario can easily be prevented with rsyslog. All you need to do is run the database writer in queued mode. This is now described in a generic way and I do not intend to duplicate it here. So please be sure to read "`Handling a massive syslog database insert rate with Rsyslog `_\ ", which describes the scenario and also includes configuration examples. Conclusion ---------- With minimal effort, you can use rsyslogd to write syslog messages to a database. You can even make it absolutely fail-safe and protect it against database server downtime. Once the messages are arrived there, you can interactively review and analyze them. In practice, the messages are also stored in text files for longer-term archival and the databases are cleared out after some time (to avoid becoming too slow). If you expect an extremely high syslog message volume, storing it in real-time to the database may outperform your database server. In such cases, either filter out some messages or used queued mode (which in general is recommended with databases). The method outlined in this paper provides an easy to setup and maintain solution for most use cases. Feedback Requested ~~~~~~~~~~~~~~~~~~ I would appreciate feedback on this paper. If you have additional ideas, comments or find bugs, please `let me know `_. References and Additional Material ---------------------------------- - `www.rsyslog.com `_ - the rsyslog site - `Paper on Syslog Server Optimization `_ Revision History ---------------- - 2005-08-02 \* `Rainer Gerhards `_ \* initial version created - 2005-08-03 \* `Rainer Gerhards `_ \* added references to demo site - 2007-06-13 \* `Rainer Gerhards `_ \* removed demo site - was torn down because too expensive for usage count - 2008-02-21 \* `Rainer Gerhards `_ \* updated reliability section, can now be done with on-demand disk queues - 2008-02-28 \* `Rainer Gerhards `_ \* added info on other databases, updated syntax to more recent one - 2010-01-29 \* Marc Schiffbauer \* added some PostgreSQL stuff, made wording more database generic, fixed some typos Copyright --------- Copyright (c) 2005-2010 `Rainer Gerhards `_, Marc Schiffbauer and `Adiscon `_. build/_sources/tutorials/high_database_rate.txt0000664000175100017510000001715712651410660021044 0ustar rgerrgerHandling a massive syslog database insert rate with Rsyslog =========================================================== *Written by* `Rainer Gerhards `_ *(2008-01-31)* Abstract -------- **In this paper, I describe how log massive amounts of** `syslog `_ **messages to a database.**\ This HOWTO is currently under development and thus a bit brief. Updates are promised ;).* The Intention ------------- Database updates are inherently slow when it comes to storing syslog messages. However, there are a number of applications where it is handy to have the message inside a database. Rsyslog supports native database writing via output plugins. As of this writing, there are plugins available for MySQL an PostgreSQL. Maybe additional plugins have become available by the time you read this. Be sure to check. In order to successfully write messages to a database backend, the backend must be capable to record messages at the expected average arrival rate. This is the rate if you take all messages that can arrive within a day and divide it by 86400 (the number of seconds per day). Let's say you expect 43,200,000 messages per day. That's an average rate of 500 messages per second (mps). Your database server MUST be able to handle that amount of message per second on a sustained rate. If it doesn't, you either need to add an additional server, lower the number of message - or forget about it. However, this is probably not your peak rate. Let's simply assume your systems work only half a day, that's 12 hours (and, yes, I know this is unrealistic, but you'll get the point soon). So your average rate is actually 1,000 mps during work hours and 0 mps during non-work hours. To make matters worse, workload is not divided evenly during the day. So you may have peaks of up to 10,000mps while at other times the load may go down to maybe just 100mps. Peaks may stay well above 2,000mps for a few minutes. So how the hack you will be able to handle all of this traffic (including the peaks) with a database server that is just capable of inserting a maximum of 500mps? The key here is buffering. Messages that the database server is not capable to handle will be buffered until it is. Of course, that means database insert are NOT real-time. If you need real-time inserts, you need to make sure your database server can handle traffic at the actual peak rate. But lets assume you are OK with some delay. Buffering is fine. But how about these massive amounts of data? That can't be hold in memory, so don't we run out of luck with buffering? The key here is that rsyslog can not only buffer in memory but also buffer to disk (this may remind you of "spooling" which gets you the right idea). There are several queuing modes available, offering differnent throughput. In general, the idea is to buffer in memory until the memory buffer is exhausted and switch to disk-buffering when needed (and only as long as needed). All of this is handled automatically and transparently by rsyslog. With our above scenario, the disk buffer would build up during the day and rsyslog would use the night to drain it. Obviously, this is an extreme example, but it shows what can be done. Please note that queue content survies rsyslogd restarts, so even a reboot of the system will not cause any message loss. How To Setup ------------ Frankly, it's quite easy. You just need to do is instruct rsyslog to use a disk queue and then configure your action. There is nothing else to do. With the following simple config file, you log anything you receive to a MySQL database and have buffering applied automatically. :: $ModLoad ommysql # load the output driver (use ompgsql for PostgreSQL) $ModLoad imudp # network reception $UDPServerRun 514 # start a udp server at port 514 $ModLoad imuxsock # local message reception $WorkDirectory /rsyslog/work # default location for work (spool) files $MainMsgQueueFileName mainq # set file name, also enables disk mode $ActionResumeRetryCount -1 # infinite retries on insert failure #for PostgreSQL replace :ommysql: by :ompgsql: below: *.* :ommysql:hostname,dbname,userid,password; The simple setup above has one drawback: the write database action is executed together with all other actions. Typically, local files are also written. These local file writes are now bound to the speed of the database action. So if the database is down, or threre is a large backlog, local files are also not (or late) written. **There is an easy way to avoid this with rsyslog.** It involves a slightly more complicated setup. In rsyslog, each action can utilize its own queue. If so, messages are simply pulled over from the main queue and then the action queue handles action processing on its own. This way, main processing and the action are de-coupled. In the above example, this means that local file writes will happen immediately while the database writes are queued. As a side-note, each action can have its own queue, so if you would like to more than a single database or send messages reliably to another host, you can do all of this on their own queues, de-coupling their processing speeds. The configuration for the de-coupled database write involves just a few more commands: :: $ModLoad ommysql # load the output driver (use ompgsql for PostgreSQL) $ModLoad imudp # network reception $UDPServerRun 514 # start a udp server at port 514 $ModLoad imuxsock # local message reception $WorkDirectory /rsyslog/work # default location for work (spool) files $ActionQueueType LinkedList # use asynchronous processing $ActionQueueFileName dbq # set file name, also enables disk mode $ActionResumeRetryCount -1 # infinite retries on insert failure # for PostgreSQL replace :ommysql: by :ompgsql: below: *.* :ommysql:hostname,dbname,userid,password; **This is the recommended configuration for this use case.** It requires rsyslog 3.11.0 or above. In this example, the main message queue is NOT disk-assisted (there is no $MainMsgQueueFileName directive). We still could do that, but have not done it because there seems to be no need. The only slow running action is the database writer and it has its own queue. So there is no real reason to use a large main message queue (except, of course, if you expect \*really\* heavy traffic bursts). Note that you can modify a lot of queue performance parameters, but the above config will get you going with default values. If you consider using this on a real busy server, it is strongly recommended to invest some time in setting the tuning parameters to appropriate values. Feedback requested ~~~~~~~~~~~~~~~~~~ I would appreciate feedback on this tutorial. If you have additional ideas, comments or find bugs (I \*do\* bugs - no way... ;)), please `let me know `_. Revision History ---------------- - 2008-01-28 \* `Rainer Gerhards`_ \* Initial Version created - 2008-01-28 \* `Rainer Gerhards`_ \* Updated to new v3.11.0 capabilities Copyright --------- Copyright (c) 2008 Rainer Gerhards and Adiscon. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license can be viewed at `http://www.gnu.org/copyleft/fdl.html `_. This documentation is part of the `rsyslog `_ project. Copyright © 2008 by `Rainer Gerhards`_ and `Adiscon `_. Released under the GNU GPL version 2 or higher. build/_sources/tutorials/recording_pri.txt0000664000175100017510000001645412651410660020113 0ustar rgerrgerRecording the Priority of Syslog Messages ========================================= *Written by* `Rainer Gerhards `_ *(2007-06-18)* Abstract -------- **The so-called priority (PRI) is very important in syslog messages, because almost all filtering in syslog.conf is based on it.** However, many syslogds (including the Linux stock sysklogd) do not provide a way to record that value. In this article, I'll give a brief overview of how PRI can be written to a log file. Background ---------- The PRI value is a combination of so-called severity and facility. The facility indicates where the message originated from (e.g. kernel, mail subsystem) while the severity provides a glimpse of how important the message might be (e.g. error or informational). Be careful with these values: they are in no way consistent across applications (especially severity). However, they still form the basis of most filtering in syslog.conf. For example, the directive (aka "selector line) :: mail.* /var/log/mail.log means that messages with the mail facility should be stored to /var/log/mail.log, no matter which severity indicator they have (that is telling us the asterisk). If you set up complex conditions, it can be annoying to find out which PRI value a specific syslog message has. Most stock syslogds do not provide any way to record them. How is it done? --------------- With `rsyslog `_, PRI recording is simple. All you need is the correct template. Even if you do not use rsyslog on a regular basis, it might be a handy tool for finding out the priority. Rsyslog provides a flexible system to specify the output formats. It is template-based. A template with the traditional syslog format looks as follows: :: $template TraditionalFormat,"%timegenerated% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n" The part in quotes is the output formats. Things between percent-signs are so-called `messages properties `_. They are replaced with the respective content from the syslog message when output is written. Everything outside of the percent signs is literal text, which is simply written as specified. Thankfully, rsyslog provides message properties for the priority. These are called "PRI", "syslogfacility" and "syslogpriority" (case is important!). They are numerical values. Starting with rsyslog 1.13.4, there is also a property "pri-text", which contains the priority in friendly text format (e.g. "local0.err<133>"). For the rest of this article, I assume that you run version 1.13.4 or higher. Recording the priority is now a simple matter of adding the respective field to the template. It now looks like this: :: $template TraditionalFormatWithPRI,"%pri-text%: %timegenerated% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n" Now we have the right template - but how to write it to a file? You probably have a line like this in your syslog.conf: :: *.* -/var/log/messages.log It does not specify a template. Consequently, rsyslog uses the traditional format. In order to use some other format, simply specify the template after the semicolon: :: *.* -/var/log/messages.log;TraditionalFormatWithPRI That's all you need to do. There is one common pitfall: you need to define the template before you use it in a selector line. Otherwise, you will receive an error. Once you have applied the changes, you need to restart rsyslogd. It will then pick the new configuration. What if I do not want rsyslogd to be the standard syslogd? ---------------------------------------------------------- If you do not want to switch to rsyslog, you can still use it as a setup aid. A little bit of configuration is required. #. Download, make and install rsyslog #. copy your syslog.conf over to rsyslog.conf #. add the template described above to it; select the file that should use it #. stop your regular syslog daemon for the time being #. run rsyslogd (you may even do this interactively by calling it with the -n additional option from a shell) #. stop rsyslogd (press ctrl-c when running interactively) #. restart your regular syslogd That's it - you can now review the priorities. Some Sample Data ---------------- Below is some sample data created with the template specified above. Note the priority recording at the start of each line. :: kern.info<6>: Jun 15 18:10:38 host kernel: PCI: Sharing IRQ 11 with 00:04.0 kern.info<6>: Jun 15 18:10:38 host kernel: PCI: Sharing IRQ 11 with 01:00.0 kern.warn<4>: Jun 15 18:10:38 host kernel: Yenta IRQ list 06b8, PCI irq11 kern.warn<4>: Jun 15 18:10:38 host kernel: Socket status: 30000006 kern.warn<4>: Jun 15 18:10:38 host kernel: Yenta IRQ list 06b8, PCI irq11 kern.warn<4>: Jun 15 18:10:38 host kernel: Socket status: 30000010 kern.info<6>: Jun 15 18:10:38 host kernel: cs: IO port probe 0x0c00-0x0cff: clean. kern.info<6>: Jun 15 18:10:38 host kernel: cs: IO port probe 0x0100-0x04ff: excluding 0x100-0x107 0x378-0x37f 0x4d0-0x4d7 kern.info<6>: Jun 15 18:10:38 host kernel: cs: IO port probe 0x0a00-0x0aff: clean. local7.notice<189>: Jun 15 18:17:24 host dd: 1+0 records out local7.notice<189>: Jun 15 18:17:24 host random: Saving random seed: succeeded local7.notice<189>: Jun 15 18:17:25 host portmap: portmap shutdown succeeded local7.notice<189>: Jun 15 18:17:25 host network: Shutting down interface eth1: succeeded local7.notice<189>: Jun 15 18:17:25 host network: Shutting down loopback interface: succeeded local7.notice<189>: Jun 15 18:17:25 host pcmcia: Shutting down PCMCIA services: cardmgr user.notice<13>: Jun 15 18:17:25 host /etc/hotplug/net.agent: NET unregister event not supported local7.notice<189>: Jun 15 18:17:27 host pcmcia: modules. local7.notice<189>: Jun 15 18:17:29 host rc: Stopping pcmcia: succeeded local7.notice<189>: Jun 15 18:17:30 host rc: Starting killall: succeeded syslog.info<46>: Jun 15 18:17:33 host [origin software="rsyslogd" swVersion="1.13.3" x-pid="2464"] exiting on signal 15. syslog.info<46>: Jun 18 10:55:47 host [origin software="rsyslogd" swVersion="1.13.3" x-pid="2367"][x-configInfo udpReception="Yes" udpPort="514" tcpReception="Yes" tcpPort="1470"] restart user.notice<13>: Jun 18 10:55:50 host rger: test syslog.info<46>: Jun 18 10:55:52 host [origin software="rsyslogd" swVersion="1.13.3" x-pid="2367"] exiting on signal 2.`` Feedback Requested ------------------ I would appreciate feedback on this paper. If you have additional ideas, comments or find bugs, please `let me know `_. References and Additional Material ---------------------------------- - `www.rsyslog.com `_ - the rsyslog site Revision History ---------------- - 2007-06-18 \* `Rainer Gerhards `_ \* initial version created Copyright --------- Copyright (c) 2007 `Rainer Gerhards `_ and `Adiscon `_. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license can be viewed at `http://www.gnu.org/copyleft/fdl.html `_. build/_sources/tutorials/tls_cert_client.txt0000664000175100017510000000541012411216133020421 0ustar rgerrgerSetting up a client =================== In this step, we configure a client machine. We from our scenario, we use zuse.example.net. You need to do the same steps for all other clients, too (in the example, that meanst turng.example.net). The client check's the server's identity and talks to it only if it is the expected server. This is a very important step. Without it, you would not detect man-in-the-middle attacks or simple malicious servers who try to get hold of your valuable log data. .. figure:: tls_cert_100.jpg :align: center :alt: Steps to do: - make sure you have a functional CA (`Setting up the CA `_) - generate a machine certificate for zuse.example.net (follow instructions in `Generating Machine Certificates `_) - make sure you copy over ca.pem, machine-key.pem ad machine-cert.pem to the client. Ensure that no user except root can access them (**even read permissions are really bad**). - configure the client so that it checks the server identity and sends messages only if the server identity is known. Please note that you have the same options as when configuring a server. However, we now use a single name only, because there is only one central server. No using wildcards make sure that we will exclusively talk to that server (otherwise, a compromised client may take over its role). If you load-balance to different server identies, you obviously need to allow all of them. It still is suggested to use explcit names. **At this point, please be reminded once again that your security needs may be quite different from what we assume in this tutorial. Evaluate your options based on your security needs.** Sample syslog.conf ~~~~~~~~~~~~~~~~~~ Keep in mind that this rsyslog.conf sends messages via TCP, only. Also, we do not show any rules to write local files. Feel free to add them. :: # make gtls driver the default $DefaultNetstreamDriver gtls # certificate files $DefaultNetstreamDriverCAFile /rsyslog/protected/ca.pem $DefaultNetstreamDriverCertFile /rsyslog/protected/machine-cert.pem $DefaultNetstreamDriverKeyFile /rsyslog/protected/machine-key.pem $ActionSendStreamDriverAuthMode x509/name $ActionSendStreamDriverPermittedPeer central.example.net $ActionSendStreamDriverMode 1 # run driver in TLS-only mode *.* @@central.example.net:10514 # forward everything to remote server Note: the example above forwards every message to the remote server. Of course, you can use the normal filters to restrict the set of information that is sent. Depending on your message volume and needs, this may be a smart thing to do. **Be sure to safeguard at least the private key (machine-key.pem)!** If some third party obtains it, you security is broken! build/_sources/tutorials/tls_cert_errmsgs.txt0000664000175100017510000000750012651410660020636 0ustar rgerrgerError Messages ============== This page covers error message you may see when setting up `rsyslog `_ with TLS. Please note that many of the message stem back to the TLS library being used. In those cases, there is not always a good explanation available in rsyslog alone. A single error typically results in two or more message being emitted: (at least) one is the actual error cause, followed by usually one message with additional information (like certificate contents). In a typical system, these message should immediately follow each other in your log. Kepp in mind that they are reported as syslog.err, so you need to capture these to actually see errors (the default rsyslog.conf's shipped by many systems will do that, recording them e.g. in /etc/messages). certificate invalid ------------------- Sample: :: not permitted to talk to peer, certificate invalid: insecure algorithm This message may occur during connection setup. It indicates that the remote peer's certificate can not be accepted. The reason for this is given in the message part that is shown in red. Please note that this red part directly stems back to the TLS library, so rsyslog does acutally not have any more information about the reason. With GnuTLS, the following reasons have been seen in practice: insecure algorith ^^^^^^^^^^^^^^^^^ The certificate contains information on which encryption algorithms are to be used. This information is entered when the certificate is created. Some older alogrithms are no longer secure and the TLS library does not accept them. Thus the connection request failed. The cure is to use a certificate with sufficiently secure alogorithms. Please note that noi encryption algorithm is totally secure. It only is secure based on our current knowledge AND on computing power available. As computers get more and more powerful, previously secure algorithms become insecure over time. As such, algorithms considered secure today may not be accepted by the TLS library in the future. So in theory, after a system upgrade, a connection request may fail with the "insecure algorithm" failure without any change in rsyslog configuration or certificates. This could be caused by a new perception of the TLS library of what is secure and what not. GnuTLS error -64 ---------------- Sample: :: unexpected GnuTLS error -64 in nsd_gtls.c:517: Error while reading file. This error points to an encoding error witht the pem file in question. It means "base 64 encoding error". From my experience, it can be caused by a couple of things, some of them not obvious: - You specified a wrong file, which is not actually in .pem format - The file was incorrectly generated - I think I have also seen this when I accidently swapped private key files and certificate files. So double-check the type of file you are using. - It may even be a result of an access (permission) problem. In theory, that should lead to another error, but in practice it sometimes seems to lead to this -64 error. info on invalid cert -------------------- Sample: :: info on invalid cert: peer provided 1 certificate(s). Certificate 1 info: certificate valid from Wed Jun 18 11:45:44 2008 to Sat Jun 16 11:45:53 2018; Certificate public key: RSA; DN: C=US,O=Sample Corp,OU=Certs,L=Somehwere,ST=CA,CN=somename; Issuer DN: C=US,O=Sample Corp,OU=Certs,L=Somewhere,ST=CA,CN=somename,EMAIL=xxx@example.com; SAN:DNSname: machine.example.net; This is **not** an error message in itself. It always follows the actual error message and tells you what is seen in the peer's certificate. This is done to give you a chance to evaluate the certificate and better understand why the initial error message was issued. Please note that you can NOT diagnose problems based on this message alone. It follows in a number of error cases and does not pinpoint any problems by itself. build/_sources/tutorials/index.txt0000664000175100017510000000033612651410660016364 0ustar rgerrgerTutorials ========= .. toctree:: :maxdepth: 2 tls_cert_summary tls database high_database_rate reliable_forwarding recording_pri failover_syslog_server log_rotation_fix_size gelf_forwarding build/_sources/tutorials/tls_cert_ca.txt0000664000175100017510000001373112411216133017533 0ustar rgerrgerSetting up the CA ================= The first step is to set up a certificate authority (CA). It must be maintained by a trustworthy person (or group) and approves the indentities of all machines. It does so by issuing their certificates. In a small setup, the administrator can provide the CA function. What is important is the the CA's private key is well-protocted and machine certificates are only issued if it is know they are valid (in a single-admin case that means the admin should not issue certificates to anyone else except himself). The CA creates a so-called self-signed certificate. That is, it approves its own authenticy. This sounds useless, but the key point to understand is that every machine will be provided a copy of the CA's certificate. Accepting this certificate is a matter of trust. So by configuring the CA certificate, the administrator tells `rsyslog `_ which certificates to trust. This is the root of all trust under this model. That is why the CA's private key is so important - everyone getting hold of it is trusted by our rsyslog instances. .. figure:: tls_cert_ca.jpg :align: center :alt: To create a self-signed certificate, use the following commands with GnuTLS (which is currently the only supported TLS library, what may change in the future). Please note that GnuTLS' tools are not installed by default on many platforms. Also, the tools do not necessarily come with the GnuTLS core package. If you do not have certtool on your system, check if there is package for the GnuTLS tools available (under Fedora, for example, this is named gnutls-utils- and it is NOT installed by default). #. generate the private key: :: certtool --generate-privkey --outfile ca-key.pem This takes a short while. Be sure to do some work on your workstation, it waits for radom input. Switching between windows is sufficient ;) #. now create the (self-signed) CA certificate itself: :: certtool --generate-self-signed --load-privkey ca-key.pem --outfile ca.pem This generates the CA certificate. This command queries you for a number of things. Use appropriate responses. When it comes to certificate validity, keep in mind that you need to recreate all certificates when this one expires. So it may be a good idea to use a long period, eg. 3650 days (roughly 10 years). You need to specify that the certificates belongs to an authority. The certificate is used to sign other certificates. Sample Screen Session ~~~~~~~~~~~~~~~~~~~~~ Text in red is user input. Please note that for some questions, there is no user input given. This means the default was accepted by simply pressing the enter key. :: [root@rgf9dev sample]# certtool --generate-privkey --outfile ca-key.pem --bits 2048 Generating a 2048 bit RSA private key... [root@rgf9dev sample]# certtool --generate-self-signed --load-privkey ca-key.pem --outfile ca.pem Generating a self signed certificate... Please enter the details of the certificate's distinguished name. Just press enter to ignore a field. Country name (2 chars): US Organization name: SomeOrg Organizational unit name: SomeOU Locality name: Somewhere State or province name: CA Common name: someName (not necessarily DNS!) UID: This field should not be used in new certificates. E-mail: Enter the certificate's serial number (decimal): Activation/Expiration time. The certificate will expire in (days): 3650 Extensions. Does the certificate belong to an authority? (Y/N): y Path length constraint (decimal, -1 for no constraint): Is this a TLS web client certificate? (Y/N): Is this also a TLS web server certificate? (Y/N): Enter the e-mail of the subject of the certificate: someone@example.net Will the certificate be used to sign other certificates? (Y/N): y Will the certificate be used to sign CRLs? (Y/N): Will the certificate be used to sign code? (Y/N): Will the certificate be used to sign OCSP requests? (Y/N): Will the certificate be used for time stamping? (Y/N): Enter the URI of the CRL distribution point: X.509 Certificate Information: Version: 3 Serial Number (hex): 485a365e Validity: Not Before: Thu Jun 19 10:35:12 UTC 2008 Not After: Sun Jun 17 10:35:25 UTC 2018 Subject: C=US,O=SomeOrg,OU=SomeOU,L=Somewhere,ST=CA,CN=someName (not necessarily DNS!) Subject Public Key Algorithm: RSA Modulus (bits 2048): d9:9c:82:46:24:7f:34:8f:60:cf:05:77:71:82:61:66 05:13:28:06:7a:70:41:bf:32:85:12:5c:25:a7:1a:5a 28:11:02:1a:78:c1:da:34:ee:b4:7e:12:9b:81:24:70 ff:e4:89:88:ca:05:30:0a:3f:d7:58:0b:38:24:a9:b7 2e:a2:b6:8a:1d:60:53:2f:ec:e9:38:36:3b:9b:77:93 5d:64:76:31:07:30:a5:31:0c:e2:ec:e3:8d:5d:13:01 11:3d:0b:5e:3c:4a:32:d8:f3:b3:56:22:32:cb:de:7d 64:9a:2b:91:d9:f0:0b:82:c1:29:d4:15:2c:41:0b:97 Exponent: 01:00:01 Extensions: Basic Constraints (critical): Certificate Authority (CA): TRUE Subject Alternative Name (not critical): RFC822name: someone@example.net Key Usage (critical): Certificate signing. Subject Key Identifier (not critical): fbfe968d10a73ae5b70d7b434886c8f872997b89 Other Information: Public Key Id: fbfe968d10a73ae5b70d7b434886c8f872997b89 Is the above information ok? (Y/N): y Signing certificate... [root@rgf9dev sample]# chmod 400 ca-key.pem [root@rgf9dev sample]# ls -l total 8 -r-------- 1 root root 887 2008-06-19 12:33 ca-key.pem -rw-r--r-- 1 root root 1029 2008-06-19 12:36 ca.pem [root@rgf9dev sample]# **Be sure to safeguard ca-key.pem!** Nobody except the CA itself needs to have it. If some third party obtains it, you security is broken! build/_sources/tutorials/failover_syslog_server.txt0000664000175100017510000000303412411216133022041 0ustar rgerrgerFailover Syslog Server ====================== There are often situations where syslog data from the local system should be sent to a central syslogd (for consolidation, archival and whatever other reasons). A common problem is that messages are lost when the central syslogd goes down. Rsyslog has the capability to work with failover servers to prevent message loss. A perquisite is that TCP based syslog forwarding is used to sent to the central server. The reason is that with UDP there is no reliable way to detect the remote system has gone away. Let's assume you have a primary and two secondary central servers. Then, you can use the following config file excerpt to send data to them: rsyslog.conf: :: *.* @@primary-syslog.example.com $ActionExecOnlyWhenPreviousIsSuspended on & @@secondary-1-syslog.example.com & @@secondary-2-syslog.example.com & /var/log/localbuffer $ActionExecOnlyWhenPreviousIsSuspended off This selector processes all messages it receives (*.*). It tries to forward every message to primary-syslog.example.com (via tcp). If it can not reach that server, it tries secondary-1-syslog.example.com, if that fails too, it tries secondary-2-syslog.example.com. If neither of these servers can be connected, the data is stored in /var/log/localbuffer. Please note that the secondaries and the local log buffer are only used if the one before them does not work. So ideally, /var/log/localbuffer will never receive a message. If one of the servers resumes operation, it automatically takes over processing again. build/_sources/tutorials/reliable_forwarding.txt0000664000175100017510000002033512651410660021257 0ustar rgerrgerReliable Forwarding of syslog Messages with Rsyslog =================================================== *Written by* `Rainer Gerhards `_ *(2008-06-27)* Abstract -------- **In this paper, I describe how to forward** `syslog `_ **messages (quite) reliable to a central rsyslog server.** This depends on rsyslog being installed on the client system and it is recommended to have it installed on the server system. Please note that industry-standard `plain TCP syslog protocol is not fully reliable `_ (thus the "quite reliable"). If you need a truely reliable solution, you need to look into RELP (natively supported by rsyslog).* The Intention ------------- Whenever two systems talk over a network, something can go wrong. For example, the communications link may go down, or a client or server may abort. Even in regular cases, the server may be offline for a short period of time because of routine maintenance. A logging system should be capable of avoiding message loss in situations where the server is not reachable. To do so, unsent data needs to be buffered at the client while the server is offline. Then, once the server is up again, this data is to be sent. This can easily be acomplished by rsyslog. In rsyslog, every action runs on its own queue and each queue can be set to buffer data if the action is not ready. Of course, you must be able to detect that "the action is not ready", which means the remote server is offline. This can be detected with plain TCP syslog and RELP, but not with UDP. So you need to use either of the two. In this howto, we use plain TCP syslog. Please note that we are using rsyslog-specific features. The are required on the client, but not on the server. So the client system must run rsyslog (at least version 3.12.0), while on the server another syslogd may be running, as long as it supports plain tcp syslog. **The rsyslog queueing subsystem tries to buffer to memory. So even if the remote server goes offline, no disk file is generated.** File on disk are created only if there is need to, for example if rsyslog runs out of (configured) memory queue space or needs to shutdown (and thus persist yet unsent messages). Using main memory and going to the disk when needed is a huge performance benefit. You do not need to care about it, because, all of it is handled automatically and transparently by rsyslog. How To Setup ------------ First, you need to create a working directory for rsyslog. This is where it stores its queue files (should need arise). You may use any location on your local system. Next, you need to do is instruct rsyslog to use a disk queue and then configure your action. There is nothing else to do. With the following simple config file, you forward anything you receive to a remote server and have buffering applied automatically when it goes down. This must be done on the client machine. .. code-block:: linux-config $ModLoad imuxsock # local message reception $WorkDirectory /rsyslog/work # default location for work (spool) files $ActionQueueType LinkedList # use asynchronous processing $ActionQueueFileName srvrfwd # set file name, also enables disk mode $ActionResumeRetryCount -1 # infinite retries on insert failure $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down *.* @@server:port The port given above is optional. It may not be specified, in which case you only provide the server name. The "$ActionQueueFileName" is used to create queue files, should need arise. This value must be unique inside rsyslog.conf. No two rules must use the same queue file. Also, for obvious reasons, it must only contain those characters that can be used inside a valid file name. Rsyslog possibly adds some characters in front and/or at the end of that name when it creates files. So that name should not be at the file size name length limit (which should not be a problem these days). Please note that actual spool files are only created if the remote server is down **and** there is no more space in the in-memory queue. By default, a short failure of the remote server will never result in the creation of a disk file as a couple of hundered messages can be held in memory by default. [These parameters can be fine-tuned. However, then you need to either fully understand how the queue works (`read elaborate doc `_) or use `professional services `_ to have it done based on your specs ;) - what that means is that fine-tuning queue parameters is far from being trivial...] If you would like to test if your buffering scenario works, you need to stop, wait a while and restart you central server. Do **not** watch for files being created, as this usually does not happen and never happens immediately. Forwarding to More than One Server ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If you have more than one server you would like to forward to, that's quickly done. Rsyslog has no limit on the number or type of actions, so you can define as many targets as you like. What is important to know, however, is that the full set of directives make up an action. So you can not simply add (just) a second forwarding rule, but need to duplicate the rule configuration as well. Be careful that you use different queue file names for the second action, else you will mess up your system. A sample for forwarding to two hosts looks like this: .. code-block:: linux-config $ModLoad imuxsock # local message reception $WorkDirectory /rsyslog/work # default location for work (spool) files # start forwarding rule 1 $ActionQueueType LinkedList # use asynchronous processing $ActionQueueFileName srvrfwd1 # set file name, also enables disk mode $ActionResumeRetryCount -1 # infinite retries on insert failure $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down *.* @@server1:port # end forwarding rule 1 # start forwarding rule 2 $ActionQueueType LinkedList # use asynchronous processing $ActionQueueFileName srvrfwd2 # set file name, also enables disk mode $ActionResumeRetryCount -1 # infinite retries on insert failure $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down *.* @@server2 # end forwarding rule 2 Note the filename used for the first rule it is "srvrfwd1" and for the second it is "srvrfwd2". I have used a server without port name in the second forwarding rule. This was just to illustrate how this can be done. You can also specify a port there (or drop the port from server1). When there are multiple action queues, they all work independently. Thus, if server1 goes down, server2 still receives data in real-time. The client will **not** block and wait for server1 to come back online. Similarily, server1's operation will not be affected by server2's state. Some Final Words on Reliability ... ----------------------------------- Using plain TCP syslog provides a lot of reliability over UDP syslog. However, plain TCP syslog is **not** a fully reliable transport. In order to get full reliability, you need to use the RELP protocol. Folow the next link to learn more about `the problems you may encounter with plain tcp syslog `_. Feedback requested ~~~~~~~~~~~~~~~~~~ I would appreciate feedback on this tutorial. If you have additional ideas, comments or find bugs (I \*do\* bugs - no way... ;)), please `let me know `_. Revision History ---------------- - 2008-06-27 \* `Rainer Gerhards `_ \* Initial Version created Copyright --------- Copyright (c) 2008 `Rainer Gerhards `_ and `Adiscon `_. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license can be viewed at `http://www.gnu.org/copyleft/fdl.html `_. build/_sources/tutorials/gelf_forwarding.txt0000664000175100017510000000433712651410660020421 0ustar rgerrgerGELF forwarding in rsyslog ========================== *Written by Florian Riedl* Situation --------- The current setup has a system with rsyslog as the central syslog server and a system with Graylog for storage and analyzing the log messages. Graylog expects the log messages to arrive in GELF (Graylog Extended Log Format). Changing the default log format to GELF --------------------------------------- To make rsyslog send GELF we basically need to create a custom template. This template will define the format in which the log messages will get sent to Graylog. :: template(name="gelf" type="list") { constant(value="{\"version\":\"1.1\",") constant(value="\"host\":\"") property(name="hostname") constant(value="\",\"short_message\":\"") property(name="msg" format="json") constant(value="\",\"timestamp\":\"") property(name="timegenerated" dateformat="unixtimestamp") constant(value="\",\"level\":\"") property(name="syslogseverity") constant(value="\"}") } This is a typical representation in the list format with all the necessary fields and format definitions that Graylog expects. Applying the template to a syslog action ---------------------------------------- The next step is applying the template to our output action. Since we are forwarding log messages to Graylog, this is usually a syslog sending action. :: # syslog forwarder via UDP action(type="omfwd" target="graylogserver" port="514" protocol="udp" template="gelf") We now have a syslog forwarding action. This uses the omfwd module. Please note that the case above only works for UDP transport. When using TCP, Graylog expects a Nullbyte as message delimiter. This is currently not possible with rsyslog. Conclusion ---------- With this quick and easy setup you can feed Graylog with the correct log message format so it can do its work. This case can be applied to a lot of different scenarios as well, but with different templates. This documentation is part of the `rsyslog `_ project. Copyright © 2008 by `Rainer Gerhards `_ and `Adiscon `_. Released under the GNU GPL version 2 or higher. build/_sources/tutorials/tls_cert_scenario.txt0000664000175100017510000000241612411216133020751 0ustar rgerrgerSample Use Case: Single Central Log Server ========================================== We have a quite simple use case. There is one central syslog server, named central.example.net. These server is being reported to by two Linux machines with name zuse.example.net and turing.example.net. Also, there is a third client - ada.example.net - which send both its own messages to the central server but also forwards messages receive from an UDP-only capable router. We hav decided to use ada.example.net because it is in the same local network segment as the router and so we enjoy TLS' security benefits for forwarding the router messages inside the corporate network. All systems (except the router) use `rsyslog `_ as the syslog software. .. figure:: tls_cert_100.jpg :align: center :alt: Please note that the CA must not necessarily be connected to the rest of the network. Actually, it may be considered a security plus if it is not. If the CA is reachable via the regular network, it should be sufficiently secured (firewal rules et al). Keep in mind that if the CA's security is breached, your overall system security is breached. In case the CA is compromised, you need to regenerate the CA's certificate as well as all individual machines certificates. build/_sources/tutorials/tls_cert_udp_relay.txt0000664000175100017510000000660512411216133021136 0ustar rgerrgerSetting up the UDP syslog relay =============================== In this step, we configure the UDP relay ada.example.net. As a reminder, that machine relays messages from a local router, which only supports UDP syslog, to the central syslog server. The router does not talk directly to it, because we would like to have TLS protection for its sensitve logs. If the router and the syslog relay are on a sufficiently secure private network, this setup can be considered reasonable secure. In any case, it is the best alternative among the possible configuration scenarios. .. figure:: tls_cert_100.jpg :align: center :alt: Steps to do: - make sure you have a functional CA (`Setting up the CA `_) - generate a machine certificate for ada.example.net (follow instructions in `Generating Machine Certificates `_) - make sure you copy over ca.pem, machine-key.pem ad machine-cert.pem to the client. Ensure that no user except root can access them (**even read permissions are really bad**). - configure the client so that it checks the server identity and sends messages only if the server identity is known. These were essentially the same steps as for any `TLS syslog client `_. We now need to add the capability to forward the router logs: - make sure that the firewall rules permit message recpetion on UDP port 514 (if you use a non-standard port for UDP syslog, make sure that port number is permitted). - you may want to limit who can send syslog messages via UDP. A great place to do this is inside the firewall, but you can also do it in rsyslog.conf via an $AllowedSender directive. We have used one in the sample config below. Please be aware that this is a kind of weak authentication, but definitely better than nothing... - add the UDP input plugin to rsyslog's config and start a UDP listener - make sure that your forwarding-filter permits to forward messages received from the remote router to the server. In our sample scenario, we do not need to add anything special, because all messages are forwarded. This includes messages received from remote hosts. **At this point, please be reminded once again that your security needs may be quite different from what we assume in this tutorial. Evaluate your options based on your security needs.** Sample syslog.conf ~~~~~~~~~~~~~~~~~~ Keep in mind that this rsyslog.conf sends messages via TCP, only. Also, we do not show any rules to write local files. Feel free to add them. :: # start a UDP listener for the remote router $ModLoad imudp # load UDP server plugin $AllowedSender UDP, 192.0.2.1 # permit only the router $UDPServerRun 514 # listen on default syslog UDP port 514 # make gtls driver the default $DefaultNetstreamDriver gtls # certificate files $DefaultNetstreamDriverCAFile /rsyslog/protected/ca.pem $DefaultNetstreamDriverCertFile /rsyslog/protected/machine-cert.pem $DefaultNetstreamDriverKeyFile /rsyslog/protected/machine-key.pem $ActionSendStreamDriverAuthMode x509/name $ActionSendStreamDriverPermittedPeer central.example.net $ActionSendStreamDriverMode 1 # run driver in TLS-only mode *.* @@central.example.net:10514 # forward everything to remote server **Be sure to safeguard at least the private key (machine-key.pem)!** If some third party obtains it, you security is broken! build/_sources/tutorials/tls_cert_machine.txt0000664000175100017510000001560312651410660020563 0ustar rgerrgerGenerating the machine certificate ================================== In this step, we generate certificates for each of the machines. Please note that both clients and servers need certificates. The certificate identifies each machine to the remote peer. The DNSName specified inside the certificate can be specified inside the $PermittedPeer config statements. For now, we assume that a single person (or group) is responsible for the whole rsyslog system and thus it is OK if that single person is in posession of all machine's private keys. This simplification permits us to use a somewhat less complicated way of generating the machine certificates. So, we generate both the private and public key on the CA (which is NOT a server!) and then copy them over to the respective machines. If the roles of machine and CA administrators are split, the private key must be generated by the machine administrator. This is done via a certificate request. This request is then sent to the CA admin, which in turn generates the certificate (containing the public key). The CA admin then sends back the certificate to the machine admin, who installs it. That way, the CA admin never get's hold of the machine's private key. Instructions for this mode will be given in a later revision of this document. **In any case, it is vital that the machine's private key is protected. Anybody able to obtain that private key can imporsonate as the machine to which it belongs, thus breaching your security.** Sample Screen Session ~~~~~~~~~~~~~~~~~~~~~ Text in red is user input. Please note that for some questions, there is no user input given. This means the default was accepted by simply pressing the enter key. **Please note:** you need to substitute the names specified below with values that match your environment. Most importantly, machine.example.net must be replaced by the actual name of the machine that will be using this certificate. For example, if you generate a certificate for a machine named "server.example.com", you need to use that name. If you generate a certificate for "client.example.com", you need to use this name. Make sure that each machine certificate has a unique name. If not, you can not apply proper access control. :: [root@rgf9dev sample]# certtool --generate-privkey --outfile key.pem --sec-param 2048 Generating a 2048 bit RSA private key... [root@rgf9dev sample]# certtool --generate-request --load-privkey key.pem --outfile request.pem Generating a PKCS #10 certificate request... Country name (2 chars): US Organization name: SomeOrg Organizational unit name: SomeOU Locality name: Somewhere State or province name: CA Common name: machine.example.net UID: Enter a dnsName of the subject of the certificate: Enter the IP address of the subject of the certificate: Enter the e-mail of the subject of the certificate: Enter a challange password: Does the certificate belong to an authority? (y/N): n Will the certificate be used for signing (DHE and RSA-EXPORT ciphersuites)? (y/N): Will the certificate be used for encryption (RSA ciphersuites)? (y/N): Is this a TLS web client certificate? (y/N): y Is this also a TLS web server certificate? (y/N): y [root@rgf9dev sample]# certtool --generate-certificate --load-request request.pem --outfile cert.pem --load-ca-certificate ca.pem --load-ca-privkey ca-key.pem Generating a signed certificate... Enter the certificate's serial number (decimal): Activation/Expiration time. The certificate will expire in (days): 1000 Extensions. Do you want to honour the extensions from the request? (y/N): Does the certificate belong to an authority? (Y/N): n Will the certificate be used for IPsec IKE operations? (y/N): Is this a TLS web client certificate? (Y/N): y Is this also a TLS web server certificate? (Y/N): y Enter the dnsName of the subject of the certificate: machine.example.net {This is the name of the machine that will use the certificate} Enter the IP address of the subject of certificate: Will the certificate be used for signing (DHE and RSA-EXPORT ciphersuites)? (Y/N): Will the certificate be used for encryption (RSA ciphersuites)? (Y/N): X.509 Certificate Information: Version: 3 Serial Number (hex): 485a3819 Validity: Not Before: Thu Jun 19 10:42:54 UTC 2008 Not After: Wed Mar 16 10:42:57 UTC 2011 Subject: C=US,O=SomeOrg,OU=SomeOU,L=Somewhere,ST=CA,CN=machine.example.net Subject Public Key Algorithm: RSA Modulus (bits 2048): b2:4e:5b:a9:48:1e:ff:2e:73:a1:33:ee:d8:a2:af:ae 2f:23:76:91:b8:39:94:00:23:f2:6f:25:ad:c9:6a:ab 2d:e6:f3:62:d8:3e:6e:8a:d6:1e:3f:72:e5:d8:b9:e0 d0:79:c2:94:21:65:0b:10:53:66:b0:36:a6:a7:cd:46 1e:2c:6a:9b:79:c6:ee:c6:e2:ed:b0:a9:59:e2:49:da c7:e3:f0:1c:e0:53:98:87:0d:d5:28:db:a4:82:36:ed 3a:1e:d1:5c:07:13:95:5d:b3:28:05:17:2a:2b:b6:8e 8e:78:d2:cf:ac:87:13:15:fc:17:43:6b:15:c3:7d:b9 Exponent: 01:00:01 Extensions: Basic Constraints (critical): Certificate Authority (CA): FALSE Key Purpose (not critical): TLS WWW Client. TLS WWW Server. Subject Alternative Name (not critical): DNSname: machine.example.net Subject Key Identifier (not critical): 0ce1c3dbd19d31fa035b07afe2e0ef22d90b28ac Authority Key Identifier (not critical): fbfe968d10a73ae5b70d7b434886c8f872997b89 Other Information: Public Key Id: 0ce1c3dbd19d31fa035b07afe2e0ef22d90b28ac Is the above information ok? (Y/N): y Signing certificate... [root@rgf9dev sample]# rm -f request.pem [root@rgf9dev sample]# ls -l total 16 -r-------- 1 root root 887 2008-06-19 12:33 ca-key.pem -rw-r--r-- 1 root root 1029 2008-06-19 12:36 ca.pem -rw-r--r-- 1 root root 1074 2008-06-19 12:43 cert.pem -rw-r--r-- 1 root root 887 2008-06-19 12:40 key.pem [root@rgf9dev sample]# # it may be a good idea to rename the files to indicate where they belong to [root@rgf9dev sample]# mv cert.pem machine-cert.pem [root@rgf9dev sample]# mv key.pem machine-key.pem [root@rgf9dev sample]# Distributing Files ~~~~~~~~~~~~~~~~~~ Provide the machine with: - a copy of ca.pem - cert.pem - key.pem This is how the relevant part of rsyslog.conf looks on the target machine: ```` :: $DefaultNetstreamDriverCAFile /home/rger/proj/rsyslog/sample/ca.pem $DefaultNetstreamDriverCertFile /home/rger/proj/rsyslog/sample/machine-cert.pem $DefaultNetstreamDriverKeyFile /home/rger/proj/rsyslog/sample/machine-key.pem **Never provide anyone with ca-key.pem!** Also, make sure nobody but the machine in question gets hold of key.pem. build/_sources/tutorials/tls_cert_summary.txt0000664000175100017510000001507612534263003020655 0ustar rgerrgerEncrypting Syslog Traffic with TLS (SSL) ======================================== Written by `Rainer Gerhards `_ (2008-07-03) .. toctree:: :maxdepth: 1 tls_cert_scenario tls_cert_ca tls_cert_machine tls_cert_server tls_cert_client tls_cert_udp_relay tls_cert_errmsgs Overview -------- This document describes a secure way to set up rsyslog TLS. A secure logging environment requires more than just encrypting the transmission channel. This document provides one possible way to create such a secure system. Rsyslog's TLS authentication can be used very flexible and thus supports a wide range of security policies. This section tries to give some advise on a scenario that works well for many environments. However, it may not be suitable for you - please assess you security needs before using the recommendations below. Do not blame us if it doesn't provide what you need ;) Our policy offers these security benefits: - syslog messages are encrypted while traveling on the wire - the syslog sender authenticates to the syslog receiver; thus, the receiver knows who is talking to it - the syslog receiver authenticates to the syslog sender; thus, the sender can check if it indeed is sending to the expected receiver - the mutual authentication prevents man-in-the-middle attacks Our secrity goals are achived via public/private key security. As such, it is vital that private keys are well protected and not accessible to third parties. If private keys have become known to third parties, the system does not provide any security at all. Also, our solution bases on X.509 certificates and a (very limited) chain of trust. We have one instance (the CA) that issues all machine certificates. The machine certificate indentifies a particular machine. hile in theory (and practice), there could be several "sub-CA" that issues machine certificates for a specific adminitrative domain, we do not include this in our "simple yet secure" setup. If you intend to use this, rsyslog supports it, but then you need to dig a bit more into the documentation (or use the forum to ask). In general, if you depart from our simple model, you should have good reasons for doing so and know quite well what you are doing - otherwise you may compromise your system security. Please note that security never comes without effort. In the scenario described here, we have limited the effort as much as possible. What remains is some setup work for the central CA, the certificate setup for each machine as well as a few configuration commands that need to be applied to all of them. Proably the most important limiting factor in our setup is that all senders and receivers must support IETF's syslog-transport-tls standard (which is not finalized yet). We use mandatory-to-implement technology, yet you may have trouble finding all required features in some implementations. More often, unfortunately, you will find that an implementation does not support the upcoming IETF standard at all - especially in the "early days" (starting May 2008) when rsyslog is the only implementation of said standard. Fortunately, rsyslog supports almost every protocol that is out there in the syslog world. So in cases where transport-tls is not available on a sender, we recommend to use rsyslog as the initial relay. In that mode, the not-capabe sender sends to rsyslog via another protocol, which then relays the message via transport-tls to either another interim relay or the final destination (which, of course, must by transport-tls capable). In such a scenario, it is best to try see what the sender support. Maybe it is possible to use industry-standard plain tcp syslog with it. Often you can even combine it with stunnel, which then, too, enables a secure delivery to the first rsyslog relay. If all of that is not possible, you can (and often must...) resort to UDP. Even though this is now lossy and insecure, this is better than not having the ability to listen to that device at all. It may even be reasonale secure if the uncapable sender and the first rsyslog relay communicate via a private channel, e.g. a dedicated network link. One final word of caution: transport-tls protects the connection between the sender and the receiver. It does not necessarily protect against attacks that are present in the message itself. Especially in a relay environment, the message may have been originated from a malicious system, which placed invalid hostnames and/or other content into it. If there is no provisioning against such things, these records may show up in the receivers' repository. -transport-tls does not protect against this (but it may help, properly used). Keep in mind that syslog-transport-tls provides hop-by-hop security. It does not provide end-to-end security and it does not authenticate the message itself (just the last sender). If you'd like to get all information very rapidly, the graphic below contains everything you need to know (from the certificate perspective) in a very condensed manner. It is no surprise if the graphic puzzles you. In this case, `simply read on `_ for full instructions. .. figure:: tls_cert.jpg :align: center :alt: TLS/SSL protected syslog Summary ~~~~~~~ If you followed the steps outlined in this documentation set, you now have a reasonable (for most needs) secure setup for the following environment: .. figure:: tls_cert_100.jpg :align: center :alt: You have learned about the security decisions involved and which we made in this example. **Be once again reminded that you must make sure yourself that whatever you do matches your security needs!** There is no guarantee that what we generally find useful actually is. It may even be totally unsuitable for your environment. In the example, we created a rsyslog certificate authority (CA). Guard the CA's files. You need them whenever you need to create a new machine certificate. We also saw how to generate the machine certificates themselfs and distribute them to the individual machines. Also, you have found some configuration samples for a server, a client and a syslog relay. Hopefully, this will enable you to set up a similar system in many environments. Please be warned that you defined some expiration dates for the certificates. After they are reached, the certificates are no longer valid and rsyslog will NOT accept them. At that point, syslog messages will no longer be transmitted (and rsyslogd will heavily begin to complain). So it is a good idea to make sure that you renew the certificates before they expire. Recording a reminder somewhere is probably a good idea. build/_sources/tutorials/tls_cert_server.txt0000664000175100017510000001145712411216133020461 0ustar rgerrgerSetting up the Central Server ============================= In this step, we configure the central server. We assume it accepts messages only via TLS protected plain tcp based syslog from those peers that are explicitely permitted to send to it. The picture below show our configuration. This step configures the server central.example.net. .. figure:: tls_cert_100.jpg :align: center :alt: ***Important:** Keep in mind that the order of configuration directives is very important in rsyslog. As such, the samples given below do only work if the given order is preserved. Re-ordering the directives can break configurations and has broken them in practice. If you intend to re-order them, please be sure that you fully understand how the configuration language works and, most importantly, which statements form a block together. Please also note that we understand the the current configuration file format is ugly. However, there has been more important work in the way of enhancing it. If you would like to contribute some time to improve the config file language, please let us know. Any help is appreciated (be it doc or coding work!).* Steps to do: - make sure you have a functional CA (`Setting up the CA `_) - generate a machine certificate for central.example.net (follow instructions in `Generating Machine Certificates `_) - make sure you copy over ca.pem, machine-key.pem ad machine-cert.pem to the central server. Ensure that no user except root can access them (**even read permissions are really bad**). - configure the server so that it accepts messages from all machines in the example.net domain that have certificates from your CA. Alternatively, you may also precisely define from which machine names messages are accepted. See sample rsyslog.conf below. In this setup, we use wildcards to ease adding new systems. We permit the server to accept messages from systems whos names match \*.example.net. :: $InputTCPServerStreamDriverPermittedPeer *.example.net This will match zuse.example.net and turing.example.net, but NOT pascal.otherdepartment.example.net. If the later would be desired, you can (and need) to include additional permitted peer config statments: :: $InputTCPServerStreamDriverPermittedPeer *.example.net $InputTCPServerStreamDriverPermittedPeer *.otherdepartment.example.net $InputTCPServerStreamDriverPermittedPeer *.example.com As can be seen with example.com, the different permitted peers need NOT to be in a single domain tree. Also, individual machines can be configured. For example, if only zuse, turing and ada should be able to talk to the server, you can achive this by: :: $InputTCPServerStreamDriverPermittedPeer zuse.example.net $InputTCPServerStreamDriverPermittedPeer turing.example.net $InputTCPServerStreamDriverPermittedPeer ada.example.net As an extension to the (upcoming) IETF syslog/tls standard, you can specify some text together with a domain component wildcard. So "\*server.example.net", "server\*.example.net" are valid permitted peers. However "server\*Fix.example.net" is NOT a valid wildcard. The IETF standard permits no text along the wildcards. The reason we use wildcards in the default setup is that it makes it easy to add systems without the need to change the central server's configuration. It is important to understand that the central server will accept names **only** (no exception) if the client certificate was signed by the CA we set up. So if someone tries to create a malicious certificate with a name "zuse.example.net", the server will **not** accept it. So a wildcard is safe as long as you ensure CA security is not breached. Actually, you authorize a client by issuing the certificate to it. **At this point, please be reminded once again that your security needs may be quite different from what we assume in this tutorial. Evaluate your options based on your security needs.** Sample syslog.conf ~~~~~~~~~~~~~~~~~~ Keep in mind that this rsyslog.conf accepts messages via TCP, only. The only other source accepted is messages from the server itself. :: $ModLoad imuxsock # local messages $ModLoad imtcp # TCP listener # make gtls driver the default $DefaultNetstreamDriver gtls # certificate files $DefaultNetstreamDriverCAFile /rsyslog/protected/ca.pem $DefaultNetstreamDriverCertFile /rsyslog/protected/machine-cert.pem $DefaultNetstreamDriverKeyFile /rsyslog/protected/machine-key.pem $InputTCPServerStreamDriverAuthMode x509/name $InputTCPServerStreamDriverPermittedPeer *.example.net $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode $InputTCPServerRun 10514 # start up listener at port 10514 **Be sure to safeguard at least the private key (machine-key.pem)!** If some third party obtains it, you security is broken! build/_sources/examples/0000775000175100017510000000000012651410761014304 5ustar rgerrgerbuild/_sources/examples/high_performance.txt0000664000175100017510000000550712651410660020352 0ustar rgerrgerReceiving massive amounts of messages with high performance =========================================================== Use Case -------- You are receiving syslog messages via UDP and or TCP at a very high data rate. You want to tune the system so that it can process as many messages as possible. All messages shall be written to a single output file. Sample Configuration -------------------- :: # load required modules module(load="imudp" threads="2" timeRequery="8" batchSize="128") module(load="imptcp" threads="3") # listeners # repeat blocks if more listeners are needed # alternatively, use array syntax: # port=["514","515",...] input(type="imudp" port="514" ruleset="writeRemoteData") input(type="imptcp" port="10514" ruleset="writeRemoteData") # now define our ruleset, which also includes # threading and queue parameters. ruleset(name="writeRemoteData" queue.type="fixedArray" queue.size="250000" queue.dequeueBatchSize="4096" queue.workerThreads="4" queue.workerThreadMinimumMessages="60000" ) { action(type="omfile" file="/var/log/remote.log" ioBufferSize="64k" flushOnTXEnd="off" asyncWriting="on") } Notes on the suggested config ----------------------------- It is highly suggested to use a recent enough Linux kernel that supports the **recvmmsg()** system call. This system call improves UDP reception speed and decreases overall system CPU utilization. We use the **imptcp** module for tcp input, as it uses more optimal results. Note, however, that it is only available on Linux and does currently *not* support TLS. If **imptcp** cannot be used, use **imtcp** instead (this will be a bit slower). When writing to the output file, we use buffered mode. This means that full buffers are written, but during processing file lines are not written until the buffer is full (and thus may be delayed) and also incomplete lines are written (at buffer boundary). When the file is closed (rsyslogd stop or HUP), the buffer is completely flushed. As this is a high-traffic use case, we assume that buffered mode does not cause any concerns. Suggested User Performance Testing ---------------------------------- Each environment is a bit different. Depending on circumstances, the **imudp** module parameters may not be optimal. In order to obtain best performance, it is suggested to measure performance level with two to four threads and somewhat lower and higher batchSize. Note that these parameters affect each other. The values given in the config above should usually work well in *high-traffic* environments. They are sub-optimal for low to medium traffic environments. See Also -------- imptcp, imtcp, imudp, ruleset() Copyright --------- Copyright (c) 2014 by `Rainer Gerhards `_ build/_sources/examples/index.txt0000664000175100017510000000012712651410660016152 0ustar rgerrgerExample Use Cases ================= .. toctree:: :maxdepth: 2 high_performance build/_sources/configuration/0000775000175100017510000000000012651410760015334 5ustar rgerrgerbuild/_sources/configuration/config_param_types.txt0000664000175100017510000000336712411216133021747 0ustar rgerrgerConfiguration Parameter Types ============================= Configuration parameter values have different data types. Unfortunately, the type currently must be guessed from the description (consider contributing to the doc to help improve it). In general, the following types are used: - **numbers** The traditional integer format. Numbers may include '.' and ',' for readability. So you can for example specify either "1000" or "1,000" with the same result. Please note that rsyslogd simply *ignores* the punctuation. From it's point of view, "1,,0.0.,.,0" also has the value 1000. - **sizes** Used for things like file size, main message queue sizes and the like. These are integers, but support modifier after the number part. For example, 1k means 1024. Supported are k(ilo), m(ega), g(iga), t(era), p(eta) and e(xa). Lower case letters refer to the traditional binary defintion (e.g. 1m equals 1,048,576) whereas upper case letters refer to their new 1000-based definition (e.g 1M equals 1,000,000). - **complete line** A string consisting of multiple characters. This is relatively seldom used and sometimes looks confusing (rsyslog v7+ has a much better approach at these types of values). - **single word** This is used when only a single word can be provided. A "single word" is a string without spaces in it. **No** quoting is necessary nor permitted (the quotes would become part of the word). - **character** A single (printable) character. Must **not** be quoted. - **boolean** The traditional boolean type, specified as "on" (1) or "off" (0). Note that some other value types are occasionally used. However, the majority of types is one of those listed above. The list is updated as need arises and time permits. build/_sources/configuration/basic_structure.txt0000664000175100017510000002144112651410660021277 0ustar rgerrgerBasic Structure =============== This section describes how rsyslog configuration basically works. Think of rsyslog as a big logging and event processing toolset. It can be considered a framework with some basic processing that is fixed in the way data flows, but is highly customizable in the details of this message flow. During configuration, this customization is done by defining and customizing the rsyslog objects. Quick overview of message flow and objects ------------------------------------------ Messages enter rsyslog with the help of input modules. Then, they are passed to ruleset, where rules are conditionally applied. When a rule matches, the message is transferred to an action, which then does something to the message, e.g. writes it to a file, database or forwards it to a remote host. Processing Principles --------------------- - inputs submit received messages to rulesets * if the ruleset is not specifically bound, the default ruleset is used - by default, there is one ruleset (RSYSLOG_DefaultRuleset) - additional rulesets can be user-defined - each ruleset contains of zero or many rules * while it is permitted to have zero rules inside a ruleset, this obviously makes no sense - a rule consists of a filter and an action list - filters provide yes/no decisions and thus control-of-flow capability - if a filter "matches" (filter says "yes"), the corresponding action list is executed. If it does not match, nothing special happens - rules are evaluated in sequence from the first to the last rule **inside** the given ruleset. No rules from unrelated rulesets are ever executed. - all rules are **always** fully evaluated, no matter if a filter matches or not (so we do **not** stop at the first match). If message processing shall stop, the "discard" action (represented by the tilde character or the stop command) must explicitly be executed. If discard is executed, message processing immediately stops, without evaluating any further rules. - an action list contains one or many actions - inside an action list no further filters are possible - to have more than one action inside a list, the ampersand character must be placed in the position of the filter, and this must immediately follow the previous action - actions consist of the action call itself (e.g. ":omusrmsg:") as well as all action-defining configuration statements ($Action... directives) - if legacy format is used (see below), $Action... directives **must** be specified in front of the action they are intended to configure - some config directives automatically refer to their previous values after being applied, others not. See the respective doc for details. Be warned that this is currently not always properly documented. - in general, rsyslog v5 is heavily outdated and its native config language is a pain. The rsyslog project strongly recommends using at least version 7, where these problems are solved and configuration is much easier. - legacy configuration statements (those starting with $) do **not** affect RainerScript objects (e.g. actions). Configuration File ------------------ Upon startup, rsyslog reads its configuration from the ``rsyslog.conf`` file by default. This file may contain references to include other config files. A different "root" configuration file can be specified via the ``-f `` rsyslogd command line option. This is usually done within some init script or similar facility. Statement Types --------------- Rsyslog supports three different types of configuration statements concurrently: - **sysklogd** - this is the plain old format, taught everywhere and still pretty useful for simple use cases. Note that some very few constructs are no longer supported because they are incompatible with newer features. These are mentioned in the compatibility docs. - **legacy rsyslog** - these are statements that begin with a dollar sign. They set some configuration parameters and modify e.g. the way actions operate. This is the only format supported in pre-v6 versions of rsyslog. It is still fully supported in v6 and above. Note that some plugins and features may still only be available through legacy format (because plugins need to be explicitly upgraded to use the new style format, and this hasn't happened to all plugins). - **RainerScript** - the new style format. This is the best and most precise format to be used for more complex cases. The rest of this page assumes RainerScript based rsyslog.conf. The rsyslog.conf files consists of statements. For old style (sysklogd & legacy rsyslog), lines do matter. For new style (RainerScript) line spacing is irrelevant. Most importantly, this means with new style actions and all other objects can split across lines as users want to. Recommended use of Statement Types ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ In general it is recommended to use RainerScript type statements, as these provide clean and easy to read control-of-flow as well as no doubt about which parameters are active. They also have no side-effects with include files, which can be a major obstacle with legacy rsyslog statements. For very simple things sysklogd statement types are still suggested, especially if the full config consists of such simple things. The classical sample is writing to files (or forwarding) via priority. In sysklogd, this looks like: :: mail.info /var/log/mail.log mail.err @server.example.net This is hard to beat in simplicity, still being taught in courses and a lot of people know this syntax. It is perfectly fine to use these constructs even in newly written config files. **As a rule of thumb, RainerScript config statements should be used when** - configuration parameters are required (e.g. the ``Action...`` type of legacy statements) - more elaborate control-of-flow is required (e.g. when multiple actions must be nested under the same condition) It is usually **not** recommended to use rsyslog legacy config format (those directives starting with a dollar sign). However, a few settings and modules have not yet been converted to RainerScript. In those cases, the legacy syntax must be used. Comments -------- There are two types of comments: - **#-Comments** - start with a hash sign (#) and run to the end of the line - **C-style Comments** - start with /\* and end with \*/, just like in the C programming language. They can be used to comment out multiple lines at once. Comment nesting is not supported, but #-Comments can be contained inside a C-style comment. Processing Order ---------------- Directives are processed from the top of rsyslog.conf to the bottom. Order matters. For example, if you stop processing of a message, obviously all statements after the stop statement are never evaluated. Flow Control Statements ~~~~~~~~~~~~~~~~~~~~~~~ Flow control is provided by: - :doc:`Control Structures <../rainerscript/control_structures>` - :doc:`Filter Conditions ` Data Manipulation Statements ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Data manipulation is achieved by **set**, **unset** and **reset** statements which are :doc:`documented here in detail <../rainerscript/variable_property_types>`. Inputs ------ Every input requires an input module to be loaded and a listener defined for it. Full details can be found inside the `rsyslog modules `_ documentation. Once loaded, inputs are defined via the **input()** object. Outputs ------- Outputs are also called "actions". A small set of actions is pre-loaded (like the output file writer, which is used in almost every rsyslog.conf), others must be loaded just like inputs. An action is invoked via the **action(type="type" ...)** object. Type is mandatory and must contain the name of the plugin to be called (e.g. "omfile" or "ommongodb"). Other parameters may be present. Their type and use depends on the output plugin in question. Rulesets and Rules ------------------ Rulesets and rules form the basis of rsyslog processing. In short, a rule is a way how rsyslog shall process a specific message. Usually, there is a type of filter (if-statement) in front of the rule. Complex nesting of rules is possible, much like in a programming language. Rulesets are containers for rules. A single ruleset can contain many rules. In the programming language analogy, one may think of a ruleset like being a program. A ruleset can be "bound" (assigned) to a specific input. In the analogy, this means that when a message comes in via that input, the "program" (ruleset) bound to it will be executed (but not any other!). There is detailed documentation available for :doc:`rsyslog rulesets <../concepts/multi_ruleset>`. For quick reference, rulesets are defined as follows: :: ruleset(name="rulesetname") { action(type="omfile" file="/path/to/file") action(type="..." ...) /* and so on... */ } build/_sources/configuration/properties.txt0000664000175100017510000001544012651410660020274 0ustar rgerrgerrsyslog Properties ================== Data items in rsyslog are called "properties". They can have different origin. The most important ones are those that stem from received messages. But there are also others. Whenever you want to access data items, you need to access the resprective property. Properties are used in - :doc:`templates ` - conditional statements The property name is case-insensitive (prior to 3.17.0, they were case-senstive). Message Properties ------------------ These are extracted by rsyslog parsers from the original message. All message properties start with a letter. The following message properties exist: **msg** the MSG part of the message (aka "the message" ;)) **rawmsg** the message excactly as it was received from the socket. Should be useful for debugging. It is also useful if a message should be forwarded totally unaltered. **rawmsg-after-pri** Almost the same as **rawmsg**, but the syslog PRI is removed. If no PRI was present, **rawmsg-after-pri** is identical to **rawmsg**. Note that the syslog PRI is header field that contains information on syslog facility and severity. It is enclosed in greater-than and less-than characters, e.g. "<191>". This field is often not written to log files, but usually needs to be present for the receiver to properly classify the message. There are some rare cases where one wants the raw message, but not the PRI. You can use this property to obtain that. In general, you should know that you need this format, otherwise stay away from the property. **hostname** hostname from the message **source** alias for HOSTNAME **fromhost** hostname of the system the message was received from (in a relay chain, this is the system immediately in front of us and not necessarily the original sender). This is a DNS-resolved name, except if that is not possible or DNS resolution has been disabled. **fromhost-ip** The same as fromhost, but always as an IP address. Local inputs (like imklog) use 127.0.0.1 in this property. **syslogtag** TAG from the message **programname** the "static" part of the tag, as defined by BSD syslogd. For example, when TAG is "named[12345]", programname is "named". **pri** PRI part of the message - undecoded (single value) **pri-text** the PRI part of the message in a textual form with the numerical PRI appended in brackes (e.g. "local0.err<133>") **iut** the monitorware InfoUnitType - used when talking to a `MonitorWare `_ backend (also for `Adiscon LogAnalyzer `_) **syslogfacility** the facility from the message - in numerical form **syslogfacility-text** the facility from the message - in text form **syslogseverity** severity from the message - in numerical form **syslogseverity-text** severity from the message - in text form **syslogpriority** an alias for syslogseverity - included for historical reasons (be careful: it still is the severity, not PRI!) **syslogpriority-text** an alias for syslogseverity-text **timegenerated** timestamp when the message was RECEIVED. Always in high resolution **timereported** timestamp from the message. Resolution depends on what was provided in the message (in most cases, only seconds) **timestamp** alias for timereported **protocol-version** The contents of the PROTCOL-VERSION field from IETF draft draft-ietf-syslog-protcol **structured-data** The contents of the STRUCTURED-DATA field from IETF draft draft-ietf-syslog-protocol **app-name** The contents of the APP-NAME field from IETF draft draft-ietf-syslog-protocol **procid** The contents of the PROCID field from IETF draft draft-ietf-syslog-protocol **msgid** The contents of the MSGID field from IETF draft draft-ietf-syslog-protocol **inputname** The name of the input module that generated the message (e.g. "imuxsock", "imudp"). Note that not all modules necessarily provide this property. If not provided, it is an empty string. Also note that the input module may provide any value of its liking. Most importantly, it is **not** necessarily the module input name. Internal sources can also provide inputnames. Currently, "rsyslogd" is defined as inputname for messages internally generated by rsyslogd, for example startup and shutdown and error messages. This property is considered useful when trying to filter messages based on where they originated - e.g. locally generated messages ("rsyslogd", "imuxsock", "imklog") should go to a different place than messages generated somewhere. **jsonmesg** *Available since rsyslog 8.3.0* The whole message object as JSON representation. Note that the JSON string will *not* include and LF and it will contain *all other message properties* specified here as respective JSON containers. It also includes all message variables in the "$!" subtree (this may be null if none are present). This property is primarily meant as an interface to other systems and tools that want access to the full property set (namely external plugins). Note that it contains the same data items potentially multiple times. For example, parts of the syslog tag will by containened in the rawmsg, syslogtag, and programname properties. As such, this property has some additional overhead. Thus, it is suggested to be used only when there is actual need for it. System Properties ----------------- These properties are provided by the rsyslog core engine. They are **not** related to the message. All system properties start with a dollar-sign. For example, ``timereported`` contains the timestamp from the message. Depending on how long the message was in the relay chain, this can be quite old. In contrast, ``$now`` is the system time when the message is being processed. Depending on your needs, you need one or the other. Usually, the message-based timestamp is the more important one, but that really depdends on the use case. The following system properties exist: **$bom** The UTF-8 encoded Unicode byte-order mask (BOM). This may be useful in templates for RFC5424 support, when the character set is know to be Unicode. **$now** The current date stamp in the format YYYY-MM-DD **$year** The current year (4-digit) **$month** The current month (2-digit) **$day** The current day of the month (2-digit) **$hour** The current hour in military (24 hour) time (2-digit) **$hhour** The current half hour we are in. From minute 0 to 29, this is always 0 while from 30 to 59 it is always 1. **$qhour** The current quarter hour we are in. Much like $HHOUR, but values range from 0 to 3 (for the four quater hours that are in each hour) **$minute** The current minute (2-digit) **$myhostname** The name of the current host as it knows itself (probably useful for filtering in a generic way) build/_sources/configuration/modules/0000775000175100017510000000000012651410760017004 5ustar rgerrgerbuild/_sources/configuration/modules/sigprov_gt.txt0000664000175100017510000000603112651410660021727 0ustar rgerrgerGuardTime Log Signature Provider (gt) ===================================== **Signature Provider Name: gt** **Author:** Rainer Gerhards **Supported Since:** since 7.3.9 **Description**: Provides the ability to sign syslog messages via the GuardTime signature services. **Configuration Parameters**: Signature providers are loaded by omfile, when the provider is selected in its "sig.providerName" parameter. Parameters for the provider are given in the omfile action instance line. This provider creates a signature file with the same base name but the extension ".gtsig" for each log file (both for fixed-name files as well as dynafiles). Both files together form a set. So you need to archive both in order to prove integrity. - **sig.hashFunction** The following hash algorithms are currently supported: - SHA1 - RIPEMD-160 - SHA2-224 - SHA2-256 - SHA2-384 - SHA2-512 - **sig.timestampService** This provides the URL of the timestamper service. If not selected, a default server is selected. This may not necessarily be a good one for your region. *Note:* If you need to supply user credentials, you can add them to the timestamper URL. If, for example, you have a user "user" with password "pass", you can do so as follows: http://user:pass@timestamper.example.net - **sig.block.sizeLimit** The maximum number of records inside a single signature block. By default, there is no size limit, so the signature is only written on file closure. Note that a signature request typically takes between one and two seconds. So signing to frequently is probably not a good idea. - **sig.keepRecordHashes** Controls if record hashes are written to the .gtsig file. This enhances the ability to spot the location of a signature breach, but costs considerable disk space (65 bytes for each log record for SHA2-512 hashes, for example). - **sig.keepTreeHashes** Controls if tree (intermediate) hashes are written to the .gtsig file. This enhances the ability to spot the location of a signature breach, but costs considerable disk space (a bit mire than the amount sig.keepRecordHashes requries). Note that both Tree and Record hashes can be kept inside the signature file. **See Also** - `How to sign log messages through signature provider Guardtime `_ **Caveats/Known Bugs:** - currently none known **Samples:** This writes a log file with it's associated signature file. Default parameters are used. :: action(type="omfile" file="/var/log/somelog" sig.provider="gt") In the next sample, we use the more secure SHA2-512 hash function, sign every 10,000 records and Tree and Record hashes are kept. :: action(type="omfile" file="/var/log/somelog" sig.provider="gt" sig.hashfunction="SHA2-512" sig.block.sizelimit="10000" sig.keepTreeHashes="on" sig.keepRecordHashes="on") build/_sources/configuration/modules/ompipe.txt0000664000175100017510000000302312651410660021033 0ustar rgerrgerompipe: Pipe Output Module ========================== **Module Name:    ompipe** **Author:**\ Rainer Gerhards **Description**: The ompipe plug-in provides the core functionality for logging output to named pipes (fifos). It is a built-in module that does not need to be loaded. **Global Configuration Directives:** - Template: [templateName] sets a new default template for file actions. **Action specific Configuration Directives:** - Pipe: string a fifo or named pipe can be used as a destination for log messages. **Caveats/Known Bugs:** None **Sample:** The following command sends all syslog messages to a remote server via TCP port 10514. ::     Module (path="builtin:ompipe")     *.* action(type="ompipe"     Pipe="NameofPipe") **Legacy Configuration Directives:** rsyslog has support for logging output to named pipes (fifos). A fifo or named pipe can be used as a destination for log messages by prepending a pipe symbol ("|") to the name of the file. This is handy for debugging. Note that the fifo must be created with the mkfifo(1) command before rsyslogd is started. **Legacy Sample:** The following command sends all syslog messages to a remote server via TCP port 10514. ::     $ModLoad ompipe     *.* |/var/log/pipe This documentation is part of the `rsyslog `_ project. Copyright © 2008-2014 by `Rainer Gerhards `_ and `Adiscon `_. Released under the ASL 2.0. build/_sources/configuration/modules/idx_output.txt0000664000175100017510000000050612651410660021751 0ustar rgerrgerOutput Modules -------------- Output modules process messages. With them, message formats can be transformed and messages be transmitted to various different targets. They are generally defined via :doc:`action <../actions>` configuration objects. .. toctree:: :glob: :maxdepth: 1 om* sigprov_gt sigprov_ksi build/_sources/configuration/modules/mmutf8fix.txt0000664000175100017510000001055012651410660021474 0ustar rgerrgerFix invalid UTF-8 Sequences (mmutf8fix) ======================================= **Module Name:** mmutf8fix **Author:** Rainer Gerhards **Available since**: 7.5.4 **Description**: The mmutf8fix module permits to fix invalid UTF-8 sequences. Most often, such invalid sequences result from syslog sources sending in non-UTF character sets, e.g. ISO 8859. As syslog does not have a way to convey the character set information, these sequences are not properly handled. While they are typically uncritical with plain text files, they can cause big headache with database sources as well as systems like ElasticSearch. The module supports different "fixing" modes and fixes. The current implementation will always replace invalid bytes with a single US ASCII character. Additional replacement modes will probably be added in the future, depending on user demand. In the longer term it could also be evolved into an any-charset-to-UTF8 converter. But first let's see if it really gets into widespread enough use. **Proper Usage**: Some notes are due for proper use of this module. This is a message modification module utilizing the action interface, which means you call it like an action. This gives great flexibility on the question on when and how to call this module. Note that once it has been called, it actually modifies the message. The original messsage is then no longer available. However, this does **not** change any properties set, used or extracted before the modification is done. One potential use case is to normalize all messages. This is done by simply calling mmutf8fix right in front of all other actions. If only a specific source (or set of sources) is known to cause problems, mmutf8fix can be conditionally called only on messages from them. This also offers performance benefits. If such multiple sources exists, it probably is a good idea to define different listeners for their incoming traffic, bind them to specific `ruleset `_ and call mmutf8fix as first action in this ruleset. **Module Configuration Parameters**: Currently none.   **Action Confguration Parameters**: - **mode** - **utf-8**/controlcharacters This sets the basic detection mode. In **utf-8** mode (the default), proper UTF-8 encoding is checked and bytes which are not proper UTF-8 sequences are acted on. If a proper multi-byte start sequence byte is detected but any of the following bytes is invalid, the whole sequence is replaced by the replacement method. This mode is most useful with non-US-ASCII character sets, which validly includes multibyte sequences. Note that in this mode control characters are NOT being replaced, because they are valid UTF-8. In **controlcharacters** mode, all bytes which do not represent a printable US-ASCII character (codes 32 to 126) are replaced. Note that this also mangles valid UTF-8 multi-byte sequences, as these are (deliberately) outside of that character range. This mode is most useful if it is known that no characters outside of the US-ASCII alphabet need to be processed. - **replacementChar** - default " " (space), a single character This is the character that invalid sequences are replaced by. Currently, it MUST be a **printable** US-ASCII character. **Caveats/Known Bugs:** - overlong UTF-8 encodings are currently not detected in utf-8 mode. **Samples:** In this snippet, we write one file without fixing UTF-8 and another one with the message fixed. Note that once mmutf8fix has run, access to the original message is no longer possible. :: module(load="mmutf8fix") action(type="omfile" file="/path/to/non-fixed.log") action(type="mmutf8fix") action(type="omfile" file="/path/to/fixed.log") In this sample, we fix only message originating from host 10.0.0.1. :: module(load="mmutf8fix") if $fromhost-ip == "10.0.0.1" then action(type="mmutf8fix") # all other actions here... This is mostly the same as the previous sample, but uses "controlcharacters" processing mode. :: module(load="mmutf8fix") if $fromhost-ip == "10.0.0.1" then action(type="mmutf8fix" mode="controlcharacters") # all other actions here... This documentation is part of the `rsyslog `_ project. Copyright © 2013-2014 by `Rainer Gerhards `_ and `Adiscon `_. Released under the GNU GPL version 3 or higher. build/_sources/configuration/modules/pmlastmsg.txt0000664000175100017510000000460612411216133021552 0ustar rgerrgerpmlastmsg: last message repeated n times ======================================== **Module Name:    pmlastmsg** **Module Type:    parser module** **Author:**\ Rainer Gerhards **Available Since**: 5.5.6 **Description**: Some syslogds are known to emit severily malformed messages with content "last message repeated n times". These messages can mess up message reception, as they lead to wrong interpretation with the standard RFC3164 parser. Rather than trying to fix this issue in pmrfc3164, we have created a new parser module specifically for these messages. The reason is that some processing overhead is involved in processing these messages (they must be recognized) and we would not like to place this toll on every user but only on those actually in need of the feature. Note that the performance toll is not large -- but if you expect a very high message rate with tenthousands of messages per second, you will notice a difference. This module should be loaded first inside `rsyslog's parser chain `_. It processes all those messages that contain a PRI, then none or some spaces and then the exact text (case-insensitive) "last message repeated n times" where n must be an integer. All other messages are left untouched. **Configuration Directives**: There do not currently exist any configuration directives for this module. **Examples:** This example is the typical use case, where some systems emit malformed "repeated msg" messages. Other than that, the default RFC5424 and RFC3164 parsers should be used. Note that when a parser is specified, the default parser chain is removed, so we need to specify all three parsers. We use this together with the default ruleset. $ModLoad pmlastmsg # this parser is NOT a built-in module # note that parser are tried in the # order they appear in rsyslog.conf, so put pmlastmsg first $RulesetParser rsyslog.lastline # as we have removed the default parser chain, we # need to add the default parsers as well. $RulesetParser rsyslog.rfc5424 $RulesetParser rsyslog.rfc3164 # now come the typical rules, like... \*.\* /path/to/file.log **Caveats/Known Bugs:** currently none This documentation is part of the `rsyslog `_ project. Copyright © 2010-2014 by `Rainer Gerhards `_ and `Adiscon `_. Released under the GNU GPL version 3 or higher. build/_sources/configuration/modules/imgssapi.txt0000664000175100017510000000307612651410660021366 0ustar rgerrgerimgssapi: GSSAPI Syslog Input Module ==================================== Provides the ability to receive syslog messages from the network protected via Kerberos 5 encryption and authentication. This module also accept plain tcp syslog messages on the same port if configured to do so. If you need just plain tcp, use :doc:`imtcp ` instead. Note: This is a contributed module, which is not supported by the rsyslog team. We recommend to use RFC5425 TLS-protected syslog instead. **Author:**\ varmojfekoj .. toctree:: :maxdepth: 1 gssapi Configuration Directives ------------------------ .. function:: $InputGSSServerRun Starts a GSSAPI server on selected port - note that this runs independently from the TCP server. .. function:: $InputGSSServerServiceName The service name to use for the GSS server. .. function:: $InputGSSServerPermitPlainTCP on\|off Permits the server to receive plain tcp syslog (without GSS) on the same port .. function:: $InputGSSServerMaxSessions Sets the maximum number of sessions supported .. function:: $InputGSSServerKeepAlive on\|off *Requires: 8.5.0 or above* *Default: off* Enables or disable keep-alive handling. Caveats/Known Bugs ------------------ - module always binds to all interfaces - only a single listener can be bound Example ------- This sets up a GSS server on port 1514 that also permits to receive plain tcp syslog messages (on the same port): :: $ModLoad imgssapi # needs to be done just once $InputGSSServerRun 1514 $InputGSSServerPermitPlainTCP on build/_sources/configuration/modules/mmexternal.txt0000664000175100017510000000343312651410660021723 0ustar rgerrgerSupport module for external message modification modules ======================================================== **Module Name:    mmexternal** **Available since:   ** 8.3.0 **Author:**\ Rainer Gerhards **Description**: This module permits to integrate external message modification plugins into rsyslog. For details on the interface specification, see rsyslog's source in the ./plugins/external/INTERFACE.md.   **Action Parameters**: - **binary** The name of the external message modification plugin to be called. This can be a full path name. - **interface.input** This can either be "msg", "rawmsg" or "fulljson". In case of "fulljson", the message object is provided as a json object. Otherwise, the respective property is provided. This setting **must** match the external plugin's expectations. Check the external plugin documentation for what needs to be used. - **output** This is a debug aid. If set, this is a filename where the plugins output is logged. Note that the output is also being processed as usual by rsyslog. Setting this parameter thus gives insight into the internal processing that happens between plugin and rsyslog core. - **forcesingleinstance** This is an expert parameter, just like the equivalent *omprog* parameter. See the message modification plugin's documentation if it is needed. **Caveats/Known Bugs:** - None. **Sample:** The following config file snippet is used to write execute an external message modification module "mmexternal.py". Note that the path to the module is specified here. This is necessary if the module is not in the default search path. :: module (load="mmexternal") # needs to be done only once inside the config action(type="mmexternal" binary="/path/to/mmexternal.py") build/_sources/configuration/modules/mmpstrucdata.txt0000664000175100017510000000677012651410660022262 0ustar rgerrgerRFC5424 structured data parsing module (mmpstrucdata) ===================================================== **Module Name:** mmpstrucdata **Author:** Rainer Gerhards **Available since**: 7.5.4 **Description**: The mmpstrucdata parses the structured data of `RFC5424 `_ into the message json variable tree. The data parsed, if available, is stored under "jsonRoot!rfc5424-sd!...". Please note that only RFC5424 messages will be processed. The difference of RFC5424 is in the message layout: the SYSLOG-MSG part only contains the structured-data part instead of the normal message part. Further down you can find a example of a structured-data part. **Module Configuration Parameters**: Currently none.   **Action Confguration Parameters**: - **jsonRoot** - default "!" Specifies into which json container the data shall be parsed to. **See Also** - `Howto anonymize messages that go to specific files `_ **Caveats/Known Bugs:** - this module is currently experimental; feedback is appreciated - property names are treated case-insensitive in rsyslog. As such, RFC5424 names are treated case-insensitive as well. If such names only differ in case (what is not recommended anyways), problems will occur. - structured data with duplicate SD-IDs and SD-PARAMS is not properly processed **Samples:** Below you can find a structured data part of a random message which has three parameters. :: [exampleSDID@32473 iut="3" eventSource="Application"eventID="1011"] In this snippet, we parse the message and emit all json variable to a file with the message anonymized. Note that once mmpstrucdata has run, access to the original message is no longer possible (execept if stored in user variables before anonymization). :: module(load="mmpstrucdata") action(type="mmpstrucdata") template(name="jsondump" type="string" string="%msg%: %$!%\\n") action(type="omfile" file="/path/to/log" template="jsondump") **A more practical one:** Take this example message (inspired by RFC5424 sample;)): ``<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"][id@2 test="tast"] BOM'su root' failed for lonvick on /dev/pts/8`` We apply this configuration: :: module(load="mmpstrucdata") action(type="mmpstrucdata") template(name="sample2" type="string" string="ALL: %$!%\\nSD: %$!RFC5424-SD%\\nIUT:%$!rfc5424-sd!exampleSDID@32473!iut%\\nRAWMSG: %rawmsg%\\n\\n") action(type="omfile" file="/path/to/log" template="sample2") This will output: ``ALL: { "rfc5424-sd": { "examplesdid@32473": { "iut": "3", "eventsource": "Application", "eventid": "1011" }, "id@2": { "test": "tast" } } } SD: { "examplesdid@32473": { "iut": "3", "eventsource": "Application", "eventid": "1011" }, "id@2": { "test": "tast" } } IUT:3 RAWMSG: <34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 [exampleSDID@32473 iut="3" eventSource="Application" eventID="1011"][id@2 test="tast"] BOM'su root' failed for lonvick on /dev/pts/8`` As you can seem, you can address each of the individual items. Note that the case of the RFC5424 parameter names has been converted to lower case. This documentation is part of the `rsyslog `_ project. Copyright © 2013 by `Rainer Gerhards `_ and `Adiscon `_. Released under the GNU GPL version 3 or higher. build/_sources/configuration/modules/omruleset.txt0000664000175100017510000001530712651410660021571 0ustar rgerrgeromruleset: ruleset output/including module ========================================== **Module Name:    omruleset** **Author:**\ Rainer Gerhards **Available Since**: 5.3.4 **Deprecated in**: 7.2.0+ **Deprecation note** This module exists only for backwards-compatibility reasons. **Do no longer use it in new configurations.** It has been replaced by the much more efficient `"call" RainerScript statement `_. The "call" statement supports everything omruleset does, but in an easier to use way. **Description**: This is a very special "output" module. It permits to pass a message object to another rule set. While this is a very simple action, it enables very complex configurations, e.g. it supports high-speed "and" conditions, sending data to the same file in a non-racy way, include-ruleset functionality as well as some high-performance optimizations (in case the rule sets have the necessary queue definitions). While it leads to a lot of power, this output module offers seamingly easy functionaltiy. The complexity (and capablities) arise from how everthing can be combined. With this module, a message can be sent to processing to another ruleset. This is somewhat similar to a "#include" in the C programming language. However, one needs to keep on the mind that a ruleset can contain its own queue and that a queue can run in various modes. Note that if no queue is defined in the ruleset, the message is enqueued into the main message queue. This most often is not optimal and means that message processing may be severely defered. Also note that when the ruleset's target queue is full and no free space can be aquired within the usual timeout, the message object may actually be lost. This is an extreme scenario, but users building an audit-grade system need to know this restriction. For regular installations, it should not really be relevant. **At minimum, be sure you understand the** :doc:`$RulesetCreateMainQueue <../ruleset/rsconf1_rulesetcreatemainqueue>` **directive as well as the importance of statement order in rsyslog.conf before using omruleset!** **Recommended Use:** - create rulesets specifically for omruleset - create these rulesets with their own main queue - decent queueing parameters (sizes, threads, etc) should be used for the ruleset main queue. If in doubt, use the same parameters as for the overall main queue. - if you use multiple levels of ruleset nesting, double check for endless loops - the rsyslog engine does not detect these **Configuration Directives**: - **$ActionOmrulesetRulesetName** ruleset-to-submit-to This directive specifies the name of the ruleset that the message provided to omruleset should be submitted to. This ruleset must already have been defined. Note that the directive is automatically reset after each :omruleset: action and there is no default. This is done to prevent accidential loops in ruleset definition, what can happen very quickly. The :omruleset: action will NOT be honored if no ruleset name has been defined. As usual, the ruleset name must be specified in front of the action that it modifies. **Examples:** This example creates a ruleset for a write-to-file action. The idea here is that the same file is written based on multiple filters, problems occur if the file is used together with a buffer. That is because file buffers are action-specific, and so some partial buffers would be written. With omruleset, we create a single action inside its own ruleset and then pass all messages to it whenever we need to do so. Of course, such a simple situation could also be solved by a more complex filter, but the method used here can also be utilized in more complex scenarios (e.g. with multiple listeners). The example tries to keep it simple. Note that we create a ruleset-specific main queue (for simplicity with the default main queue parameters) in order to avoid re-queueing messages back into the main queue. :: $ModLoad omruleset # define ruleset for commonly written file $RuleSet CommonAction $RulesetCreateMainQueue on *.* /path/to/file.log #switch back to default ruleset $ruleset RSYSLOG_DefaultRuleset # begin first action # note that we must first specify which ruleset to use for omruleset: $ActionOmrulesetRulesetName CommonAction mail.info :omruleset: # end first action # begin second action # note that we must first specify which ruleset to use for omruleset: $ActionOmrulesetRulesetName CommonAction :FROMHOST, isequal, "myhost.example.com" :omruleset: #end second action # of course, we can have "regular" actions alongside :omrulset: actions *.* /path/to/general-message-file.log The next example is used to creat a high-performance nested and filter condition. Here, it is first checked if the message contains a string "error". If so, the message is forwarded to another ruleset which then applies some filters. The advantage of this is that we can use high-performance filters where we otherwise would need to use the (much slower) expression-based filters. Also, this enables pipeline processing, in that second ruleset is executed in parallel to the first one. :: $ModLoad omruleset # define "second" ruleset $RuleSet nested $RulesetCreateMainQueue on # again, we use our own queue mail.* /path/to/mailerr.log kernel.* /path/to/kernelerr.log auth.* /path/to/autherr.log #switch back to default ruleset $ruleset RSYSLOG_DefaultRuleset # begin first action - here we filter on "error" # note that we must first specify which ruleset to use for omruleset: $ActionOmrulesetRulesetName nested :msg, contains, "error" :omruleset: #end first action # begin second action - as an example we can do anything else in # this processing. Note that these actions are processed concurrently # to the ruleset "nested" :FROMHOST, isequal, "myhost.example.com" /path/to/host.log #end second action # of course, we can have "regular" actions alongside :omrulset: actions *.* /path/to/general-message-file.log **Caveats/Known Bugs:** The current configuration file language is not really adequate for a complex construct like omruleset. Unfortunately, more important work is currently preventing me from redoing the config language. So use extreme care when nesting rulesets and be sure to test-run your config before putting it into production, ensuring you have a suffciently large probe of the traffic run over it. If problems arise, the `rsyslog debug log `_ is your friend. This documentation is part of the `rsyslog `_ project. Copyright © 2009-2014 by `Rainer Gerhards `_ and `Adiscon `_. Released under the GNU GPL version 3 or higher. build/_sources/configuration/modules/omhdfs.txt0000664000175100017510000000513612411216133021022 0ustar rgerrgeromhdfs: Hadoop Filesystem Output Module ======================================= **Module Name:    omhdfs** **Available since:   ** 5.7.1 **Author:**\ Rainer Gerhards **Description**: This module supports writing message into files on Hadoop's HDFS file system. **Configuration Directives**: - **$OMHDFSFileName** [name] The name of the file to which the output data shall be written. - **$OMHDFSHost** [name] Name or IP address of the HDFS host to connect to. - **$OMHDFSPort** [name] Port on which to connect to the HDFS host. - **$OMHDFSDefaultTemplate** [name] Default template to be used when none is specified. This saves the work of specifying the same template ever and ever again. Of course, the default template can be overwritten via the usual method. **Caveats/Known Bugs:** Building omhdfs is a challenge because we could not yet find out how to integrate Java properly into the autotools build process. The issue is that HDFS is written in Java and libhdfs uses JNI to talk to it. That requires that various system-specific environment options and pathes be set correctly. At this point, we leave this to the user. If someone know how to do it better, please drop us a line! - In order to build, you need to set these environment variables BEFORE running ./configure: - JAVA\_INCLUDES - must have all include pathes that are needed to build JNI C programms, including the -I options necessary for gcc. An example is # export JAVA\_INCLUDES="-I/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86\_64/include -I/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86\_64/include/linux" - JAVA\_LIBS - must have all library pathes that are needed to build JNI C programms, including the -l/-L options necessary for gcc. An example is # export export JAVA\_LIBS="-L/usr/java/jdk1.6.0\_21/jre/lib/amd64 -L/usr/java/jdk1.6.0\_21/jre/lib/amd64/server -ljava -ljvm -lverify" - As of HDFS architecture, you must make sure that all relevant environment variables (the usual Java stuff and HADOOP's home directory) are properly set. - As it looks, libhdfs makes Java throw exceptions to stdout. There is no known work-around for this (and it usually should not case any troubles. **Sample:** :: $ModLoad omhdfs $OMHDFSFileName /var/log/logfile \*.\* :omhdfs: This documentation is part of the `rsyslog `_ project. Copyright © 2010-2014 by `Rainer Gerhards `_ and `Adiscon `_. Released under the GNU GPL version 3 or higher. build/_sources/configuration/modules/omlibdbi.txt0000664000175100017510000001270612651410660021333 0ustar rgerrgeromlibdbi: Generic Database Output Module ======================================== **Module Name:** omlibdbi **Author:** Rainer Gerhards **Description**: This modules supports a large number of database systems via `libdbi `_. Libdbi abstracts the database layer and provides drivers for many systems. Drivers are available via the `libdbi-drivers `_ project. As of this writing, the following drivers are available: - `Firebird/Interbase `_ - `FreeTDS `_ (provides access to `MS SQL Server `_ and `Sybase `_) - `MySQL `_ (also supported via the native ommysql plugin in rsyslog) - `PostgreSQL `_\ (also supported via the native ommysql plugin in rsyslog) - `SQLite/SQLite3 `_ The following drivers are in various stages of completion: - `Ingres `_ - `mSQL `_ - `Oracle `_ These drivers seem to be quite usable, at least from an rsyslog point of view. Libdbi provides a slim layer between rsyslog and the actual database engine. We have not yet done any performance testing (e.g. omlibdbi vs. ommysql) but honestly believe that the performance impact should be irrelevant, if at all measurable. Part of that assumption is that rsyslog just does the "insert" and most of the time is spent either in the database engine or rsyslog itself. It's hard to think of any considerable time spent in the libdbi abstraction layer. **Setup** In order for this plugin to work, you need to have libdbi, the libdbi driver for your database backend and the client software for your database backend installed. There are libdbi packages for many distributions. Please note that rsyslogd requires a quite recent version (0.8.3) of libdbi. It may work with older versions, but these need some special ./configure options to support being called from a dlopen()ed plugin (as omlibdbi is). So in short, you probably save you a lot of headache if you make sure you have at least libdbi version 0.8.3 on your system. **Configuration Directives**: - **$ActionLibdbiDriverDirectory** /path/to/dbd/drivers This is a global setting. It points libdbi to its driver directory. Usually, you do not need to set it. If you installed libdbi-driver's at a non-standard location, you may need to specify the directory here. If you are unsure, do not use this configuration directive. Usually, everything works just fine.\ - **$ActionLibdbiDriver** drivername Name of the dbidriver to use, see libdbi-drivers documentation. As a quick excerpt, at least those were available at the time of this writiting "mysql" (suggest to use ommysql instead), "firebird" (Firbird and InterBase), "ingres", "msql", "Oracle", "sqlite", "sqlite3", "freetds" (for Microsoft SQL and Sybase) and "pgsql" (suggest to use ompgsql instead). - **$ActionLibdbiHost** hostname The host to connect to. - **$ActionLibdbiUserName** user The user used to connect to the database. - **$ActionlibdbiPassword** That user's password. - **$ActionlibdbiDBName** db The database that shall be written to. - **selector line:** :omlibdbi:;template executes the recently configured omlibdbi action. The ;template part is optional. If no template is provided, a default template is used (which is currently optimized for MySQL - sorry, folks...) **Caveats/Known Bugs:** You must make sure that any templates used for omlibdbi properly escape strings. This is usually done by supplying the SQL (or STDSQL) option to the template. Omlibdbi rejects templates without this option for security reasons. However, omlibdbi does not detect if you used the right option for your backend. Future versions of rsyslog (with full expression  support) will provide advanced ways of handling this situation. So far, you must be careful. The default template provided by rsyslog is suitable for MySQL, but not necessarily for your database backend. Be careful! If you receive the rsyslog error message "libdbi or libdbi drivers not present on this system" you may either not have libdbi and its drivers installed or (very probably) the version is earlier than 0.8.3. In this case, you need to make sure you have at least 0.8.3 and the libdbi driver for your database backend present on your system. I do not have most of the database supported by omlibdbi in my lab. So it received limited cross-platform tests. If you run into troubles, be sure the let us know at `http://www.rsyslog.com `_. **Sample:** The following sample writes all syslog messages to the database "syslog\_db" on mysqlserver.example.com. The server is MySQL and being accessed under the account of "user" with password "pwd" (if you have empty passwords, just remove the $ActionLibdbiPassword line). :: $ModLoad omlibdbi $ActionLibdbiDriver mysql $ActionLibdbiHost mysqlserver.example.com $ActionLibdbiUserName user $ActionLibdbiPassword pwd $ActionLibdbiDBName syslog\_db \*.\* :omlibdbi: This documentation is part of the `rsyslog `_ project. Copyright © 2008-2014 by `Rainer Gerhards `_ and `Adiscon `_. Released under the GNU GPL version 3 or higher. build/_sources/configuration/modules/gssapi.txt0000664000175100017510000000402112651410660021027 0ustar rgerrgerGSSAPI module support in rsyslog v3 =================================== What is it good for. - client-serverauthentication - Log messages encryption Requirements. - Kerberos infrastructure - rsyslog, rsyslog-gssapi Configuration. Let's assume there are 3 machines in kerberos Realm: - the first is running KDC (Kerberos Authentication Service and Key Distribution Center), - the second is a client sending its logs to the server, - the third is receiver, gathering all logs. 1. KDC: - Kerberos database must be properly set-up on KDC machine first. Use kadmin/kadmin.local to do that. Two principals need to be add in our case: #. sender@REALM.ORG - client must have ticket for pricipal sender - REALM.ORG is kerberos Realm #. host/receiver.mydomain.com@REALM.ORG - service principal - Use ktadd to export service principal and transfer it to /etc/krb5.keytab on receiver 2. CLIENT: - set-up rsyslog, in /etc/rsyslog.conf - $ModLoad omgssapi - load output gss module - $GSSForwardServiceName otherThanHost - set the name of service principal, "host" is the default one - \*.\* :omgssapi:receiver.mydomain.com - action line, forward logs to receiver - kinit root - get the TGT ticket - service rsyslog start 3. SERVER: - set-up rsyslog, in /etc/rsyslog.conf - $ModLoad `imgssapi `_ - load input gss module - $InputGSSServerServiceName otherThanHost - set the name of service principal, "host" is the default one - $InputGSSServerPermitPlainTCP on - accept GSS and TCP connections (not authenticated senders), off by default - $InputGSSServerRun 514 - run server on port - service rsyslog start The picture demonstrate how things work. .. figure:: gssapi.png :align: center :alt: rsyslog gssapi support rsyslog gssapi support This documentation is part of the `rsyslog `_ project. Copyright © 2008 by `Rainer Gerhards `_ and `Adiscon `_. Released under the GNU GPL version 3 or higher. build/_sources/configuration/modules/imudp.txt0000664000175100017510000003010412651410660020660 0ustar rgerrgerimudp: UDP Syslog Input Module ============================== .. index:: ! imudp =========================== =========================================================================== **Module Name:**  **imudp** **Author:** `Rainer Gerhards `_ =========================== =========================================================================== Provides the ability to receive syslog messages via UDP. Multiple receivers may be configured by specifying multiple input statements. Note that in order to enable UDP reception, Firewall rules probably need to be modified as well. Also, SELinux may need additional rules. Configuration Parameters ------------------------ .. index:: imudp; module parameters Module Parameters ^^^^^^^^^^^^^^^^^ .. function:: TimeRequery *Default: 2* This is a performance optimization. Getting the system time is very costly. With this setting, imudp can be instructed to obtain the precise time only once every n-times. This logic is only activated if messages come in at a very fast rate, so doing less frequent time calls should usually be acceptable. The default value is two, because we have seen that even without optimization the kernel often returns twice the identical time. You can set this value as high as you like, but do so at your own risk. The higher the value, the less precise the timestamp. **Note:** the timeRequery is done based on executed system calls (**not** messages received). So when batch sizes are used, multiple messages are received with one system call. All of these messages always receive the same timestamp, as they are effectively received at the same time. When there is very high traffic and successive system calls immediately return the next batch of messages, the time requery logic kicks in, which means that by default time is only queried for every second batch. Again, this should not cause a too-much deviation as it requires messages to come in very rapidly. However, we advise not to set the "timeRequery" parameter to a large value (larger than 10) if input batches are used. .. function:: SchedulingPolicy *Default: unset* Can be used the set the scheduler priority, if the necessary functionality is provided by the platform. Most useful to select "fifo" for real-time processing under Linux (and thus reduce chance of packet loss). .. function:: SchedulingPriority *Default: unset* Scheduling priority to use. .. function:: batchSize *Default: 32* This parameter is only meaningful if the system support recvmmsg() (newer Linux OSs do this). The parameter is silently ignored if the system does not support it. If supported, it sets the maximum number of UDP messages that can be obtained with a single OS call. For systems with high UDP traffic, a relatively high batch size can reduce system overhead and improve performance. However, this parameter should not be overdone. For each buffer, max message size bytes are statically required. Also, a too-high number leads to reduced efficiency, as some structures need to be completely initialized before the OS call is done. We would suggest to not set it above a value of 128, except if experimental results show that this is useful. .. function:: threads *Available since: 7.5.5* *Default: 1* Number of worker threads to process incoming messages. These threads are utilized to pull data off the network. On a busy system, additional threads (but not more than there are CPUs/Cores) can help improving performance and avoiding message loss. Note that with too many threads, performance can suffer. There is a hard upper limit on the number of threads that can be defined. Currently, this limit is set to 32. It may increase in the future when massive multicore processors become available. .. index:: imudp; input parameters Input Parameters ^^^^^^^^^^^^^^^^ ..index:: imudp; address (input parameter) .. function:: Address *Default: \** Local IP address (or name) the UDP server should bind to. Use \"*" to bind to all of the machine's addresses. .. index:: single: imudp; port (input parameter) .. function:: Port *Default: 514* Specifies the port the server shall listen to.. Either a single port can be specified or an array of ports. If multiple ports are specified, a listener will be automatically started for each port. Thus, no additional inputs need to be configured. Single port: Port="514" Array of ports: Port=["514","515","10514","..."] .. function:: Ruleset *Default: RSYSLOG_DefaultRuleset* Binds the listener to a specific :doc:`ruleset <../../concepts/multi_ruleset>`. .. function:: RateLimit.Interval [number] *Available since: 7.3.1* *Default: 0* The rate-limiting interval in seconds. Value 0 turns off rate limiting. Set it to a number of seconds (5 recommended) to activate rate-limiting. .. function:: RateLimit.Burst [number] *Available since: 7.3.1* *Default: 10000* Specifies the rate-limiting burst in number of messages. .. function:: name [name] *Available since: 8.3.3* *Default: imudp* specifies the value of the inputname property. In older versions, this was always "imudp" for all listeners, which still is the default. Starting with 7.3.9 it can be set to different values for each listener. Note that when a single input statement defines multipe listner ports, the inputname will be the same for all of them. If you want to differentiate in that case, use "name.appendPort" to make them unique. Note that the "name" parameter can be an empty string. In that case, the corresponding inputname property will obviously also be the empty string. This is primarily meant to be used together with "name.appendPort" to set the inputname equal to the port. .. function:: InputName [name] *Available since: 7.3.9* **Deprecated** This provides the same functionality as "name". It is the historical parameter name and should not be used in new configurations. It is scheduled to be removed some time in the future. .. function:: name.appendPort [on/off] *Available since: 7.3.9* *Default: off* Appends the port the the inputname property. Note that when no "name" is specified, the default of "imudp" is used and the port is appended to that default. So, for example, a listner port of 514 in that case will lead to an inputname of "imudp514". The ability to append a port is most useful when multiple ports are defined for a single input and each of the inputnames shall be unique. Note that there currently is no differentiation between IPv4/v6 listeners on the same port. .. function:: InputName.AppendPort [on/off] *Available since: 7.3.9* **Deprecated** This provides the same functionality as "name.appendPort". It is the historical parameter name and should not be used in new configurations. It is scheduled to be removed some time in the future. .. function:: defaultTZ *Default: unset* This is an **experimental** parameter; details may change at any time and it may also be discoutinued without any early warning. Permits to set a default timezone for this listener. This is useful when working with legacy syslog (RFC3164 et al) residing in different timezones. If set it will be used as timezone for all messages **that do not contain timezone info**. Currently, the format **must** be "+/-hh:mm", e.g. "-05:00", "+01:30". Other formats, including TZ names (like EST) are NOT yet supported. Note that consequently no daylight saving settings are evaluated when working with timezones. If an invalid format is used, "interesting" things can happen, among them malformed timestamps and rsyslogd segfaults. This will obviously be changed at the time this feature becomes non-experimental. .. function:: rcvbufSize [size] *Available since: 7.5.3* *Default: unset* This request a socket receive buffer of specific size from the operating system. It is an expert parameter, which should only be changed for a good reason. Note that setting this parameter disables Linux auto-tuning, which usually works pretty well. The default value is 0, which means "keep the OS buffer size unchanged". This is a size value. So in addition to pure integer values, sizes like "256k", "1m" and the like can be specified. Note that setting very large sizes may require root or other special privileges. Also note that the OS may slightly adjust the value or shrink it to a system-set max value if the user is not sufficiently privileged. Technically, this parameter will result in a setsockopt() call with SO\_RCVBUF (and SO\_RCVBUFFORCE if it is available). See Also -------- - `rsyslog video tutorial on how to store remote messages in a separate file `_. - Description of `rsyslog statistic counters `_. This also describes all imudp counters. Caveats/Known Bugs ------------------ - Scheduling parameters are set **after** privileges have been dropped. In most cases, this means that setting them will not be possible after privilege drop. This may be worked around by using a sufficiently-privileged user account. Samples ------- This sets up an UPD server on port 514: :: module(load="imudp") # needs to be done just once input(type="imudp" port="514") The following sample is mostly equivalent to the first one, but request a larger rcvuf size. Note that 1m most probably will not be honored by the OS until the user is sufficiently privileged. :: module(load="imudp") # needs to be done just once input(type="imudp" port="514" rcvbufSize="1m") In the next example, we set up three listeners at ports 10514, 10515 and 10516 and assign a listner name of "udp" to it, followed by the port number: :: module(load="imudp") input(type="imudp" port=["10514","10515","10516"] inputname="udp" inputname.appendPort="on") The next example is almost equal to the previous one, but now the inputname property will just be set to the port number. So if a message was received on port 10515, the input name will be "10515" in this example whereas it was "udp10515" in the previous one. Note that to do that we set the inputname to the empty string. :: module(load="imudp") input(type="imudp" port=["10514","10515","10516"] inputname="" inputname.appendPort="on") Legacy Configuration Directives ------------------------------- Legacy configuration parameters should **not** be used when crafting new configuration files. It is easy to get things wrong with them. ====================================== ==================== ======================= Directive Equivalent Parameter Requires ====================================== ==================== ======================= $UDPServerTimeRequery *TimeRequery* $IMUDPSchedulingPolicy *SchedulingPolicy* 4.7.4+, 5.7.3+, 6.1.3+ $IMUDPSchedulingPriority *SchedulingPriority* 4.7.4+, 5.7.3+, 6.1.3+ $UDPServerAddress Address $UDPServerRun Port $InputUDPServerBindRuleset Ruleset 5.3.2+ ====================================== ==================== ======================= Note: module parameters are given in *italics*. All others are input parameters. Multiple receivers may be configured by specifying $UDPServerRun multiple times. Legacy Sample ^^^^^^^^^^^^^ This sets up an UDP server on port 514: :: $ModLoad imudp # needs to be done just once $UDPServerRun 514 This documentation is part of the `rsyslog `_ project. Copyright © 2009-2014 by `Rainer Gerhards `_ and `Adiscon `_. Released under the GNU GPL version 3 or higher. build/_sources/configuration/modules/ommysql.txt0000664000175100017510000000650612651410660021254 0ustar rgerrgerommysql: MySQL Database Output Module ===================================== **Module Name:** ommysql **Author:** Michael Meckelein (Initial Author) / Rainer Gerhards **Description**: This module provides native support for logging to MySQL databases. It offers superior performance over the more generic `omlibdbi `_ module. **Configuration Directives**: ommysql mostly uses the "old style" configuration, with almost everything on the action line itself. A few newer features are being migrated to the new style-config directive configuration system. - **$ActionOmmysqlServerPort ** Permits to select a non-standard port for the MySQL server. The default is 0, which means the system default port is used. There is no need to specify this directive unless you know the server is running on a non-standard listen port. - **$OmMySQLConfigFile ** Permits the selection of an optional MySQL Client Library configuration file (my.cnf) for extended configuration functionality. The use of this configuration directive is necessary only if you have a non-standard environment or if fine-grained control over the database connection is desired. - **$OmMySQLConfigSection ** Permits the selection of the section within the configuration file specified by the **$OmMySQLConfigFile** directive. This will likely only be used where the database administrator provides a single configuration file with multiple profiles. This configuration directive is ignored unless **$OmMySQLConfigFile** is also used in the rsyslog configration file. If omitted, the MySQL Client Library default of "client" will be used. - Action parameters: **:ommysql:database-server,database-name,database-userid,database-password** All parameters should be filled in for a successful connect. Note rsyslog contains a canned default template to write to the MySQL database. It works on the MonitorWare schema. This template is: :: $template tpl,"insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag%')",SQL As you can see, the template is an actual SQL statement. Note the ",SQL" option: it tells the template processor that the template is used for SQL processing, thus quote characters are quoted to prevent security issues. You can not assign a template without ",SQL" to a MySQL output action. If you would like to change fields contents or add or delete your own fields, you can simply do so by modifying the schema (if required) and creating your own custom template. **Sample:** The following sample writes all syslog messages to the database "syslog\_db" on mysqlserver.example.com. The server is being accessed under the account of "user" with password "pwd". :: $ModLoad ommysql $ActionOmmysqlServerPort 1234 # use non-standard port \*.\*      :ommysql:mysqlserver.example.com,syslog\_db,user,pwd This documentation is part of the `rsyslog `_ project. Copyright © 2008-2014 by `Rainer Gerhards `_ and `Adiscon `_. Released under the GNU GPL version 3 or higher. build/_sources/configuration/modules/omrelp.txt0000664000175100017510000001563512651410660021054 0ustar rgerrgeromrelp: RELP Output Module ========================== **Module Name:    omrelp** **Author:**\ Rainer Gerhards **Description**: This module supports sending syslog messages over the reliable RELP protocol. For RELP's advantages over plain tcp syslog, please see the documentation for :doc:`imrelp ` (the server counterpart).  Setup Please note that `librelp `__ is required for imrelp (it provides the core relp protocol implementation). **Action Configuration Parameters**: This module supports RainerScript configuration starting with rsyslog 7.3.10. For older versions, legacy configuration directives must be used. - **target** (mandatory) The target server to connect to. - **template** (not mandatory, default "RSYSLOG\_ForwardFormat") Defines the template to be used for the output. - **timeout** (not mandatory, default 90) Timeout for relp sessions. If set too low, valid sessions may be considered dead and tried to recover. - **windowSize** (not mandatory, default 0) This is an **expert parameter**. It permits to override the RELP window size being used by the client. Changing the window size has both an effect on performance as well as potential message duplication in failure case. A larger window size means more performance, but also potentially more duplicated messages - and vice versa. The default 0 means that librelp's default window size is being used, which is considered a compromise between goals reached. For your information: at the time of this writing, the librelp default window size is 128 messages, but this may change at any time. Note that there is no equivalent server parameter, as the client proposes and manages the window size in RELP protocol. - **tls** (not mandatory, values "on","off", default "off") If set to "on", the RELP connection will be encrypted by TLS, so that the data is protected against observers. Please note that both the client and the server must have set TLS to either "on" or "off". Other combinations lead to unpredictable results. *Attention when using GnuTLS 2.10.x or older* Versions older than GnuTLS 2.10.x may cause a crash (Segfault) under certain circumstances. Most likely when an imrelp inputs and an omrelp output is configured. The crash may happen when you are receiving/sending messages at the same time. Upgrade to a newer version like GnuTLS 2.12.21 to solve the problem. - **tls.compression** (not mandatory, values "on","off", default "off") The controls if the TLS stream should be compressed (zipped). While this increases CPU use, the network bandwidth should be reduced. Note that typical text-based log records usually compress rather well. - **tls.permittedPeer** peer Places access restrictions on this forwarder. Only peers which have been listed in this parameter may be connected to. This guards against rouge servers and man-in-the-middle attacks. The validation bases on the certficate the remote peer presents. The *peer* parameter lists permitted certificate fingerprints. Note that it is an array parameter, so either a single or multiple fingerprints can be listed. When a non-permitted peer is connected to, the refusal is logged together with it's fingerprint. So if the administrator knows this was a valid request, he can simple add the fingerprint by copy and paste from the logfile to rsyslog.conf. It must be noted, though, that this situation should usually not happen after initial client setup and administrators should be alert in this case. Note that usually a single remote peer should be all that is ever needed. Support for multiple peers is primarily included in support of load balancing scenarios. If the connection goes to a specific server, only one specific certificate is ever expected (just like when connecting to a specific ssh server). To specify multiple fingerprints, just enclose them in braces like this: :: tls.permittedPeer=["SHA1:...1", "SHA1:....2"] To specify just a single peer, you can either specify the string directly or enclose it in braces. - **tls.authMode** mode Sets the mode used for mutual authentication. Supported values are either "*fingerprint*\ " or "*name"*. Fingerprint mode basically is what SSH does. It does not require a full PKI to be present, instead self-signed certs can be used on all peers. Even if a CA certificate is given, the validity of the peer cert is NOT verified against it. Only the certificate fingerprint counts. In "name" mode, certificate validation happens. Here, the matching is done against the certificate's subjectAltName and, as a fallback, the subject common name. If the certificate contains multiple names, a match on any one of these names is considered good and permits the peer to talk to rsyslog. - **tls.prioritystring** (not mandatory, string) This parameter permits to specify the so-called "priority string" to GnuTLS. This string gives complete control over all crypto parameters, including compression setting. For this reason, when the prioritystring is specified, the "tls.compression" parameter has no effect and is ignored. Full information about how to construct a priority string can be found in the GnuTLS manual. At the time of this writing, this information was contained in `section 6.10 of the GnuTLS manual `__. **Note: this is an expert parameter.** Do not use if you do not exactly know what you are doing. - **localclientip** ip_address (not mandatory, string) omrelp uses ip_address as local client address while connecting to remote logserver. **Sample:** The following sample sends all messages to the central server "centralserv" at port 2514 (note that that server must run imrelp on port 2514). :: module(load="omrelp") action(type="omrelp" target="centralserv" port="2514") **Legacy Configuration Directives**: This module uses old-style action configuration to keep consistent with the forwarding rule. So far, no additional configuration directives can be specified. To send a message via RELP, use :: *.*  :omrelp::;