diff --git a/doc/code_snippets/snippets/config/instances.enabled/audit_log/config.yaml b/doc/code_snippets/snippets/config/instances.enabled/audit_log/config.yaml index 257ef47c8a..edb57fa298 100644 --- a/doc/code_snippets/snippets/config/instances.enabled/audit_log/config.yaml +++ b/doc/code_snippets/snippets/config/instances.enabled/audit_log/config.yaml @@ -10,7 +10,7 @@ groups: audit_log: to: file file: 'audit_tarantool.log' - filter: [ user_create,data_operations,ddl ] + filter: [user_create,data_operations,ddl,custom] format: json - spaces: [ bands ] + spaces: [bands] extract_key: true \ No newline at end of file diff --git a/doc/code_snippets/snippets/config/instances.enabled/audit_log/myapp.lua b/doc/code_snippets/snippets/config/instances.enabled/audit_log/myapp.lua new file mode 100644 index 0000000000..2055deba62 --- /dev/null +++ b/doc/code_snippets/snippets/config/instances.enabled/audit_log/myapp.lua @@ -0,0 +1,31 @@ +-- myapp.lua -- +local audit = require('audit') + +-- Log message string +audit.log('Hello, Alice!') +-- Log format string and arguments +audit.log('Hello, %s!', 'Bob') +-- Log table with audit log field values +audit.log({type = 'custom_hello', description = 'Hello, World!'}) +audit.log({type = 'custom_farewell', user = 'eve', module = 'custom', description = 'Farewell, Eve!'}) +-- Create a new log module +local my_audit = audit.new({type = 'custom_hello', module = 'my_module'}) +my_audit:log('Hello, Alice!') +my_audit:log({tag = 'admin', description = 'Hello, Bob!'}) +-- Log 'Hello!' message with the VERBOSE severity level +audit.log({severity = 'VERBOSE', description = 'Hello!'}) + +-- Log 'Hello!' message with a shortcut helper function +audit.verbose('Hello!') + +-- Like audit.log(), a shortcut helper function accepts a table of options +audit.verbose({description = 'Hello!'}) + +-- Severity levels are available for custom loggers +local my_logger = audit.new({module = 'my_module'}) +my_logger:log({severity = 'ALARM', description = 'Alarm'}) +my_logger:alarm('Alarm') + +-- Overwrite session_type and remote fields +audit.log({type = 'custom_hello', description = 'Hello!', + session_type = 'my_session', remote = 'my_remote'}) \ No newline at end of file diff --git a/doc/enterprise/audit_log.rst b/doc/enterprise/audit_log.rst index 8ddb65d05e..d85b0db494 100644 --- a/doc/enterprise/audit_log.rst +++ b/doc/enterprise/audit_log.rst @@ -1,8 +1,10 @@ -.. _enterprise_audit_module: +.. _enterprise_audit_module: Audit module ============ +**Example on GitHub**: `audit_log `_ + The audit module available in Tarantool Enterprise Edition writes messages that record Tarantool events in plain text, CSV, or JSON format. @@ -14,11 +16,171 @@ It is up to each company to decide exactly what activities to audit and what act System administrators, security engineers, and people in charge of the company may want to audit different events for different reasons. Tarantool provides such an option for each of them. -.. _audit-log-events: +.. _audit-log-configure: + +Configuring audit log +--------------------- + +The section describes how to enable and configure audit logging and write logs to a selected destination -- a file, a pipe, +or a system logger. + +Read more: :ref:`Audit log configuration reference `. + +.. _audit-log-enable: + +Enable audit logging +~~~~~~~~~~~~~~~~~~~~ + +To enable audit logging, define the log location using the +:ref:`audit_log.to ` option in the ``audit_log`` section of the configuration file. +Possible log locations: + +* a :ref:`file ` +* a :ref:`pipe ` +* a :ref:`system logger ` + +To disable audit logging, set the ``audit_log`` option to ``devnull``. + +In the configuration below, the :ref:`audit_log.to ` option is set to ``file``. +It means that the logs are written to a file. +In this case, you should also define a file path (for example, ``audit_tarantool.log``) using +the :ref:`audit_log.file ` option. +If the option is omitted, the default path is used: ``var/log/instance001/audit.log``. + +.. literalinclude:: /code_snippets/snippets/config/instances.enabled/audit_log/config.yaml + :language: yaml + :start-at: audit_log: + :end-at: 'audit_tarantool.log' + :dedent: + +If you log to a file, Tarantool reopens the audit log at `SIGHUP `_. + +.. _audit-log-filters: + +Filter the events +~~~~~~~~~~~~~~~~~ + +Tarantool's extensive filtering options help you write only the events you need to the audit log. +To select the recorded events, use the :ref:`audit_log.filter ` option. +Its value can be a list of events and event groups. +You can customize the filters and use different combinations of them for your purposes. +Possible filtering options: + +- Filter by event. You can set a list of :ref:`events ` to be recorded. For example, select + ``password_change`` to monitor the users who have changed their passwords: + + .. code-block:: yaml + + audit_log: + filter: [password_change] + +- Filter by group. You can specify a list of :ref:`event groups ` to be recorded. For example, + select ``auth`` and ``priv`` to see the events related to authorization and granted privileges: + + .. code-block:: yaml + + audit_log: + filter: [auth,priv] + +- Filter by group and event. You can specify a group and a certain event depending on the purpose. + In the configuration below, ``user_create``, ``data_operations``, ``ddl``, and ``custom`` are selected to see the events related to + + * user creation + * space creation, altering, and dropping + * data modification or selection from spaces + * :ref:`custom events ` (any events added manually using the audit module API) + + .. literalinclude:: /code_snippets/snippets/config/instances.enabled/audit_log/config.yaml + :language: yaml + :start-at: filter: + :end-at: [user_create,data_operations,ddl] + :dedent: + +.. _audit-log-format: + +Set the format of audit log events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Use the :ref:`audit_log.format ` option to choose the format of audit log events +-- plain text, CSV, or JSON. + +JSON is used by default. It is more convenient to receive log events, analyze them, and integrate them with other systems if needed. +The plain format can be efficiently compressed. +The CSV format allows you to view audit log events in tabular form. + +.. literalinclude:: /code_snippets/snippets/config/instances.enabled/audit_log/config.yaml + :language: yaml + :start-at: format: + :end-at: json + :dedent: + +.. _audit-log-spaces: + +Specify the spaces to be logged +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Since: :doc:`3.0.0 `, :ref:`audit_log.spaces ` is used to specify +a list of space names for which data operation events should be logged. + +In the configuration below, only the events from the ``bands`` space are logged: + +.. literalinclude:: /code_snippets/snippets/config/instances.enabled/audit_log/config.yaml + :language: yaml + :start-at: spaces: + :end-at: [bands] + :dedent: + +.. _audit-log-extract: + +Specify the logging mode in DML events +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Since: :doc:`3.0.0 `, it is possible to +force the audit subsystem to log the primary key instead of a full tuple in DML operations. +To do so, set the :ref:`audit_log.extract_key ` option to ``true``. + +.. literalinclude:: /code_snippets/snippets/config/instances.enabled/audit_log/config.yaml + :language: yaml + :start-at: extract_key: + :end-at: true + :dedent: + +.. _audit-log-run-read: + +Reading audit logs +------------------ + +The audit log entry in the file might look as follows: + +.. _audit-log-read: + +Read commands +------------- + +To easily read the audit log events in the needed form, use the different commands: + +* ``cat`` -- prints one or more files + +* ``grep`` -- prints a specific text + +* ``head`` -- prints the first N lines of the file + +* ``tail`` -- prints the last N lines of the file + + .. note:: + + These are the basic commands to help you read the logs. If necessary, you can use other commands. + +.. _audit-log-events: Audit log events ---------------- +.. _audit-log-events-types: + +Events types +~~~~~~~~~~~~ + The Tarantool audit log module can record various events that you can monitor and decide whether you need to take actions: @@ -33,7 +195,7 @@ decide whether you need to take actions: * System events -- events related to modification or configuration of resources. For example, such logs record the replacement of a space. -* :ref:`User-defined events `-- any events added manually using +* :ref:`Custom events `-- any events added manually using the audit module API. The full list of available audit log events is provided in the table below: @@ -48,7 +210,7 @@ The full list of available audit log events is provided in the table below: - Type of event written to the audit log - Severity level - Example of an event description - * - Audit log enabled for events + * - Audit log enabled for events - ``audit_enable`` - ``VERBOSE`` - @@ -56,15 +218,15 @@ The full list of available audit log events is provided in the table below: - ``custom`` - ``INFO`` (default) - - * - User authorized successfully + * - User authorized successfully - ``auth_ok`` - ``VERBOSE`` - ``Authenticate user `` - * - User authorization failed + * - User authorization failed - ``auth_fail`` - ``ALARM`` - ``Failed to authenticate user `` - * - User logged out or quit the session + * - User logged out or quit the session - ``disconnect`` - ``VERBOSE`` - ``Close connection`` @@ -84,11 +246,11 @@ The full list of available audit log events is provided in the table below: - ``role_drop`` - ``INFO`` - ``Drop role `` - * - User disabled + * - User disabled - ``user_disable`` - ``INFO`` - ``Disable user `` - * - User enabled + * - User enabled - ``user_enable`` - ``INFO`` - ``Enable user `` @@ -132,40 +294,40 @@ The full list of available audit log events is provided in the table below: - ``space_create`` - ``INFO`` - ``Create space `` - * - Space altered + * - Space altered - ``space_alter`` - ``INFO`` - ``Alter space `` - * - Space dropped + * - Space dropped - ``space_drop`` - ``INFO`` - ``Drop space `` - * - Tuple inserted into space + * - Tuple inserted into space - ``space_insert`` - ``VERBOSE`` - ``Insert tuple into space `` - * - Tuple replaced in space + * - Tuple replaced in space - ``space_replace`` - ``VERBOSE`` - ``Replace tuple with in space `` - * - Tuple deleted from space + * - Tuple deleted from space - ``space_delete`` - ``VERBOSE`` - ``Delete tuple from space `` .. note:: - + The ``eval`` event displays data from the ``console`` module and the ``eval`` function of the ``net.box`` module. For more on how they work, see :ref:`Module console ` - and :ref:`Module net.box -- eval `. + and :ref:`Module net.box -- eval `. To separate the data, specify ``console`` or ``binary`` in the session field. .. _audit-log-structure: Structure of audit log event ----------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Each audit log event contains a number of fields that can be used to filter and aggregate the resulting logs. An example of a Tarantool audit log entry in JSON: @@ -203,7 +365,9 @@ Each event consists of the following fields: - A unique identifier of audit log event - ``cb44fb2b-5c1f-4c4b-8f93-1dd02a76cec0`` * - ``severity``. Since :doc:`3.0.0 ` - - A severity level + - A severity level. Each system audit event has a severity level determined by its importance. + :ref:`Custom events ` have the ``INFO`` severity level by default. + - ``VERBOSE`` * - ``remote`` - Remote host that triggered the event @@ -232,10 +396,11 @@ Each event consists of the following fields: You can set all these parameters only once. + .. _audit-log-event-groups: Event groups ------------- +~~~~~~~~~~~~ Built-in event groups are used to filter the event types that you want to audit. For example, you can set to record only authorization events, @@ -280,251 +445,122 @@ Tarantool provides the following event groups: It is recommended that you select only those groups whose events your company needs to monitor and analyze. -.. _audit-log-custom: +.. _audit-log-custom: -Create user-defined events --------------------------- +Creating custom events +---------------------- Tarantool provides an API for writing user-defined audit log events. -To enable custom events, specify the ``custom`` value in the :ref:`audit_log.filter ` option. - -To add a new event, use the ``audit.log()`` function that takes one of the following values: - -* Message string. Printed to the audit log with type ``message``. - Example: ``audit.log('Hello, World!')``. - -* Format string and arguments. Passed to string format and then output to the audit log with type ``message``. - Example: ``audit.log('Hello, %s!', 'World')``. - -* Table with audit log field values. The table must contain at least one field -- ``description``. - Example: ``audit.log({type = 'custom_hello', description = 'Hello, World!'})``. - -Using the option ``audit.new()``, you can create a new log module that allows you -to avoid passing all custom audit log fields each time ``audit.log()`` is called. -It takes a table of audit log field values (same as ``audit.log()``). The ``type`` -of the log module for writing user-defined events must either be ``message`` or -have the ``custom_`` prefix. - -Example -~~~~~~~ - -.. code-block:: lua - - audit.log({type = 'custom_hello', module = 'my_module', description = 'Hello, Alice!' }) - audit.log({type = 'custom_hello', module = 'my_module', tag = 'admin', description = 'Hello, Bob!'}) - - -Some user-defined audit log fields (``time``, ``remote``, ``session_type``) -are set in the same way as for a system event. -If a field is not overwritten, it is set to the same value as for a system event. - -Some audit log fields you can overwrite with ``audit.new()`` and ``audit.log()``: - -* ``type`` -* ``user`` -* ``module`` -* ``tag`` -* ``description`` - -.. note:: - - To avoid confusion with system events, the value of the type field must either be ``message`` (default) - or begin with the ``custom_`` prefix. Otherwise, you receive the error message. - User-defined events are filtered out by default.. - - -.. _audit-log-example: - -Example: using Tarantool audit log ----------------------------------- - -The example shows how to enable and configure audit logging and write logs to a selected destination (a file, a pipe, -or a system logger). - -Before starting this example: - -#. Install the :ref:`tt ` utility. - -#. Create a tt environment in the current directory by executing the :ref:`tt init ` command. - -#. Inside the ``instances.enabled`` directory of the created tt environment, create the ``audit_log`` directory. - -#. Inside ``instances.enabled/audit_log``, create the ``instances.yml`` and ``config.yaml`` files: - - - ``instances.yml`` specifies instances to run in the current environment and should look like this: - - .. literalinclude:: /code_snippets/snippets/config/instances.enabled/audit_log/instances.yml - :language: yaml - :end-at: instance001: - :dedent: - - - The ``config.yaml`` file stores an audit log configuration (the ``audit_log`` section). The configuration is described in detail in the next section. - -.. _audit-log-enable: - -Enable audit logging -~~~~~~~~~~~~~~~~~~~~ - -To enable audit logging, define the log location using the -:ref:`audit_log.to ` option in the ``audit_log`` section of the configuration file. -Possible logs locations: - -* a :ref:`file ` -* a :ref:`pipe ` -* a :ref:`system logger ` - -To disable audit logging, set the ``audit_log`` option to ``devnull``. - -In this tutorial, the logs are written to a file. To do this, set the -:ref:`audit_log.to ` option to ``file``. -After that, you can define a file path (for example, ``audit_tarantool.log``) using -the :ref:`audit_log.file ` option. -If the option is omitted, the default path is used: ``var/log/instance001/audit.log``. - -The configuration might look as follows: +To enable custom events, specify the ``custom`` value in the :ref:`audit_log.filter ` option: .. literalinclude:: /code_snippets/snippets/config/instances.enabled/audit_log/config.yaml :language: yaml - :start-at: audit_log: - :end-at: audit_tarantool.log + :start-at: filter: + :end-at: [user_create,data_operations,ddl,custom] :dedent: -If you log to a file, Tarantool reopens the audit log at `SIGHUP `_. - -.. _audit-log-filters: - -Filter the events -~~~~~~~~~~~~~~~~~ - -Tarantool's extensive filtering options help you write only the events you need to the audit log. - -To select the recorded events, use the :ref:`audit_log.filter ` option. -Its value can be a list of events and event groups. -You can customize the filters and use different combinations of them for your purposes. -Possible filtering options: - -- Filter by event. You can set a list of :ref:`events ` to be recorded. For example, select - ``password_change`` to monitor the users who have changed their passwords: - - .. code-block:: yaml - - audit_log: - filter: [ password_change ] - -- Filter by group. You can specify a list of :ref:`event groups ` to be recorded. For example, - select ``auth`` and ``priv`` to see only events related to authorization and granted privileges. - - .. code-block:: yaml +.. _audit-log-custom-new: - audit_log: - filter: [ auth,priv ] +Add a new custom event +~~~~~~~~~~~~~~~~~~~~~~ -- Filter by group and event. You can specify a group and a certain event depending on the purpose. - For example, you can select ``user_create``, ``data_operations``, and ``ddl`` to see the events related to - - * user creation - * space creation, altering, and dropping - * data modification or selection from spaces +To add a new event, use the ``audit.log()`` function that takes one of the following values: - Let's add this filter to our configuration file: +* Message string. Printed to the audit log with type ``message``: - .. literalinclude:: /code_snippets/snippets/config/instances.enabled/audit_log/config.yaml + .. literalinclude:: /code_snippets/snippets/config/instances.enabled/audit_log/myapp.lua :language: yaml - :start-at: audit_log: - :end-at: audit_tarantool.log + :start-after: Log message string + :end-before: Log format string and arguments :dedent: +* Format string and arguments. Passed to string format and then output to the audit log with type ``message``: -.. _audit-log-format: - -Set the format of audit log events -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Use the :ref:`audit_log.format ` option to choose the format of audit log events --- plain text, CSV, or JSON. - -JSON is used by default. It is more convenient to receive log events, analyze them, and integrate them with other systems if needed. -The plain format can be efficiently compressed. -The CSV format allows you to view audit log events in tabular form. - -In this tutorial, set the format to JSON: - -.. literalinclude:: /code_snippets/snippets/config/instances.enabled/audit_log/config.yaml - :language: yaml - :start-at: audit_log: - :end-at: format: json - :dedent: - -.. _audit-log-spaces: + .. literalinclude:: /code_snippets/snippets/config/instances.enabled/audit_log/myapp.lua + :language: yaml + :start-after: Log format string and arguments + :end-before: Log table with audit log field values + :dedent: -Specify the spaces to be logged -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +* Table with audit log field values. The table must contain at least one field -- ``description``. -Since: :doc:`3.0.0 `, it is possible to :ref:`specify a list of space names ` -for which data operation events should be logged. + .. literalinclude:: /code_snippets/snippets/config/instances.enabled/audit_log/myapp.lua + :language: yaml + :start-after: Log table with audit log field values + :end-at: description = 'Farewell, Eve!'}) + :dedent: -To log the events from the ``bands`` space only, specify the option in the configuration file: +Alternatively, you can use ``audit.new()`` to create a new log module. +This allows you to avoid passing all custom audit log fields each time ``audit.log()`` is called. +The ``audit.new()`` function takes a table of audit log field values (same as ``audit.log()``). +The ``type`` of the log module for writing custom events must either be ``message`` or have the ``custom_`` prefix. -.. literalinclude:: /code_snippets/snippets/config/instances.enabled/audit_log/config.yaml +.. literalinclude:: /code_snippets/snippets/config/instances.enabled/audit_log/myapp.lua :language: yaml - :liines: 15 + :start-after: Create a new log module + :end-before: Log 'Hello!' message with the VERBOSE severity level :dedent: -.. _audit-log-extract: - -Specify the logging mode in DML events -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Since: :doc:`3.0.0 `, it is possible to -force the audit subsystem to log the primary key instead of a full tuple in DML operations. +.. _audit-log-custom-field-overwrite: -To do so, set the :ref:`audit_log.extract_key ` option to ``true``. +Overwrite custom event fields +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The resulting configuration in ``config.yaml`` now looks as follows: +It is possible to overwrite most of the custom audit log :ref:`fields ` using ``audit.new()`` or ``audit.log()``. +The only audit log field that cannot be overwritten is ``time``. -.. literalinclude:: /code_snippets/snippets/config/instances.enabled/audit_log/config.yaml +.. literalinclude:: /code_snippets/snippets/config/instances.enabled/audit_log/myapp.lua :language: yaml + :start-after: Overwrite session_type and remote fields + :end-at: remote = 'my_remote'}) :dedent: -.. _audit-log-run-example: - -Check the example -~~~~~~~~~~~~~~~~~ - -After the configuration is done, execute the :ref:`tt start ` command from the :ref:`tt environment directory `: - - .. code-block:: console +If omitted, the ``session_type`` is set to the current session type, ``remote`` is set to the remote peer address. - $ tt start audit_log - • Starting an instance [audit_log:instance001]... - -After that, connect to the instance with :ref:`tt connect `: +.. note:: -.. code-block:: console + To avoid confusion with system events, the value of the type field must either be ``message`` (default) + or begin with the ``custom_`` prefix. Otherwise, you receive the error message. + User-defined events are filtered out by default. - $ tt connect audit_log:instance001 +.. _audit-log-custom-severity: -In the interactive console. run the following commands: +Severity level +~~~~~~~~~~~~~~ +By default, custom events have the ``INFO`` :ref:`severity level `. +To override the level, you can: -.. _audit-log-read: +* specify the ``severity`` field +* use a shortcut function -Use read commands ------------------ +The following shortcuts are available: -To easily read the audit log events in the needed form, use the different commands: +.. container:: table -* ``cat`` -- prints one or more files + .. list-table:: + :widths: 40 60 + :header-rows: 1 -* ``grep`` -- prints a specific text + * - Shortcut + - Equivalent + * - ``audit.verbose(...)`` + - ``audit.log({severity = 'VERBOSE', ...})`` + * - ``audit.info(...)`` + - ``audit.log({severity = 'INFO', ...})`` + * - ``audit.warning(...)`` + - ``audit.log({severity = 'WARNING', ...})`` + * - ``audit.alarm(...)`` + - ``audit.log({severity = 'ALARM', ...})`` -* ``head`` -- prints the first N lines of the file +**Example** -* ``tail`` -- prints the last N lines of the file +.. literalinclude:: /code_snippets/snippets/config/instances.enabled/audit_log/myapp.lua + :language: yaml + :start-at: audit.log({severity = 'VERBOSE' + :end-at: my_logger:alarm('Alarm') + :dedent: - .. note:: - - These are the basic commands to help you read the logs. If necessary, you can use other commands. .. _audit-log-tips: