Skip to content

Commit

Permalink
Refactor project, better separate implementation.
Browse files Browse the repository at this point in the history
  • Loading branch information
Chavjoh committed Nov 3, 2024
1 parent 97c7f1e commit 4cdf1e5
Show file tree
Hide file tree
Showing 20 changed files with 649 additions and 603 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/snyk.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,6 @@ jobs:
args: --sarif-file-output=snyk.sarif --include-provided-dependencies=false

- name: Upload result to GitHub Code Scanning
uses: github/codeql-action/upload-sarif@v2
uses: github/codeql-action/upload-sarif@v3
with:
sarif_file: snyk.sarif
163 changes: 43 additions & 120 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# Elasticsearch Appender

[v1]: https://github.com/Chavaillaz/elasticsearch-log4j-appender/wiki/Appender-1.x-‐-Log4j-1.x
[v2]: https://github.com/Chavaillaz/elasticsearch-log4j-appender/wiki/Appender-2.x-‐-Log4j-2.x

![Quality Gate](https://github.com/chavaillaz/elasticsearch-log4j-appender/actions/workflows/sonarcloud.yml/badge.svg)
![Dependency Check](https://github.com/chavaillaz/elasticsearch-log4j-appender/actions/workflows/snyk.yml/badge.svg)
[![Maven Central](https://maven-badges.herokuapp.com/maven-central/com.chavaillaz/elasticsearch-log4j-appender/badge.svg)](https://maven-badges.herokuapp.com/maven-central/com.chavaillaz/elasticsearch-log4j-appender)
Expand All @@ -8,21 +11,21 @@
Elasticsearch appender allows you to send log events directly from Log4j to an elastic cluster. The delivery of logs is
asynchronous (i.e. not on the main thread) and therefore will not block execution of the program.

| Appender version | Log4j version | Elasticsearch version |
|------------------|---------------|-----------------------|
| 1.0.0 | 1.2.17 | 7.x |
| 2.0.0 | 2.17.1 | 7.x |
| 2.1.0 | 2.17.2 | 8.x |
| 2.1.1 | 2.19.0 | 8.x |
| 2.1.2 | 2.20.0 | 8.x |
| 2.1.3 | 2.24.1 | 8.x |
| Appender version | Log4j version | Elasticsearch version | Documentation |
|------------------|---------------|-----------------------|-----------------|
| 1.0.0 | 1.2.17 | 7.x | [Version 1][v1] |
| 2.0.0 | 2.17.1 | 7.x | [Version 2][v2] |
| 2.1.0 | 2.17.2 | 8.x | [Version 2][v2] |
| 2.1.1 | 2.19.0 | 8.x | [Version 2][v2] |
| 2.1.2 | 2.20.0 | 8.x | [Version 2][v2] |
| 2.1.3 | 2.24.1 | 8.x | [Version 2][v2] |
| 3.0.0 | 2.24.1 | 8.x | See below |

## Installation

The dependency is available in maven central (see badge and table above for the version):

```xml

<dependency>
<groupId>com.chavaillaz</groupId>
<artifactId>elasticsearch-log4j-appender</artifactId>
Expand All @@ -33,36 +36,27 @@ You then have to configure log4j in order to include this appender (see configur

## Configuration

In the configuration file, add a new appender `ElasticsearchAppender` (from package `com.chavaillaz.appender.log4j`)
with the following properties **(please note that for Log4j2 they all start with an uppercase letter)**:

- `applicationName` (default `unknown`). It can also be specified as environment variable or system
property `APPLICATION`.
- `hostName` (default is the machine host name). It can also be specified as environment variable or system
property `HOST`.
- `environmentName` (default `local`). It can be specified also as environment variable or system property `ENV`.
- `elasticConverter` (default `com.chavaillaz.appender.log4j.converter.DefaultEventConverter`) is the class used to
convert a logging event into a key/value document to be stored in Elasticsearch. It can also be specified as
environment variable or system property `CONVERTER`.
- `elasticIndex` (default `ha`) and `elasticIndexSuffix` (default `-yyyy.MM.dd`) form together the index name where the
messages are sent to. Note that `elasticIndexSuffix` must contain a format pattern suitable for `DateTimeFormatter`.
They can also both be specified with environment variables or system properties `INDEX` and `INDEX_SUFFIX`.
- `elasticUrl` is the address of the server (or its load balancer) in the format `[scheme://][host]:[port]`. The scheme
is optional and defaults to `http`.
- `elasticApiKey` (encoded) or `elasticUser` + `elasticPassword` are the credentials for the server.
- `elasticParallelExecution` (default `true`) specifies the way the messages are sent to the server
(`true` send them in a separate thread and `false` send them sequentially).
- `elasticBatchSize` (default `1`) is the number of messages threshold triggering the sending to the server.
- `elasticBatchInitialDelay` (default `1000`) is the time in milliseconds before a first batch of messages is sent to
the server (after appender startup, even if the batch of messages is incomplete, meaning not reaching the threshold).
- `elasticBatchDelay` (default `1000`) is the time in milliseconds between two cleanups of incomplete batches. If after
this time there are less than `elasticBatchSize` messages waiting to be sent, the batch is sent nonetheless. Note that
once `elasticBatchSize` messages are waiting, the batch is sent immediately, without any delay.

Note that `elasticUrl` is the only mandatory configuration to give, except if you need to overwrite the default value of
another ones.

### XML file example (log4j version 2)
In the Log4j configuration file, add a new appender `ElasticsearchAppender` using package
`com.chavaillaz.appender.log4j.elastic` with the following properties:

| Appender Property | Environment / System variable | Default value | Description |
|-------------------|-------------------------------|-----------------------------|-----------------------------------------------------------------------------------------------------------------------------------------|
| Application | APP | `unknown` | The name of the application generating the logs. |
| Host | HOST | Machine host name | The name of the host on which the application is running. |
| Environment | ENV | `local` | The name of the environment in which the application is running. |
| Converter | CONVERTER | `[...].DefaultLogConverter` | The path of the class used to convert logging events into key/value documents to be stored. |
| Index | INDEX | `ha` | The name of the Elasticsearch index to which the documents are sent. |
| IndexSuffix | INDEX_SUFFIX | - | The suffix added to the index name (using current date) in a format pattern suitable for `DateTimeFormatter`. |
| Url | ELASTIC_URL | - | The address of Elasticsearch in the format `scheme://host:port`. |
| User | ELASTIC_USER | - | The username to use as credentials to access Elasticsearch. |
| Password | ELASTIC_PASSWORD | - | The password to use as credentials to access Elasticsearch. |
| ApiKey | ELASTIC_API_KEY | - | The API key (already encoded) to use as credentials to access Elasticsearch. |
| FlushThreshold | - | `100` | The threshold number of messages triggering the transmission of documents to the server. |
| FlushInterval | - | `5000` | The time (ms) between two automatic flushes, which are triggering the transmission of logs, even if not reaching the defined threshold. |

Note that `Url` is the only mandatory configuration, except if you need to overwrite the default value of another ones.

## XML file example

```xml
<?xml version="1.0" encoding="UTF-8"?>
Expand All @@ -73,18 +67,16 @@ another ones.
</Console>
<ElasticsearchAppender name="Elasticsearch">
<PatternLayout pattern="%msg"/>
<ApplicationName>myApplication</ApplicationName>
<EnvironmentName>local</EnvironmentName>
<ElasticConverter>com.chavaillaz.appender.log4j.converter.DefaultEventConverter</ElasticConverter>
<ElasticUrl>http://localhost:9300</ElasticUrl>
<ElasticIndex>ha</ElasticIndex>
<ElasticIndexSuffix>-yyyy.MM.dd</ElasticIndexSuffix>
<ElasticUser>elastic</ElasticUser>
<ElasticPassword>changeme</ElasticPassword>
<ElasticParallelExecution>true</ElasticParallelExecution>
<ElasticBatchSize>10</ElasticBatchSize>
<ElasticBatchInitialDelay>1000</ElasticBatchInitialDelay>
<ElasticBatchDelay>1000</ElasticBatchDelay>
<Application>myApplication</Application>
<Environment>local</Environment>
<Converter>com.chavaillaz.appender.log4j.DefaultLogConverter</Converter>
<Index>ha</Index>
<IndexSuffix>-yyyy.MM.dd</IndexSuffix>
<Url>http://localhost:9300</Url>
<User>elastic</User>
<Password>changeme</Password>
<FlushThreshold>100</FlushThreshold>
<FlushInterval>5000</FlushInterval>
</ElasticsearchAppender>
</Appenders>
<Loggers>
Expand All @@ -96,75 +88,6 @@ another ones.
</Configuration>
```

### XML file example (log4j version 1)

```xml
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">

<appender name="ConsoleAppender" class="org.apache.log4j.ConsoleAppender">
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d{ISO8601} %p (%t) [%c{1}::%M] - %m%n"/>
</layout>
</appender>

<appender name="ElasticAppender" class="com.chavaillaz.appender.log4j.ElasticsearchAppender">
<param name="applicationName" value="myApplication"/>
<param name="environmentName" value="myEnvironment"/>
<param name="elasticConverter" value="com.package.CustomEventConverter"/>
<param name="elasticUrl" value="myElasticsearchUrl"/>
<param name="elasticIndex" value="myIndex"/>
<param name="elasticIndexSuffix" value="-yyyy.MM.dd"/>
<param name="elasticUser" value="myUser"/>
<param name="elasticPassword" value="myPassword"/>
<param name="elasticParallelExecution" value="true"/>
<param name="elasticBatchSize" value="10"/>
<param name="elasticBatchInitialDelay" value="1000"/>
<param name="elasticBatchDelay" value="1000"/>
</appender>

<!-- Avoid to propagate the logs from ElasticAppender to itself (cyclic) -->
<category name="com.chavaillaz.appender" additivity="false">
<priority value="warn"/>
<appender-ref ref="ConsoleAppender"/>
</category>

<root>
<priority value="info"/>
<appender-ref ref="ConsoleAppender"/>
<appender-ref ref="ElasticAppender"/>
</root>

</log4j:configuration>
```

### Property file example (log4j version 1)

```
log4j.appender.ELASTIC=com.chavaillaz.appender.log4j.ElasticsearchAppender
log4j.appender.ELASTIC.applicationName=myApplication
log4j.appender.ELASTIC.environmentName=myEnvironment
log4j.appender.ELASTIC.elasticConverter=com.package.CustomEventConverter
log4j.appender.ELASTIC.elasticUrl=myElasticsearchUrl
log4j.appender.ELASTIC.elasticIndex=myIndex
log4j.appender.ELASTIC.elasticIndexSuffix=-yyyy.MM.dd
log4j.appender.ELASTIC.elasticUser=myUser
log4j.appender.ELASTIC.elasticPassword=myPassword
log4j.appender.ELASTIC.elasticParallelExecution=true
log4j.appender.ELASTIC.elasticBatchSize=10
log4j.appender.ELASTIC.elasticBatchInitialDelay=1000
log4j.appender.ELASTIC.elasticBatchDelay=1000
# Avoid to propagate the logs from ElasticAppender to itself (cyclic)
log4j.logger.com.chavaillaz.appender=WARN, CONSOLE
log4j.logger.org.apache.http=WARN, CONSOLE
log4j.logger.org.elasticsearch.client=WARN, CONSOLE
log4j.additivity.com.chavaillaz.appender=false
log4j.additivity.org.apache.http=false
log4j.additivity.org.elasticsearch.client=false
```

## Contributing

If you have a feature request or found a bug, you can:
Expand Down
2 changes: 1 addition & 1 deletion pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

<groupId>com.chavaillaz</groupId>
<artifactId>elasticsearch-log4j-appender</artifactId>
<version>2.1.0-SNAPSHOT</version>
<version>3.0.0-SNAPSHOT</version>
<packaging>jar</packaging>

<name>${project.groupId}.${project.artifactId}</name>
Expand Down
55 changes: 55 additions & 0 deletions src/main/java/com/chavaillaz/appender/CommonUtils.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
package com.chavaillaz.appender;

import java.net.InetAddress;
import java.net.UnknownHostException;
import java.util.Optional;

import javax.net.ssl.SSLContext;
import lombok.SneakyThrows;
import lombok.experimental.UtilityClass;
import org.apache.http.conn.ssl.TrustAllStrategy;
import org.apache.http.ssl.SSLContextBuilder;

/**
* Common utility methods.
*/
@UtilityClass
public class CommonUtils {

/**
* Searches an environment or system property value.
*
* @param key The key to search in the properties
* @param defaultValue The default value to use in case the given key is not found
* @return The value found in environment/system properties or the given default value
*/
public static String getProperty(String key, String defaultValue) {
return Optional.ofNullable(System.getenv(key)).orElse(System.getProperty(key, defaultValue));
}

/**
* Creates a permissive SSL context trusting everything.
*
* @return The SSL context
*/
@SneakyThrows
public static SSLContext createSSLContext() {
return new SSLContextBuilder()
.loadTrustMaterial(null, TrustAllStrategy.INSTANCE)
.build();
}

/**
* Gets the current host name of the machine.
*
* @return The host name or {@code localhost} if it cannot be determined
*/
public static String getInitialHostname() {
try {
return InetAddress.getLocalHost().getCanonicalHostName();
} catch (UnknownHostException e) {
return "localhost";
}
}

}
82 changes: 82 additions & 0 deletions src/main/java/com/chavaillaz/appender/LogConfiguration.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
package com.chavaillaz.appender;

import java.time.Duration;

/**
* Configuration for logs transmission.
*/
public interface LogConfiguration {

/**
* Gets the name of the application generating the logs.
*
* @return The application name
*/
String getApplication();

/**
* Sets the name of the application generating the logs.
*
* @param application The application name
*/
void setApplication(String application);

/**
* Gets the name of the host on which the application is running.
*
* @return The host name
*/
String getHost();

/**
* Sets the name of the host on which the application is running.
*
* @param host The host name
*/
void setHost(String host);

/**
* Sets the environment in which the application is running.
*
* @return The environment name
*/
String getEnvironment();

/**
* Sets the environment in which the application is running.
*
* @param environment The environment name
*/
void setEnvironment(String environment);

/**
* Gets the threshold number of messages triggering the transmission of the logs.
*
* @return The threshold number of messages
*/
long getFlushThreshold();

/**
* Sets the threshold number of messages triggering the transmission of the logs.
*
* @param messageNumber The threshold number of messages
*/
void setFlushThreshold(long messageNumber);

/**
* Gets the maximum time interval between two flushes.
* After each interval, the transmission of logs is triggered, even if not reaching the threshold number of messages.
*
* @return The maximum interval between two flushes
*/
Duration getFlushInterval();

/**
* Gets the maximum time interval between two flushes.
* After each interval, the transmission of logs is triggered, even if not reaching the threshold number of messages.
*
* @param flushInterval The maximum interval between two flushes
*/
void setFlushInterval(Duration flushInterval);

}
31 changes: 31 additions & 0 deletions src/main/java/com/chavaillaz/appender/LogDelivery.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
package com.chavaillaz.appender;

import java.util.List;
import java.util.Map;

/**
* Transmission of logs to an external monitoring system.
*/
public interface LogDelivery extends AutoCloseable {

/**
* Sends the given record data.
*
* @param document The record data
*/
void send(Map<String, Object> document);

/**
* Sends the given list of record data.
*
* @param documents The list of record data
*/
void send(List<Map<String, Object>> documents);

/**
* Sends the pending records (generating a partially filled batch).
* Intended to be called periodically to clean out pending messages.
*/
void flush();

}
Loading

0 comments on commit 4cdf1e5

Please sign in to comment.