Skip to content

Commit

Permalink
prep v0.0.1 proof of concept
Browse files Browse the repository at this point in the history
  • Loading branch information
R. S. Doiel committed Jan 24, 2025
1 parent 0b671fd commit 28a5c56
Show file tree
Hide file tree
Showing 73 changed files with 2,662 additions and 39 deletions.
4 changes: 2 additions & 2 deletions CITATION.cff
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ authors:
given-names: R. S.
orcid: 'https://orcid.org/0000-0003-0900-6903'
repository-code: 'git+https://github.com/caltechlibrary/logagent'
version: 0.0.0
license-url: 'https://spdx.org/licenses/BSD-3-Clause'
version: 0.0.1
license-url: 'https://data.caltech.edu/license'
keywords:
- logging
- agents
73 changes: 73 additions & 0 deletions INSTALL.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
<html lang="en-US">
<head>
<title>logagent</title>
<link rel="stylesheet" href="/css/site.css">
</head>
<body>
<nav>
<ul>
<li><a href="/">Home</a></li>
<li><a href="index.html">README</a></li>
<li><a href="LICENSE">LICENSE</a></li>
<li><a href="INSTALL.html">INSTALL</a></li>
<li><a href="user_manual.html">User Manual</a></li>
<li><a href="about.html">About</a></li>
<li><a href="search.html">Search</a></li>
<li><a href="https://github.com/caltechlibrary/logagent">GitHub</a></li>
</ul>
</nav>
<section>
<h1 id="installation-for-development-of-log-agent">Installation for
development of <strong>Log Agent</strong></h1>
<p><strong>Log Agent</strong> is an experimental, proof of concept, log
processor. Log Agent scans each line of a log file for an explicit
string, finds the IP address and in the log line and applies an
designated action.</p>
<p>It provides a <code>logagent</code> command line program.</p>
<h2 id="quick-install-with-curl-or-irm">Quick install with curl or
irm</h2>
<p>There is an experimental installer.sh script that can be run with the
following command to install latest table release. This may work for
macOS, Linux and if you’re using Windows with the Unix subsystem. This
would be run from your shell (e.g. Terminal on macOS).</p>
<pre class="shell"><code>curl https://caltechlibrary.github.io/logagent/installer.sh | sh</code></pre>
<p>This will install <code>logagent</code> in your
<code>$HOME/bin</code> directory.</p>
<p>If you are running Windows 10 or 11 use the Powershell command
below.</p>
<div class="sourceCode" id="cb2"><pre
class="sourceCode ps1"><code class="sourceCode powershell"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a><span class="fu">irm</span> https<span class="op">://</span>caltechlibrary<span class="op">.</span><span class="fu">github</span><span class="op">.</span><span class="fu">io</span><span class="op">/</span>logagent<span class="op">/</span>installer<span class="op">.</span><span class="fu">ps1</span> <span class="op">|</span> <span class="fu">iex</span></span></code></pre></div>
<h2 id="installing-from-source">Installing from source</h2>
<h3 id="required-software">Required software</h3>
<ol type="1">
<li>Git (to clone the cold repository on GitHub)</li>
<li>Deno &gt;= 2.1.7</li>
</ol>
<h3 id="compiling-logagent">Compiling <strong>LogAgent</strong></h3>
<p>Deno is used to compile the TypeScript and dependent JavaScript files
into an executable.</p>
<ol type="1">
<li>Use Deno’s task to build project</li>
</ol>
<pre class="shell"><code>deno task build</code></pre>
<p>This will provide the <code>logagent</code> command in the “bin”
folder in your repository directory.</p>
<p>You can check to make sure <code>logagent</code> works for your
system. The compiled version is self contain and can be copied someplace
in your path.</p>
<h3 id="manual-install-on-posix">Manual install on POSIX</h3>
<pre class="shell"><code>mkdir -p $HOME/bin
export PATH=&quot;$HOME/bin:$PATH&quot;
cp bin/logagent $HOME/bin/
export MANPATH=&quot;$MANPATH:$HOME/man&quot;
cp -vR man $HOME/</code></pre>
<h3 id="manual-install-for-windows-via-powershell">Manual install for
Windows via Powershell</h3>
<div class="sourceCode" id="cb5"><pre
class="sourceCode ps1"><code class="sourceCode powershell"><span id="cb5-1"><a href="#cb5-1" aria-hidden="true" tabindex="-1"></a>mkdir <span class="va">$HOME</span><span class="op">/</span>bin</span>
<span id="cb5-2"><a href="#cb5-2" aria-hidden="true" tabindex="-1"></a><span class="va">$PATH</span> <span class="op">=</span> <span class="op">[</span>Environment<span class="op">]::</span>GetEnvironmentVariable<span class="op">(</span><span class="st">&quot;PATH&quot;</span><span class="op">)</span></span>
<span id="cb5-3"><a href="#cb5-3" aria-hidden="true" tabindex="-1"></a><span class="op">[</span>Environment<span class="op">]::</span>SetEnvironmentVariable<span class="op">(</span><span class="st">&quot;PATH&quot;</span><span class="op">,</span> <span class="st">&quot;</span><span class="va">$PATH</span><span class="st">;</span><span class="va">$HOME</span><span class="st">/bin&quot;</span><span class="op">)</span></span>
<span id="cb5-4"><a href="#cb5-4" aria-hidden="true" tabindex="-1"></a><span class="fu">copy</span> <span class="op">./</span>bin<span class="op">/</span>logagent<span class="op">.</span><span class="fu">exe</span> <span class="va">$HOME</span><span class="op">/</span>bin<span class="op">/</span></span></code></pre></div>
</section>
</body>
</html>
64 changes: 64 additions & 0 deletions INSTALL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
Installation for development of **Log Agent**
===========================================

**Log Agent** is an experimental, proof of concept, log processor. Log Agent scans each line of a log file for an explicit string, finds the IP address and in the log line and applies an designated action.

It provides a `logagent` command line program.

Quick install with curl or irm
------------------------------

There is an experimental installer.sh script that can be run with the following command to install latest table release. This may work for macOS, Linux and if you’re using Windows with the Unix subsystem. This would be run from your shell (e.g. Terminal on macOS).

~~~shell
curl https://caltechlibrary.github.io/logagent/installer.sh | sh
~~~

This will install `logagent` in your `$HOME/bin` directory.

If you are running Windows 10 or 11 use the Powershell command below.

~~~ps1
irm https://caltechlibrary.github.io/logagent/installer.ps1 | iex
~~~

Installing from source
----------------------

### Required software

1. Git (to clone the cold repository on GitHub)
2. Deno >= 2.1.7

### Compiling **LogAgent**

Deno is used to compile the TypeScript and dependent JavaScript files into an executable.

1. Use Deno's task to build project

~~~shell
deno task build
~~~

This will provide the `logagent` command in the "bin" folder in your repository directory.

You can check to make sure `logagent` works for your system. The compiled version is self contain and can be copied someplace in your path.

### Manual install on POSIX

~~~shell
mkdir -p $HOME/bin
export PATH="$HOME/bin:$PATH"
cp bin/logagent $HOME/bin/
export MANPATH="$MANPATH:$HOME/man"
cp -vR man $HOME/
~~~

### Manual install for Windows via Powershell

~~~ps1
mkdir $HOME/bin
$PATH = [Environment]::GetEnvironmentVariable("PATH")
[Environment]::SetEnvironmentVariable("PATH", "$PATH;$HOME/bin")
copy ./bin/logagent.exe $HOME/bin/
~~~
170 changes: 170 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,170 @@
#
# A Deno project makefile
#
PROJECT = LogAgent

PROGRAMS = logagent

TS_MODS = logagent.ts

GIT_GROUP = caltechlibrary

VERSION = $(shell grep '"version":' codemeta.json | cut -d\" -f 4)

BRANCH = $(shell git branch | grep '* ' | cut -d\ -f 2)

MAN_PAGES_1 = $(shell ls -1 *.1.md | sed -E 's/\.1.md/.1/g')

MAN_PAGES_3 = $(shell ls -1 *.3.md | sed -E 's/\.3.md/.3/g')

MAN_PAGES_7 = $(shell ls -1 *.7.md | sed -E 's/\.7.md/.7/g')

RELEASE_DATE=$(shell date +'%Y-%m-%d')

RELEASE_HASH=$(shell git log --pretty=format:%h -n 1)

HTML_PAGES = $(shell ls -1 *.html) # $(shell ls -1 *.md | grep -v 'nav.md' | sed -E 's/.md/.html/g')

OS = $(shell uname)

EXT =
ifeq ($(OS), Windows)
EXT = .exe
endif

PREFIX = $(HOME)

TS_MODS = $(shell ls -1 *.ts | grep -v _test.ts | grep -v deps.ts | grep -v version.ts)

build: version.ts CITATION.cff about.md bin compile installer.sh installer.ps1

bin: .FORCE
mkdir -p bin

compile: .FORCE
deno task build

check: .FORCE
deno task check

version.ts: codemeta.json
deno task version.ts

format: $(shell ls -1 *.ts | grep -v version.ts | grep -v deps.ts)

$(shell ls -1 *.ts | grep -v version.ts): .FORCE
deno fmt $@

man: $(MAN_PAGES_1) # $(MAN_PAGES_3) $(MAN_PAGES_7)

$(MAN_PAGES_1): .FORCE
mkdir -p man/man1
pandoc $@.md --from markdown --to man -s >man/man1/$@

CITATION.cff: codemeta.json
deno task CITATION.cff

about.md: codemeta.json
deno task about.md

status:
git status

save:
if [ "$(msg)" != "" ]; then git commit -am "$(msg)"; else git commit -am "Quick Save"; fi
git push origin $(BRANCH)

website: $(HTML_PAGES) .FORCE
make -f website.mak
cd presentations && make || exit 1

presentations: .FORCE
cd presentations && make || exit 1

publish: website .FORCE
./publish.bash

htdocs: .FORCE
deno task htdocs
deno task transpile

test: .FORCE
deno task test
deno task editor_test.ts

install: compile .FORCE
mkdir -p "${HOME}/bin"
cp -v "./bin/logagent$(EXT)" "${HOME}/bin"
cp -vR "./man" "${HOME}/"

uninstall: .FORCE
rm "${HOME}/bin/logagent$(EXT)"
rm man/man1/logagent.1

installer.sh: .FORCE
@echo '' | pandoc --metadata title="Installer" --metadata git_org_or_person="$(GIT_GROUP)" --metadata-file codemeta.json --template codemeta-bash-installer.tmpl >installer.sh
chmod 775 installer.sh
git add -f installer.sh

installer.ps1: .FORCE
@echo '' | pandoc --metadata title="Installer" --metadata git_org_or_person="$(GIT_GROUP)" --metadata-file codemeta.json --template codemeta-ps1-installer.tmpl >installer.ps1
chmod 775 installer.ps1
git add -f installer.ps1

clean:
if [ -d bin ]; then rm -fR bin/*; fi
if [ -d dist ]; then rm -fR dist/*; fi

release: clean build man website distribute_docs dist/Linux-x86_64 dist/Linux-aarch64 dist/macOS-x86_64 dist/macOS-arm64 dist/Windows-x86_64
echo "Ready to do ./release.bash"

setup_dist: .FORCE
@rm -fR dist
@mkdir -p dist

distribute_docs: website man setup_dist
@cp README.md dist/
@cp LICENSE dist/
@cp codemeta.json dist/
@cp CITATION.cff dist/
@cp *.1.md dist/
@cp INSTALL.md dist/
@cp -vR man dist/

dist/Linux-x86_64: .FORCE
@mkdir -p dist/bin
deno task dist_linux_x86_64
@cd dist && zip -r $(PROJECT)-v$(VERSION)-Linux-x86_64.zip LICENSE codemeta.json CITATION.cff *.md bin/*
@rm -fR dist/bin

dist/Linux-aarch64: .FORCE
@mkdir -p dist/bin
deno task dist_linux_aarch64
@cd dist && zip -r $(PROJECT)-v$(VERSION)-Linux-aarch64.zip LICENSE codemeta.json CITATION.cff *.md bin/*
@rm -fR dist/bin

dist/macOS-x86_64: .FORCE
@mkdir -p dist/bin
deno task dist_macos_x86_64
@cd dist && zip -r $(PROJECT)-v$(VERSION)-macOS-x86_64.zip LICENSE codemeta.json CITATION.cff *.md bin/*
@rm -fR dist/bin

dist/macOS-arm64: .FORCE
@mkdir -p dist/bin
deno task dist_macos_aarch64
@cd dist && zip -r $(PROJECT)-v$(VERSION)-macOS-arm64.zip LICENSE codemeta.json CITATION.cff *.md bin/*
@rm -fR dist/bin

dist/Windows-x86_64: .FORCE
@mkdir -p dist/bin
deno task dist_windows_x86_64
@cd dist && zip -r $(PROJECT)-v$(VERSION)-Windows-x86_64.zip LICENSE codemeta.json CITATION.cff *.md bin/*
@rm -fR dist/bin

dist/Windows-aarch64: .FORCE
@mkdir -p dist/bin
deno task dist_windows_v86_64 #deno task dist_windows_aarch64 <-- switch to native when Deno supports Windows ARM64
@cd dist && zip -r $(PROJECT)-v$(VERSION)-Windows-aarch64.zip LICENSE codemeta.json CITATION.cff *.md bin/*
@rm -fR dist/bin

.FORCE:
24 changes: 16 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,27 +1,27 @@

# Log Agent

This is a simplified log processor that looks for explicit text on a line, parses the line for an IP address and then applies the action associted with the text. It is inspired by fail2ban but written in response to it's complexity. Caltech Library needed a simple tool to do a narrow task that was oddly challanging using fail2ban.
This is a simplified log processor that looks for explicit text on a line, parses the line for an IP address and then applies the associated action. It is inspired by fail2ban but is written in response fail2ban's complexity. Caltech Library needed a simple tool to do a narrow task that was oddly challenging using fail2ban. There is always a balancing act between a tool features and those that are simpler targeting a more specific issue.

## approach
## Approach

Log agent reads input line by line. If checks if a tag (explicit sub string) is contained in that line. If a match is found then the agent extracts any IP addresses identified before applying a rule associated with the tag.
Log agent reads input line by line. If checks if a tag (explicit sub-string) is contained in that line. If a match is found then the agent extracts any IP addresses identified before applying a rule associated with the tag.

The log agent requires a configuration file written in yaml. The configuration holds an array of object where each object is an agent configuration. The object has the following attributes.
The log agent requires a configuration file written in YAML. The configuration holds an array of objects. Each object has the following attributes.

tag
: The explicit string to search for
: The explicit search string (i.e. not regular expressions)

action
: The command to execute if tag is found

Here's an example configuration YAML file.

~~~yaml
~~~YAML
- tag: BadBot
action: |
sudo iptables -A INPUT -p tcp -m multiport
--dports 80,443 -s {ipaddress} -j DROP
--dports 80,443 -s {ipaddress} -j DROP
~~~

If the text "BadBot" is found in the log line. and the IP address "156.59.198.136" was found in the log line then the following command would be executed.
Expand Down Expand Up @@ -62,4 +62,12 @@ sudo logagent badbots.yaml /var/log/nginx/access.log --dry_run
: display license

-d, --dry_run
: don't take any actions, instead write each action to standard out. This lets you cature them in a bash or Powershell script.
: don't take any actions, instead write each action to standard out. This lets you capture them in a bash or Powershell script.

For more information see the following documentation pages.

- [User Manual](user_manual.md)
- [Installation](INSTALL.md) and compiling Log Agent from source
- [LICENSE](LICENSE)
- [About Log Agent](about.md)
- [Cite with CITATION.cff](CITATION.cff)
Loading

0 comments on commit 28a5c56

Please sign in to comment.