Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Various fixes and improvements. #417

Merged
merged 8 commits into from
Nov 8, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 15 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,63 +4,63 @@

Metafacture is a toolkit for processing semi-structured data with a focus on library metadata. It provides a versatile set of tools for reading, writing and transforming data. Metafacture can be used as a stand-alone application or as a Java library in other applications. The name Metafacture is a portmanteau of the words *meta*data and manu*facture*.

Metafacture includes a [large number of modules](https://github.com/metafacture/metafacture-documentation/blob/master/flux-commands.md) for operating on semi-structured data. These modules can be combined to build pipelines to perform complex metadata processing tasks. The pipelines can be constructed either in Java code or with the domain-specific language **Flux**. One of the core features of Metafacture is the **Metamorph** module. Metamorph is an xml-based language for specifying transformations of semi-structured data. It can be seamlessly integrated into Java code.
Metafacture includes a [large number of modules](https://github.com/metafacture/metafacture-documentation/blob/master/flux-commands.md) for operating on semi-structured data. These modules can be combined to build pipelines to perform complex metadata processing tasks. The pipelines can be constructed either in Java code or with the domain-specific language **Flux**. One of the core features of Metafacture is the **Metamorph** module. Metamorph is an XML-based language for specifying transformations of semi-structured data. It can be seamlessly integrated into Java code.

At its heart Metafacture is a framework for implementing modules for metadata processing. This makes Metafacture easily extendable with additional modules. The [plugins and tools page](https://github.com/metafacture/metafacture-core/wiki/Plugins-and-Tools) on the wiki shows supplementary packages and projects which extend Metafacture.

Originally, Metafacture was developed as part of the [Culturegraph](http://culturegraph.org) platform but it is developed independently now and used by others, too: [see who uses Metafacture](https://github.com/metafacture/metafacture-core/wiki/Who-uses-Metafacture).
Originally, Metafacture was developed as part of the [Culturegraph](http://www.culturegraph.org) platform but it is developed independently now and used by others, too: [see who uses Metafacture](https://github.com/metafacture/metafacture-core/wiki/Who-uses-Metafacture).

# Getting started

You can either use Metafacture as a stand-alone application or include it as a Java library in your own projects.

## Metafacture as a stand-alone application

If you are only interested in running Flux scripts without doing any Java programming this is the way to go. The instructions assume that you are using a *nix-like shell. [See more information in the wiki page about Flux](https://github.com/metafacture/metafacture-core/wiki/Flux-user-guide).
If you are only interested in running Flux scripts without doing any Java programming this is the way to go. The instructions assume that you are using a \*nix-like shell. [See more information in the wiki page about Flux](https://github.com/metafacture/metafacture-core/wiki/Flux-user-guide).

1. Download the latest distribution package from the [metafacture-core/releases](https://github.com/metafacture/metafacture-core/releases) page. Make sure that you do download a distribution package and _not_ a source code package (the file name should include *-dist*).
1. Download the latest distribution package from the [release page](https://github.com/metafacture/metafacture-core/releases). Make sure that you do download a distribution package and *not* a source code package (the file name should include `*-dist*`).

2. Extract the downloaded archive:
```bash
$ tar xzf metafacture-core-VERSION-dist.tar.gz
$ tar xzf metafacture-core-$VERSION-dist.tar.gz
```
This will create a new directory containing a ready-to-use metafacture distribution.
This will create a new directory containing a ready-to-use Metafacture distribution.
3. Change into the newly created directory:
```bash
$ cd metafacture-core-VERSION
$ cd metafacture-core-$VERSION
```
4. Run one of the example scripts:
```bash
$ ./flux.sh examples/read/marc21/read-marc21.flux
```
This example will print a number of marc21 records on standard out.
This example will print a number of MARC 21 records on standard output.

The _examples_ folder contains many more examples which provide a good starting point for learning metafacture. If you have any questions please join our [mailing list](http://lists.dnb.de/mailman/listinfo/metafacture) or use our issue-based discussion forum over at [metafacture-documentation](https://github.com/metafacture/metafacture-documentation).
The `examples` folder contains many more examples which provide a good starting point for learning Metafacture. If you have any questions please join our [mailing list](http://lists.dnb.de/mailman/listinfo/metafacture) or use our issue-based discussion forum over at [metafacture-documentation](https://github.com/metafacture/metafacture-documentation).


## Using Metafacture as a Java libary
## Using Metafacture as a Java library

If you want use Metafacture in your own Java projects all you need to add some dependencies to your project. As of Metafacture 5 the single metafacture-core package has been replaced with a number of domain-specific packages. You can find the list of packages on [Maven Central](https://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.metafacture%22).
If you want to use Metafacture in your own Java projects all you need is to add some dependencies to your project. As of Metafacture 5, the single metafacture-core package has been replaced with a number of domain-specific packages. You can find the list of packages on [Maven Central](https://search.maven.org/search?q=g:org.metafacture).

Alternatively, you can simply guess the package names from the top-level folders in the source code repository -- they are the same. For instance, if you want to use Metamorph in your project, simply add the following dependency to your `pom.xml`:

```xml
<dependency>
<groupId>org.metafacture</groupId>
<artifactId>metamorph</artifactId>
<version>VERSION</version>
<version>$VERSION</version>
</dependency>
```

or if Gradle is your build tool of choice use:

```groovy
dependencies {
implementation 'org.metafacture:metamorph:VERSION'
implementation 'org.metafacture:metamorph:$VERSION'
}
```

Our integration server automatically publishes successful builds of all branches as snapshot versions on [Sonatype OSS Repository](https://oss.sonatype.org/index.html#nexus-search;quick~metafacture). The version number is derived from the branch name. Snapshot builds from the master branch always have the version "master-SNAPSHOT".
Occasionally, we publish snapshot builds on [Sonatype OSS Repository](https://oss.sonatype.org/index.html#nexus-search;gav~org.metafacture~~~~). The version number is derived from the branch name. Snapshot builds from the master branch always have the version `master-SNAPSHOT`.

<!--
TODO: Link to getting started tutorial
Expand All @@ -77,7 +77,7 @@ $ git clone https://github.com/metafacture/metafacture-core.git
$ cd metafacture-core
```

2. Invoke the gradle-wrapper to download Gradle and build metafacture-core (on Windows call `gradlew.bat install`):
2. Invoke the Gradle wrapper to download Gradle and build metafacture-core (on Windows call `gradlew.bat install` instead):

```bash
$ ./gradlew install
Expand Down
15 changes: 15 additions & 0 deletions build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -64,8 +64,14 @@ subprojects {
targetCompatibility = 1.8

tasks.withType(JavaCompile) {
'all -processing -rawtypes -serial'.split().each {
options.compilerArgs << "-Xlint:${it}"
}

options.compilerArgs << '-Werror'
options.encoding = 'UTF-8'
}

task sourceJar(type: Jar) {
dependsOn tasks.classes
from sourceSets.main.allJava
Expand Down Expand Up @@ -98,6 +104,15 @@ subprojects {
checkstyleTest.enabled = false
}

test {
systemProperties['org.slf4j.simpleLogger.defaultLogLevel'] = 'warn'

testLogging {
showStandardStreams = true
exceptionFormat = 'full'
}
}

signing {
required {
scmInfo.isRelease() && gradle.taskGraph.hasTask(tasks.uploadArchives)
Expand Down
1 change: 1 addition & 0 deletions config/checkstyle/checkstyle.xml
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
<module name="CustomImportOrder">
<property name="customImportOrderRules" value="SAME_PACKAGE(2)###THIRD_PARTY_PACKAGE###STANDARD_JAVA_PACKAGE###SPECIAL_IMPORTS###STATIC"/>
<property name="separateLineBetweenGroups" value="true"/>
<property name="sortImportsInGroupAlphabetically" value="true"/>
</module>
<module name="CyclomaticComplexity"/>
<module name="DeclarationOrder"/>
Expand Down
7 changes: 0 additions & 7 deletions metafacture-biblio/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -31,10 +31,3 @@ dependencies {
testImplementation 'junit:junit:4.12'
testImplementation 'org.mockito:mockito-core:2.5.5'
}

test {
testLogging {
showStandardStreams = true
exceptionFormat = 'full'
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,7 @@ else if (RECORD.equals(localName) && NAMESPACE.equals(uri)) {
}

@Override
@SuppressWarnings("fallthrough")
public void endElement(final String uri, final String localName,
final String qName) throws SAXException {
if (SUBFIELD.equals(localName)) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
package org.metafacture.biblio.marc21;

import org.metafacture.biblio.iso2709.RecordBuilder;
import org.metafacture.biblio.iso2709.RecordFormat;
import org.metafacture.framework.FluxCommand;
import org.metafacture.framework.FormatException;
import org.metafacture.framework.ObjectReceiver;
Expand Down Expand Up @@ -71,7 +72,7 @@
public final class Marc21Encoder extends
DefaultStreamPipe<ObjectReceiver<String>> {

private static final int NAME_LENGTH = Marc21Constants.MARC21_FORMAT.TAG_LENGTH +
private static final int NAME_LENGTH = RecordFormat.TAG_LENGTH +
Marc21Constants.MARC21_FORMAT.getIndicatorLength();

private final RecordBuilder builder;
Expand Down Expand Up @@ -157,7 +158,7 @@ private void startField(final String name) {
if (name.length() != NAME_LENGTH) {
throw new FormatException("invalid entity name: " + name);
}
final char[] tag = new char[Marc21Constants.MARC21_FORMAT.TAG_LENGTH];
final char[] tag = new char[RecordFormat.TAG_LENGTH];
final char[] indicators = new char[Marc21Constants.MARC21_FORMAT.getIndicatorLength()];
name.getChars(0, tag.length, tag, 0);
name.getChars(tag.length, name.length(), indicators, 0);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,8 @@
import org.metafacture.framework.annotations.Out;
import org.metafacture.framework.helpers.DefaultStreamPipe;

import java.text.Normalizer.Form;
import java.text.Normalizer;
import java.text.Normalizer.Form;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,8 @@

import org.metafacture.framework.StreamReceiver;

import java.text.Normalizer.Form;
import java.text.Normalizer;
import java.text.Normalizer.Form;

/**
* Parser context for the PICA+ parser.The context implements
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -188,7 +188,7 @@ private void applySetters(final T target, final Map<String, String> setterValues
}
}

private Object convertValue(final String value, final Class<?> type) {
private <T extends Enum<T>> Object convertValue(final String value, final Class<?> type) {
final Object result;

if (type == boolean.class) {
Expand All @@ -199,7 +199,7 @@ else if (type == int.class) {
}
else if (type.isEnum()) {
@SuppressWarnings("unchecked") // protected by type.isEnum() check
final Class<Enum> enumType = (Class<Enum>) type;
final Class<T> enumType = (Class<T>) type;
result = Enum.valueOf(enumType, value.toUpperCase());
}
else {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,8 @@
import java.util.Collections;
import java.util.Comparator;
import java.util.List;
import java.util.Map.Entry;
import java.util.Map;
import java.util.Map.Entry;

/**
* Replaces Strings by other Strings.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ public final class ObjectExceptionCatcher<T> extends

private String logPrefix;
private boolean logStackTrace;
private boolean logExceptionMessage = true;

/**
* Creates an instance of {@link ObjectExceptionCatcher} without a log message
Expand Down Expand Up @@ -108,13 +109,19 @@ public boolean isLogStackTrace() {
return logStackTrace;
}

/*package-private*/ void setLogExceptionMessage(final boolean logExceptionMessage) {
this.logExceptionMessage = logExceptionMessage;
}

@Override
public void process(final T obj) {
try {
getReceiver().process(obj);
}
catch (final Exception e) { // checkstyle-disable-line IllegalCatch
LOG.error("{}'{}' while processing object: {}", logPrefix, e.getMessage(), obj);
if (logExceptionMessage) {
LOG.error("{}'{}' while processing object: {}", logPrefix, e.getMessage(), obj);
}

if (logStackTrace) {
final StringWriter stackTraceWriter = new StringWriter();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,13 @@

package org.metafacture.flowcontrol;

import org.junit.Rule;
import org.junit.Test;
import org.metafacture.framework.ObjectReceiver;
import org.mockito.Mock;
import org.mockito.Mockito;
import org.mockito.junit.MockitoJUnit;
import org.mockito.junit.MockitoRule;

/**
* Tests for class {@link CloseSuppressor}.
Expand All @@ -28,10 +32,15 @@
*/
public final class CloseSuppressorTest {

@Rule
public final MockitoRule mockitoRule = MockitoJUnit.rule();

@Mock
private ObjectReceiver<Object> receiver;

@Test
public void testSuppression() {
final CloseSuppressor<Object> supressor = new CloseSuppressor<>(3);
final ObjectReceiver<Object> receiver = Mockito.mock(ObjectReceiver.class);
supressor.setReceiver(receiver);
supressor.closeStream();
supressor.closeStream();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ public void setup() {
.when(exceptionThrowingModule).process(anyString());
exceptionCatcher = new ObjectExceptionCatcher<>();
exceptionCatcher.setReceiver(exceptionThrowingModule);
exceptionCatcher.setLogExceptionMessage(false);
}

@Test
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,8 @@
import java.util.Collection;
import java.util.Collections;
import java.util.List;
import java.util.Map.Entry;
import java.util.Map;
import java.util.Map.Entry;

/**
* Prints Flux help for a given {@link ObjectFactory}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ public FormetaRecordsReader() {
}

@Override // checkstyle-disable-line CyclomaticComplexity
@SuppressWarnings("fallthrough")
public void process(final Reader reader) {
assert !isClosed();

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -254,6 +254,7 @@ public void shouldEncodeEntityStreamToPojoWithSimplePojoMap() {
.stringField2);
}

@SuppressWarnings("unchecked")
private <T> ObjectReceiver<T> createObjectReceiverMock() {
// There is no type safe to create a mock with Mockito#mock(Class).
// Hence, we have to use an unchecked cast here:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,8 @@

import java.io.IOException;
import java.util.HashMap;
import java.util.Map.Entry;
import java.util.Map;
import java.util.Map.Entry;
import java.util.Properties;
import java.util.regex.Pattern;

Expand Down
1 change: 1 addition & 0 deletions metafacture-json/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -24,4 +24,5 @@ dependencies {
implementation 'com.jayway.jsonpath:json-path:2.6.0'
testImplementation 'junit:junit:4.12'
testImplementation 'org.mockito:mockito-core:2.5.5'
testRuntimeOnly 'org.slf4j:slf4j-simple:1.7.21'
}
7 changes: 0 additions & 7 deletions metafacture-mangling/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,3 @@ dependencies {
testImplementation 'junit:junit:4.12'
testImplementation 'org.mockito:mockito-core:2.5.5'
}

test {
testLogging {
showStandardStreams = true
exceptionFormat = 'full'
}
}
14 changes: 7 additions & 7 deletions metafacture-runner/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -30,9 +30,9 @@ configurations {
}

dependencies {
compile project(':metafacture-framework')
compile project(':metafacture-commons')
compile project(':metafacture-flux')
implementation project(':metafacture-framework')
implementation project(':metafacture-commons')
implementation project(':metafacture-flux')

plugins project(':metafacture-biblio')
plugins project(':metafacture-csv')
Expand Down Expand Up @@ -67,7 +67,7 @@ dependencies {
// class loader which was used to load the classes of the slf4j-api. Until
// a solution is found for this problem, the binding need to be placed on the
// class path:
runtime 'org.slf4j:slf4j-log4j12:1.7.21'
runtimeOnly 'org.slf4j:slf4j-log4j12:1.7.21'

// The following dependencies are placed in the "provided" scope to prevent
// them from being included in the class path but still have them available
Expand All @@ -88,7 +88,7 @@ jar {
manifest {
attributes(
'Implementation-Title': project.name,
'Implementation-Version': version,
'Implementation-Version': archiveVersion,
'Main-Class': 'org.metafacture.runner.Flux',
'Class-Path':
configurations.runtimeClasspath.files.collect { it.name }.join(' '))
Expand All @@ -104,7 +104,7 @@ jar {

distributions {
main {
baseName = parent.project.name
distributionBaseName = parent.project.name
contents {
dirMode = 0755
fileMode = 0644
Expand Down Expand Up @@ -141,7 +141,7 @@ def metamorph() {
distTar {
compression = Compression.GZIP
classifier = 'dist'
extension = 'tar.gz'
archiveExtension = 'tar.gz'
}

distZip {
Expand Down
7 changes: 0 additions & 7 deletions metafacture-triples/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -24,10 +24,3 @@ dependencies {
testImplementation 'junit:junit:4.12'
testImplementation 'org.mockito:mockito-core:2.5.5'
}

test {
testLogging {
showStandardStreams = true
exceptionFormat = 'full'
}
}
Loading