Add module or function to delete all empty/null values

[TobiasNx]

When using fix to map a data set one has to address all fields that could have empty or null values with an additional not-equal function.

e.g.
https://gitlab.com/oersi/oersi-etl/-/blob/develop/data/production/edu-sharing.fix#L44

do map(node.description, description)
  not_equals(string: '')
end

The problem with that is that you have to know all fields that could have empty values and for each of them you have to add the not-equals function. This would expand the FIX quiet a bit with all the exceptions.

A Flux-module or a FIX-function would be good that would delete all metadata fields with emtpy values in a data set per record. So that no additional exception is necessary. Since different data sets have different values marking that it is empty as null or "" It would be good to add an attribute to that functions so one can determine what an empty value is.

[blackwinter]

For null values, there's org.metafacture.mangling.NullFilter. I'm not aware of anything for filtering literals based on their content (org.metafacture.strings.StringFilter/filter-strings works on record level). So maybe a org.metafacture.mangling.StringFilter needs to be added (with a configurable pattern). To metafacture-core, though, right?

[fsteeg]

So maybe a org.metafacture.mangling.StringFilter needs to be added (with a configurable pattern). To metafacture-core, though, right?

Yes, this sounds good. We'd have two StringFilter then though (one for the record level, one for the literal level). Maybe we can make NullFilter configurable instead? One could argue that conceptually it's always about null values, but they are represented differently, sometimes actual null, sometimes strings like "NULL" or "nil", sometimes by the empty string. Or a different name for the new class like StringLiteralFilter.

[dr0i]

+1 for enhancing NullFilter. It's highly arguable how to identify a null and how/if to treat it.

[blackwinter]

Well, that's a bit of stretch ;) It was explicitly designed for (Java) null values so downstream receivers won't choke on them.

But I won't oppose expanding NullFilter's scope to include semantically "null" values.

[blackwinter]

Will this issue be resolved by #60? It adds the vacuum() function (998da3d).

[TobiasNx]

If I remember it vaccuum() only deletes empty strings not null values. But I might be wrong.

Also in context of OERSI we have he module filter-null-values: https://gitlab.com/oersi/oersi-etl/-/blob/master/data/production/hoou-to-oersi.flux#L14

[blackwinter]

If I remember it vaccuum() only deletes empty strings not null values.

That's correct. But it seems that the new Metafix implementation ignores null values automatically, although I haven't verified it to the full extent.

If you believe that's sufficient, you can add #60 under "Linked pull requests". Otherwise, just leave this ticket open.

[fsteeg]

Closing, supported by vacuum & filter-null-values.