Skip to content
This repository has been archived by the owner on Jan 27, 2025. It is now read-only.

Consolidate path traversal #170

Merged
merged 63 commits into from
Apr 8, 2022
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
425dc49
Unify ReservedField usage, align structure of insert methods (#102)
fsteeg Feb 18, 2022
1a3278a
Replace existing values in set_array and set_hash (#102, #111)
fsteeg Feb 22, 2022
8c42874
Remove appendTo method, hide FixPath behind record (#102, #92, #89)
fsteeg Feb 22, 2022
c25ede3
Implement transformIn using findIn & insertInto (#102, #121, #147)
fsteeg Feb 28, 2022
692b4f9
Move transform methods to Record, avoid FixPath in FixMethod (#102)
fsteeg Feb 28, 2022
bc59bf4
Merge remote-tracking branch 'origin/master' into 102-path
fsteeg Mar 4, 2022
5ec283d
Exclude .project and metafix.err files in editorconfigChecker
fsteeg Mar 4, 2022
61e3748
Update passing integration tests (#102, #92, #121, #149, #155)
fsteeg Mar 4, 2022
34c96f1
Don't use `path` in equals, hashCode and toString methods (#102)
fsteeg Mar 4, 2022
e9dd4ab
Tweak `lookup` behavior, partial `delete` support (#102, #149)
fsteeg Mar 4, 2022
1bcc5ca
Integrate feedback from #170 (see #102)
fsteeg Mar 4, 2022
be764e4
Throw exception on transform of non-string field (#102, #100, #133)
fsteeg Mar 7, 2022
8b27a52
Integrate feedback from #170 (see #102)
fsteeg Mar 7, 2022
93aaf2e
Some cleanup as suggested in #170 (see #102)
fsteeg Mar 8, 2022
6f455eb
Fix test to actually use internal map (#170, #102)
fsteeg Mar 10, 2022
b376291
Remove unused `MetafixToDo.Extension` (#170, #102)
fsteeg Mar 10, 2022
c479c26
Merge remote-tracking branch 'origin/master' into 102-path
fsteeg Mar 10, 2022
50c45f7
Enable passing integration tests (#102, #113, #130, #156)
fsteeg Mar 10, 2022
238e840
Handle null results directly in transform (#102, #149, #170)
fsteeg Mar 10, 2022
63bf98a
Add failing test case for asterisk in `replace_all()`. (#170)
blackwinter Mar 10, 2022
006e3ac
Remove unnecessary parens, add unit test (#102, #149, #170)
fsteeg Mar 10, 2022
44dc0be
Merge remote-tracking branch 'origin/102-path' into 102-path
fsteeg Mar 10, 2022
5494f42
Add test cases from review (#102, #170)
fsteeg Mar 21, 2022
1e8ac6e
Remove untested $append, $first, $last support on hashes (#102)
fsteeg Mar 18, 2022
31e844b
Merge remote-tracking branch 'origin/master' into 102-path
fsteeg Mar 22, 2022
158c4cf
Add unit tests for failing integration test, mark as TODO (#170)
fsteeg Mar 22, 2022
e027e1c
Update passing integration tests (#102, #149, #170)
fsteeg Mar 22, 2022
b189d26
Merge remote-tracking branch 'origin/master' into 102-path
fsteeg Mar 24, 2022
96f65f4
Fix typo in integration test folder name (#170, #205)
fsteeg Mar 24, 2022
cbcde34
Add unit tests for failing integration test, mark as TODO (#205)
fsteeg Mar 24, 2022
64d8d98
Only create new path if old value's path doesn't match (#102, #170)
fsteeg Mar 25, 2022
8cfe1a8
Add unit test for record transformation. (#102, #170)
blackwinter Mar 25, 2022
e2cf74c
Fix nested removal from array (#102, #113, #170)
fsteeg Mar 25, 2022
2378455
Add unit test for record transformation. (#102, #170)
blackwinter Mar 25, 2022
16ac0a0
Merge remote-tracking branch 'origin/master' into 102-path
fsteeg Mar 28, 2022
2063b07
Update passing integration test, mark failing as TODO (#206)
fsteeg Mar 28, 2022
f210dd3
Preserve order in Record#transform (#102, #170)
fsteeg Mar 29, 2022
a4d557e
Explicitly set path in record test (#102, #170)
fsteeg Mar 29, 2022
a816a73
Add failing test case for unsuccessful lookup. (f210dd3, #170)
blackwinter Mar 30, 2022
ec9f8ef
Delete reversed null results after ordered transform (#102, #170)
fsteeg Mar 31, 2022
725097a
Add failing test case for asterisk in `replace_all()` with repeated f…
blackwinter Mar 31, 2022
2bd779e
Merge remote-tracking branch 'origin/master' into 102-path
fsteeg Mar 31, 2022
40ab054
Mark failing strictness tests as TODOs (#102, #170)
fsteeg Mar 31, 2022
d8cc4f8
Extract try-catch to avoid Eclipse compiler bug (#210)
fsteeg Mar 31, 2022
f4c4faf
Merge remote-tracking branch 'origin/102-path' into 102-path
fsteeg Mar 31, 2022
4324378
Update path for repeated fields to indexed array (#102, #170)
fsteeg Apr 1, 2022
cb4f27e
Add failing integration test for asterisk in `replace_all()` with rep…
blackwinter Apr 1, 2022
1d495ab
Add unit tests for new integration test from cb4f27e (#170)
fsteeg Apr 1, 2022
ca1cba1
Update path when adding, rename on copy/move field (#102, #170)
fsteeg Apr 1, 2022
82bee32
Add failing test case for multiple `replace_all()` on same subfield. …
blackwinter Apr 1, 2022
049c709
Set correct path after inserting transformed value (#102, #170)
fsteeg Apr 4, 2022
7b74937
Add failing test case for `prepend()` on array with wildcard. (#170)
blackwinter Apr 5, 2022
4961b95
Tweak test to produce correct order (#170)
fsteeg Apr 5, 2022
31cf8e9
Replace matchType.ifString.orElseThrow with asString (#102, #170)
fsteeg Apr 5, 2022
eafca9e
Revert "Tweak test to produce correct order (#170)"
blackwinter Apr 5, 2022
0dfcfb8
Update strictness tests for changed behaviour. (#170, 4961b95)
blackwinter Apr 5, 2022
2b476e2
Replace path heuristic with actual value check in lookup (#170)
fsteeg Apr 7, 2022
5e5bf5f
Update strictness integration tests to correspond to unit tests
fsteeg Apr 8, 2022
540f43f
Update TODOs on implicit append in unit & integration tests (#170)
fsteeg Apr 8, 2022
6779544
Improve `*` wildcard behavior, additional tests (#102, #170)
fsteeg Apr 8, 2022
746fcf4
Wrap NumberFormatException into IllegalStateException (#170)
fsteeg Apr 8, 2022
9b8fa39
Tweak shouldNotTrimIndexedArray, add shouldNotTrimHash test (#170)
fsteeg Apr 8, 2022
5a3fa9b
Handle NumberFormatException in `apply` (#170)
fsteeg Apr 8, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion metafix/src/main/java/org/metafacture/metafix/FixBind.java
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ public enum FixBind implements FixContext {
public void execute(final Metafix metafix, final Record record, final List<String> params, final Map<String, String> options, final List<Expression> expressions) {
final RecordTransformer recordTransformer = metafix.getRecordTransformer();
final String scopeVariable = options.get("var");
Value.asList(new FixPath(options.get("path")).findIn(record), a -> {
Value.asList(record.get(options.get("path")), a -> {
for (int i = 0; i < a.size(); ++i) {
final Value value = a.get(i);

Expand Down
91 changes: 40 additions & 51 deletions metafix/src/main/java/org/metafacture/metafix/FixMethod.java
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ public void apply(final Metafix metafix, final Record record, final List<String>
add_field {
@Override
public void apply(final Metafix metafix, final Record record, final List<String> params, final Map<String, String> options) {
new FixPath(params.get(0)).appendIn(record, params.get(1));
record.add(params.get(0), new Value(params.get(1)));
}
},
array { // array-from-hash
Expand All @@ -120,7 +120,9 @@ public void apply(final Metafix metafix, final Record record, final List<String>
copy_field {
@Override
public void apply(final Metafix metafix, final Record record, final List<String> params, final Map<String, String> options) {
record.copy(params);
final String oldName = params.get(0);
final String newName = params.get(1);
Value.asList(record.get(oldName), a -> a.forEach(v -> record.add(newName, v)));
}
},
format {
Expand All @@ -129,7 +131,7 @@ public void apply(final Metafix metafix, final Record record, final List<String>
final String field = params.get(0);

record.getList(field, oldValues -> {
final String newValue = String.format(params.get(1), oldValues.stream().toArray());
final String newValue = String.format(params.get(1), oldValues.stream().map(v -> v.asString()).toArray());
blackwinter marked this conversation as resolved.
Show resolved Hide resolved
record.replace(field, new Value(Arrays.asList(new Value(newValue))));
});
}
Expand All @@ -149,8 +151,12 @@ public void apply(final Metafix metafix, final Record record, final List<String>
move_field {
@Override
public void apply(final Metafix metafix, final Record record, final List<String> params, final Map<String, String> options) {
record.copy(params);
new FixPath(params.get(0)).removeNestedFrom(record);
final String oldName = params.get(0);
final String newName = params.get(1);
Value.asList(record.get(oldName), a -> a.forEach(v -> {
record.add(newName, v);
record.remove(oldName);
}));
}
},
parse_text {
Expand Down Expand Up @@ -195,10 +201,11 @@ public void apply(final Metafix metafix, final Record record, final List<String>
@Override
public void apply(final Metafix metafix, final Record record, final List<String> params, final Map<String, String> options) {
final String joinChar = options.get("join_char");
new FixPath(params.get(0)).replaceIn(record, params.subList(1, params.size()).stream()
.filter(f -> literalString(f) || new FixPath(f).findIn(record) != null)
.map(f -> literalString(f) ? new Value(f.substring(1)) : Value.asList(new FixPath(f).findIn(record), null).asArray().get(0))
final Value newValue = new Value(params.subList(1, params.size()).stream()
.filter(f -> literalString(f) || record.get(f) != null)
.map(f -> literalString(f) ? new Value(f.substring(1)) : Value.asList(record.get(f), null).asArray().get(0))
.map(Value::asString).collect(Collectors.joining(joinChar != null ? joinChar : " ")));
record.set(params.get(0), newValue);
}

private boolean literalString(final String s) {
Expand All @@ -210,8 +217,7 @@ private boolean literalString(final String s) {
public void apply(final Metafix metafix, final Record record, final List<String> params, final Map<String, String> options) {
final String field = params.get(0);
final int max = getInteger(params, 1);

new FixPath(field).replaceIn(record, String.valueOf(RANDOM.nextInt(max)));
record.set(field, new Value(String.valueOf(RANDOM.nextInt(max))));
}
},
reject {
Expand All @@ -223,7 +229,7 @@ public void apply(final Metafix metafix, final Record record, final List<String>
remove_field {
@Override
public void apply(final Metafix metafix, final Record record, final List<String> params, final Map<String, String> options) {
params.forEach(p -> new FixPath(p).removeNestedFrom(record));
params.forEach(p -> record.remove(p));
}
},
rename {
Expand All @@ -234,7 +240,7 @@ public void apply(final Metafix metafix, final Record record, final List<String>

final UnaryOperator<String> operator = s -> s.replaceAll(search, replace);

new FixPath(params.get(0)).transformIn(record, (m, c) -> m
record.transform(params.get(0), (m, c) -> m
.ifArray(a -> c.accept(renameArray(a, operator)))
.ifHash(h -> c.accept(renameHash(h, operator)))
.orElseThrow()
Expand Down Expand Up @@ -267,35 +273,22 @@ public void apply(final Metafix metafix, final Record record, final List<String>
@Override
public void apply(final Metafix metafix, final Record record, final List<String> params, final Map<String, String> options) {
final String field = params.get(0);
final List<String> toAdd = params.subList(1, params.size());
if (field.endsWith(DOT_APPEND)) {
record.addAll(field.replace(DOT_APPEND, EMPTY), toAdd);
}
else {
record.put(field, newArray(toAdd.stream().map(Value::new)));
}
final Value newValue = newArray(params.subList(1, params.size()).stream().map(Value::new));
record.set(field, newValue);
}
},
set_field {
@Override
public void apply(final Metafix metafix, final Record record, final List<String> params, final Map<String, String> options) {
new FixPath(params.get(0)).replaceIn(record, params.get(1));
record.set(params.get(0), new Value(params.get(1)));
}
},
set_hash {
@Override
public void apply(final Metafix metafix, final Record record, final List<String> params, final Map<String, String> options) {
final String field = params.get(0);

final Value value = record.get(field.replace(DOT_APPEND, EMPTY));
final Value newValue = Value.newHash(h -> options.forEach((f, v) -> h.put(f, new Value(v))));

if (field.endsWith(DOT_APPEND) && value.isArray()) {
value.asArray().add(newValue);
}
else {
record.put(field, newValue);
}
record.set(field, newValue);
}
},
vacuum {
Expand All @@ -313,19 +306,19 @@ public void apply(final Metafix metafix, final Record record, final List<String>
@Override
public void apply(final Metafix metafix, final Record record, final List<String> params, final Map<String, String> options) {
final String value = params.get(1);
new FixPath(params.get(0)).transformIn(record, s -> s + value);
record.transform(params.get(0), s -> s + value);
}
},
capitalize {
@Override
public void apply(final Metafix metafix, final Record record, final List<String> params, final Map<String, String> options) {
new FixPath(params.get(0)).transformIn(record, s -> s.substring(0, 1).toUpperCase() + s.substring(1));
record.transform(params.get(0), s -> s.substring(0, 1).toUpperCase() + s.substring(1));
}
},
count {
@Override
public void apply(final Metafix metafix, final Record record, final List<String> params, final Map<String, String> options) {
new FixPath(params.get(0)).transformIn(record, (m, c) -> m
record.transform(params.get(0), (m, c) -> m
.ifArray(a -> c.accept(new Value(a.size())))
.ifHash(h -> c.accept(new Value(h.size())))
);
Expand All @@ -334,7 +327,7 @@ public void apply(final Metafix metafix, final Record record, final List<String>
downcase {
@Override
public void apply(final Metafix metafix, final Record record, final List<String> params, final Map<String, String> options) {
new FixPath(params.get(0)).transformIn(record, s -> s.toLowerCase());
record.transform(params.get(0), s -> s.toLowerCase());
}
},
filter {
Expand All @@ -345,7 +338,7 @@ public void apply(final Metafix metafix, final Record record, final List<String>

final Predicate<Value> predicate = s -> search.matcher(s.asString()).find();

new FixPath(params.get(0)).transformIn(record, (m, c) -> m
record.transform(params.get(0), (m, c) -> m
.ifArray(a -> c.accept(newArray(a.stream().filter(invert ? predicate.negate() : predicate))))
);
}
Expand All @@ -354,14 +347,14 @@ public void apply(final Metafix metafix, final Record record, final List<String>
@Override
public void apply(final Metafix metafix, final Record record, final List<String> params, final Map<String, String> options) {
final String search = params.get(1);
new FixPath(params.get(0)).transformIn(record, s -> String.valueOf(s.indexOf(search))); // TODO: multiple
record.transform(params.get(0), s -> String.valueOf(s.indexOf(search))); // TODO: multiple
}
},
join_field {
@Override
public void apply(final Metafix metafix, final Record record, final List<String> params, final Map<String, String> options) {
final String joinChar = params.size() > 1 ? params.get(1) : "";
new FixPath(params.get(0)).transformIn(record, (m, c) -> m
record.transform(params.get(0), (m, c) -> m
.ifArray(a -> c.accept(new Value(a.stream().map(Value::asString).collect(Collectors.joining(joinChar)))))
);
}
Expand Down Expand Up @@ -390,14 +383,14 @@ public void apply(final Metafix metafix, final Record record, final List<String>
}

final String defaultValue = map.get(Maps.DEFAULT_MAP_KEY); // TODO: Catmandu uses 'default'
new FixPath(params.get(0)).transformIn(record, k -> map.getOrDefault(k, defaultValue));
record.transform(params.get(0), k -> map.getOrDefault(k, defaultValue));
}
},
prepend {
@Override
public void apply(final Metafix metafix, final Record record, final List<String> params, final Map<String, String> options) {
final String value = params.get(1);
new FixPath(params.get(0)).transformIn(record, s -> value + s);
record.transform(params.get(0), s -> value + s);
}
},
replace_all {
Expand All @@ -406,13 +399,13 @@ public void apply(final Metafix metafix, final Record record, final List<String>
final String search = params.get(1);
final String replace = params.get(2);

new FixPath(params.get(0)).transformIn(record, s -> s.replaceAll(search, replace));
record.transform(params.get(0), s -> s.replaceAll(search, replace));
}
},
reverse {
@Override
public void apply(final Metafix metafix, final Record record, final List<String> params, final Map<String, String> options) {
new FixPath(params.get(0)).transformIn(record, (m, c) -> m
record.transform(params.get(0), (m, c) -> m
.ifArray(a -> {
final List<Value> list = a.stream().collect(Collectors.toList());
Collections.reverse(list);
Expand All @@ -433,7 +426,7 @@ public void apply(final Metafix metafix, final Record record, final List<String>
final Comparator<Value> comparator = numeric ?
Comparator.comparing(function.andThen(Integer::parseInt)) : Comparator.comparing(function);

new FixPath(params.get(0)).transformIn(record, (m, c) -> m
record.transform(params.get(0), (m, c) -> m
.ifArray(a -> c.accept(new Value((uniq ? unique(a.stream()) : a.stream())
.sorted(reverse ? comparator.reversed() : comparator).collect(Collectors.toList()))))
);
Expand All @@ -448,7 +441,7 @@ public void apply(final Metafix metafix, final Record record, final List<String>
final Function<String, Value> splitFunction = s ->
newArray(Arrays.stream(splitPattern.split(s)).map(Value::new));

new FixPath(params.get(0)).transformIn(record, (m, c) -> m
record.transform(params.get(0), (m, c) -> m
.ifArray(a -> c.accept(newArray(a.stream().map(Value::asString).map(splitFunction))))
.ifHash(h -> c.accept(Value.newHash(n -> h.forEach((f, w) -> n.put(f, splitFunction.apply(w.asString()))))))
.ifString(s -> c.accept(splitFunction.apply(s)))
Expand All @@ -462,7 +455,7 @@ public void apply(final Metafix metafix, final Record record, final List<String>
final Integer end = params.size() > 2 ? offset + getInteger(params, 2) : null;
// TODO: final String replacement = params.size() > 3 ? params.get(3) : null;

new FixPath(params.get(0)).transformIn(record, s -> {
record.transform(params.get(0), s -> {
final int length = s.length();
return offset > length ? s : end == null || end > length ? s.substring(offset) : s.substring(offset, end);
});
Expand All @@ -471,42 +464,38 @@ public void apply(final Metafix metafix, final Record record, final List<String>
sum {
@Override
public void apply(final Metafix metafix, final Record record, final List<String> params, final Map<String, String> options) {
new FixPath(params.get(0)).transformIn(record, (m, c) -> m
record.transform(params.get(0), (m, c) -> m
.ifArray(a -> c.accept(new Value(a.stream().map(Value::asString).mapToInt(Integer::parseInt).sum())))
);
}
},
trim {
@Override
public void apply(final Metafix metafix, final Record record, final List<String> params, final Map<String, String> options) {
new FixPath(params.get(0)).transformIn(record, String::trim);
record.transform(params.get(0), String::trim);
}
},
uniq {
@Override
public void apply(final Metafix metafix, final Record record, final List<String> params, final Map<String, String> options) {
new FixPath(params.get(0)).transformIn(record, (m, c) -> m
record.transform(params.get(0), (m, c) -> m
.ifArray(a -> c.accept(newArray(unique(a.stream()))))
);
}
},
upcase {
@Override
public void apply(final Metafix metafix, final Record record, final List<String> params, final Map<String, String> options) {
new FixPath(params.get(0)).transformIn(record, s -> s.toUpperCase());
record.transform(params.get(0), s -> s.toUpperCase());
}
};

private static final Pattern NAMED_GROUP_PATTERN = Pattern.compile("\\(\\?<(.+?)>");

private static final String EMPTY = "";
private static final String DOT_APPEND = "." + Value.ReservedField.$append.name();

private static final String FILEMAP_SEPARATOR_OPTION = "sep_char";
private static final String FILEMAP_DEFAULT_SEPARATOR = ",";

private static final Random RANDOM = new Random();

private static final Map<String, Fix> INCLUDE_FIX = new HashMap<>();

blackwinter marked this conversation as resolved.
Show resolved Hide resolved
}
Loading