Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

1058-enrichWithCulturegraphRvkWithFix #1921

Merged
merged 19 commits into from
Jun 4, 2024
+2,324 −2
Merged

1058-enrichWithCulturegraphRvkWithFix #1921

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
ca6c3ef
Add extraction and transformation of culturegraph RVK
dr0i Apr 21, 2020
ed0344f
Add filter to omit empty records
dr0i Apr 22, 2020
f535aa3
Add rule to make id AND rvk mandatory
dr0i Apr 23, 2020
95d87a6
Add junit test
dr0i Apr 24, 2020
573cf05
Test runner
dr0i Apr 27, 2020
d9235a3
WIP
dr0i Apr 30, 2020
3454ae9
Use fix instead of morph for culturegraph #1058
TobiasNx Oct 12, 2023
bdbdefa
Update logger dependencies
TobiasNx Oct 12, 2023
87f63af
Catch FileNotFoundException #1058
TobiasNx Oct 16, 2023
3fdcc5c
Catch Exception (#1058)
dr0i May 28, 2024
8fb1fe3
Remove metafacture-triples; add metafacture-elasticsearch (#1058)
dr0i May 28, 2024
26a55a7
Update test data (#1058)
dr0i May 28, 2024
554949b
Format
dr0i May 28, 2024
848c948
Reduce complexity of flow (#1058)
dr0i May 28, 2024
f6c150f
Generate also CSV from CultureGraph as hbz-RVK concordance (#1085)
dr0i May 28, 2024
eae4a69
Enforce two columns' CSV; set ID to first column (#1058)
dr0i May 31, 2024
ee6ad10
Exclude tmp directory for editorconfig
dr0i May 31, 2024
9c4a21a
Create record for each ID; separate CSV and JSON (#1058)
dr0i Jun 3, 2024
e297026
Update src/main/resources/rvk/cg-to-rvk-json.fix
dr0i Jun 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion src/main/java/org/lobid/resources/run/CulturegraphXmlFilterHbzRvkToCsv.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@

import org.metafacture.biblio.marc21.MarcXmlHandler;
import org.metafacture.csv.CsvEncoder;
import org.metafacture.json.JsonDecoder;
import org.metafacture.json.JsonEncoder;
import org.metafacture.io.FileOpener;
import org.metafacture.io.ObjectWriter;
import org.metafacture.xml.XmlDecoder;
Expand All @@ -28,9 +30,13 @@ public static void main(String... args) {
if (args.length > 1) OUTPUT_FILE = args[1];

final FileOpener opener = new FileOpener();
JsonDecoder jsonDecoder = new JsonDecoder();
jsonDecoder.setRecordPath("records");
try {
opener.setReceiver(new XmlDecoder()).setReceiver(new MarcXmlHandler())
.setReceiver(new Metafix("src/main/resources/fix-cg-to-es.fix"))
.setReceiver(new Metafix("src/main/resources/rvk/cg-to-rvk-csv.fix"))
.setReceiver(new JsonEncoder())
.setReceiver(jsonDecoder)
.setReceiver(new CsvEncoder())
.setReceiver(new ObjectWriter<>(OUTPUT_FILE));
} catch (IOException e) {
Expand Down
2 changes: 1 addition & 1 deletion src/main/java/org/lobid/resources/run/CulturegraphXmlFilterHbzToJson.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ public static void main(String... args) {
final FileOpener opener = new FileOpener();
try {
opener.setReceiver(new XmlDecoder()).setReceiver(new MarcXmlHandler())
.setReceiver(new Metafix("src/main/resources/fix-cg-to-es.fix"))
.setReceiver(new Metafix("src/main/resources/rvk/cg-to-rvk-json.fix"))
.setReceiver(new JsonEncoder())
.setReceiver(new JsonToElasticsearchBulk(ELASTICSEARCH_INDEX_TYPE_NAME, ELASTICSEARCH_INDEX_NAME))
.setReceiver(new ObjectWriter<>(JSON_FILE));
Expand Down
40 changes: 40 additions & 0 deletions src/main/resources/rvk/cg-to-rvk-csv.fix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
set_array("records[]")
set_array("@id[]")
set_array("rvk[]")

do list(path: "084??", "var": "$i")
if any_match("$i.2", "rvk")
copy_field("$i.a","rvk[].$append")
end
end

uniq("rvk[]")
join_field("rvk[]",",")


do list(path: "035??", "var": "$i")
if any_match("$i.a", "^\\(DE-605\\)(.*)")
copy_field("$i.a","@id[].$append")
end
end
replace_all("id[].*","^\\(DE-605\\)(.*)","$1")

do list(path: "@id[]", "var": "$i")
copy_field("$i","records[].$append.id")
copy_field("rvk[]","records[].$last.rvk[]")
end
replace_all("records[].*.id","^\\(DE-605\\)(.*)","$1")

vacuum()

# Filter records without RVK
unless exists("rvk[]")
reject()
end

# Filter records without hbz ids
unless exists("@id[]")
reject()
end

retain("records[]")
17 changes: 9 additions & 8 deletions src/main/resources/fix-cg-to-es.fix → src/main/resources/rvk/cg-to-rvk-json.fix
View file
Open in desktop
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need two fixes?

Copy link
Member

@dr0i dr0i Jun 3, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In json we don't need a record for every id on its own - the search is done by the search engine, in contrast to a csv where we need a single unique key. The json is more performant when using search engines, the csv is the only way to go when using tables.
Could also be, if the csv is working great, that we can get rid of jsonaltogether.

Original file line number Diff line number Diff line change
@@ -1,18 +1,19 @@
do list(path: "035??", "var": "$i")
if any_match("$i.a", "^\\(DE-605\\)(.*)")
copy_field("$i.a","id")
end
end
replace_all("id","^\\(DE-605\\)(.*)","$1")

set_array("rvk[]")

do list(path: "084??", "var": "$i")
if any_match("$i.2", "rvk")
copy_field("$i.a","rvk[].$append")
end
end
join_field("rvk[]",", ")

set_array("id")
dr0i marked this conversation as resolved.
Show resolved Hide resolved
do list(path: "035??", "var": "$i")
if any_match("$i.a", "^\\(DE-605\\)(.*)")
copy_field("$i.a","id.$append")
end
end
replace_all("id.*","^\\(DE-605\\)(.*)","$1")
join_field("id",", ")

retain("rvk[]","id")
vacuum()
Expand Down
4 changes: 2 additions & 2 deletions src/test/resources/cg/output-es-bulk.ndjson
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{"index":{"_index":"cg","_type":"rvk"}}
{"rvk":["MG 11380, MF 3390, MG 11380, MC 7200, MF 3390, MG 11380"],"id":"990067531130206441"}
{"rvk":["MG 11380","MF 3390","MG 11380","MC 7200","MF 3390","MG 11380"],"id":"990067531130206441"}
{"index":{"_index":"cg","_type":"rvk"}}
{"rvk":["NQ 2360, NY 4760, BD 7100, NQ 2360, NY 4760, NQ 2360, NY 4760, BD 7100, NQ 2360, NY 4760"],"id":"990063668050206441"}
{"rvk":["NQ 2360","NY 4760","BD 7100","NQ 2360","NY 4760","NQ 2360","NY 4760","BD 7100","NQ 2360","NY 4760"],"id":"990019247190206441, 990063057720206441, 990063668050206441"}
6 changes: 4 additions & 2 deletions src/test/resources/cg/output.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,4 @@
"990067531130206441","MG 11380, MF 3390, MG 11380, MC 7200, MF 3390, MG 11380"
"990063668050206441","NQ 2360, NY 4760, BD 7100, NQ 2360, NY 4760, NQ 2360, NY 4760, BD 7100, NQ 2360, NY 4760"
"990067531130206441","MG 11380,MF 3390,MC 7200"
"990019247190206441","NQ 2360,NY 4760,BD 7100"
"990063057720206441","NQ 2360,NY 4760,BD 7100"
"990063668050206441","NQ 2360,NY 4760,BD 7100"
Loading