NoSuchFileException if fcrepo 4 export didn't create directories alongside ACL files

[cjcolvar]

I ran a successful export of a fcrepo 4.7.5 using the import/export tool version 0.3.0, but when trying to migrate the export to fcrepo 5 I run into a NoSuchFileException:

java.lang.RuntimeException: java.nio.file.NoSuchFileException: archivo_fcrepo4.7.5_export/fedora/rest/af/21/ab/71/af21ab71-c690-4c3b-a608-a76dd1529044
at org.fcrepo.upgrade.utils.F47ToF5UpgradeManager.convertAcl(F47ToF5UpgradeManager.java:372)
at org.fcrepo.upgrade.utils.F47ToF5UpgradeManager.lambda$upgradeRdfAndCreateHeaders$5(F47ToF5UpgradeManager.java:262)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
at org.fcrepo.upgrade.utils.F47ToF5UpgradeManager.upgradeRdfAndCreateHeaders(F47ToF5UpgradeManager.java:200)
at org.fcrepo.upgrade.utils.F47ToF5UpgradeManager.processFile(F47ToF5UpgradeManager.java:149)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1939)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
at org.fcrepo.upgrade.utils.F47ToF5UpgradeManager.processDirectory(F47ToF5UpgradeManager.java:114)
at org.fcrepo.upgrade.utils.F47ToF5UpgradeManager.start(F47ToF5UpgradeManager.java:108)
at org.fcrepo.upgrade.utils.UpgradeUtilDriver.run(UpgradeUtilDriver.java:75)
at org.fcrepo.upgrade.utils.UpgradeUtilDriver.main(UpgradeUtilDriver.java:60)
Caused by: java.nio.file.NoSuchFileException: archivo_fcrepo4.7.5_export/fedora/rest/af/21/ab/71/af21ab71-c690-4c3b-a608-a76dd1529044
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:171)
at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
at java.base/java.nio.file.Files.readAttributes(Files.java:1853)
at java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:220)
at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:277)
at java.base/java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:323)
at java.base/java.nio.file.FileTreeIterator.(FileTreeIterator.java:71)
at java.base/java.nio.file.Files.walk(Files.java:3898)
at java.base/java.nio.file.Files.walk(Files.java:3953)
at org.fcrepo.upgrade.utils.F47ToF5UpgradeManager.convertAcl(F47ToF5UpgradeManager.java:347)
... 19 more

From what I can tell the ACL that is trying to be converted is empty. It does not contain any child nodes. Other ACLs that were processed before this error did contain children. Because they had children they had directories created for them alongside the turtle file. The ones that didn't have children did not have directories created which I believe led to the above error. I wrote a quick script to create these empty directories for all ACLs that were missing them and the migration ran successfully. My guess is that there needs to be a check added before attempting the File.walk to skip if the directory doesn't exist.

fcrepo-upgrade-utils/src/main/java/org/fcrepo/upgrade/utils/F47ToF5UpgradeManager.java

Line 347 in b2c43e1

try (final Stream<Path> list = Files.walk(aclDirectory)) {

[cjcolvar]

Looks like there is a fix for this recently implemented in a fork: TAMULib#1

[whikloj]

Sorry @cjcolvar totally missed this. My memory of 4.7 is slowing weakening. Do you know if these ACLs are empty? Or perhaps had data and then it was removed?

[cjcolvar]

@whikloj The original data in Fedora 4.7 is for a Samvera (ActiveFedora-based) application. It created nodes in Fedora for ACLs but I think did so in a different way then the upgrade tool expects. It is possible that some of these ACLs didn't have any RDF statements of interest to the upgrade tool.

I changed my approach to working around this issue and instead of generating the empty directories I ended up preserving the old triples and letting them get processed as any normal triple would. This worked better for me because ActiveFedora hasn't been setup to handle the new ACLs created by the upgrade tool. This was also the approach I took for another issue I had with the upgrade tool where we needed to turn off special handling of certain predicates. So I think ultimately that is what would be best: add flags/options to skip special handling of specific predicates.

Here's the diff of changes I made that allowed a successful (and quick) upgrade.

[whikloj]

Would you be able to package up an object or two of your 4.7 Avalon. Without any sensitive data of course, but then I could build an actual test to see what you are starting with and understand what you are ending up with too.

Also, are you just migrating from 4.7 to 5.x right now or are you trying to move on from there to 6.x?

[cjcolvar]

Sure, I'll try to pack something up. The goal was moving to 6.x.