Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle spaces in file names in the import scripts #4189

Open
wants to merge 3 commits into
base: release-10.1
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 16 additions & 12 deletions harvest/batch-import-marc.sh
Original file line number Diff line number Diff line change
Expand Up @@ -127,28 +127,32 @@ else
local LOGFILE
if [ $# -eq 1 ]
then
LOGFILE=$BASEPATH/log/`basename $1`.log
> $LOGFILE
LOGFILE=$BASEPATH/log/`basename "$1"`.log
> "$LOGFILE"
else
LOGFILE=$BASEPATH/log/`basename $1`_and_more.log
LOGFILE=$BASEPATH/log/`basename "$1"`_and_more.log
echo -e "This log is for the following files: \n$FILES\n" > $LOGFILE
fi
cat -u - >> $LOGFILE
cat -u - >> "$LOGFILE"
}
fi

# Process all the files in the target directory:
find -L $BASEPATH -maxdepth 1 \( -iname "*.xml" -o -iname "*.mrc" -o -iname "*.marc" \) -type f -print0 | sort -z | xargs -0 -r -n $MAX_BATCH_COUNT | \
while read -d $'\n' files
find -L $BASEPATH -maxdepth 1 \( -iname "*.xml" -o -iname "*.mrc" -o -iname "*.marc" \) -type f -print0 \
| sort -z \
| xargs -0 -r -n $MAX_BATCH_COUNT \
| while read -d $'\n' files
do
# Logging output handled by log() function
# PROPERTIES_FILE passed via environment
$VUFIND_HOME/import-marc.sh $files 2> >(log $files)
if [ "$?" -eq "0" ] && [ $MOVE_DATA == true ]
then
for file in $files
do
mv $file $BASEPATH/processed/`basename $file`
$VUFIND_HOME/import-marc.sh "$files" 2> >(log "$files")

if [ "$?" -eq 0 ] && [ "$MOVE_DATA" == true ]; then
# Convert the space-separated file string into an array
IFS= read -r -a file_array <<< "$files"

for file in "${file_array[@]}"; do
mv "$file" "$BASEPATH/processed/$(basename "$file")"
done
fi
done
41 changes: 29 additions & 12 deletions import-marc.sh
Original file line number Diff line number Diff line change
Expand Up @@ -130,12 +130,14 @@ fi
#####################################################
# Normalize file paths to absolute paths
#####################################################
NORMALIZED_PATHS=""
for f in $*; do
MARC_PATH=`dirname $f`
MARC_PATH=`cd $MARC_PATH && pwd`
MARC_FILE=`basename $f`
NORMALIZED_PATHS="${NORMALIZED_PATHS} $MARC_PATH/$MARC_FILE"
NORMALIZED_PATHS=()

for f in "$@"; do
MARC_PATH=$(dirname "$f")
MARC_PATH=$(cd "$MARC_PATH" && pwd) # Resolve the full path to prevent relative path issues
MARC_FILE=$(basename "$f")
# Add the full path to the array
NORMALIZED_PATHS+=("$MARC_PATH/$MARC_FILE")
done

#####################################################
Expand All @@ -153,16 +155,31 @@ then
mkdir -p $SOLRJ_DIR
for file in $VUFIND_HOME/solr/vendor/server/solr-webapp/webapp/WEB-INF/lib/solr*.jar $VUFIND_HOME/solr/vendor/server/solr-webapp/webapp/WEB-INF/lib/http*.jar
do
ln -s $file $SOLRJ_DIR/`basename $file`
ln -s $file $SOLRJ_DIR/`basename "$file"`
done
fi

#####################################################
# Execute Importer
#####################################################

RUN_CMD="$JAVA $INDEX_OPTIONS -Duser.timezone=UTC -Dlog4j.configuration=file://$LOG4J_CONFIG $EXTRA_SOLRMARC_SETTINGS -jar $JAR_FILE $PROPERTIES_FILE -solrj $SOLRJ_DIR -lib_local "$VUFIND_HOME/import/lib_local\;$VUFIND_HOME/solr/vendor/modules/analysis-extras/lib" $NORMALIZED_PATHS"
echo "Now Importing $NORMALIZED_PATHS ..."
# solrmarc writes log messages to stderr, write RUN_CMD to the same place
echo "`date '+%h %d, %H:%M:%S'` $RUN_CMD" >&2
exec $RUN_CMD
# Build the command as an array
RUN_CMD=(
"$JAVA"
$INDEX_OPTIONS
-Duser.timezone=UTC
-Dlog4j.configuration="file://$LOG4J_CONFIG"
$EXTRA_SOLRMARC_SETTINGS
-jar "$JAR_FILE"
"$PROPERTIES_FILE"
-solrj "$SOLRJ_DIR"
-lib_local "$VUFIND_HOME/import/lib_local;$VUFIND_HOME/solr/vendor/modules/analysis-extras/lib"
"${NORMALIZED_PATHS[@]}"
)

# Debugging output
echo "Now Importing: ${NORMALIZED_PATHS[*]}"
echo "$(date '+%h %d, %H:%M:%S') ${RUN_CMD[*]}" >&2

# Execute the command using the array
exec "${RUN_CMD[@]}"
dltj marked this conversation as resolved.
Show resolved Hide resolved
Loading