Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add backend: ProQuest Federated Search Gateway #3991

Open
wants to merge 76 commits into
base: dev
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 60 commits
Commits
Show all changes
76 commits
Select commit Hold shift + click to select a range
9263c83
Add ProQuest Federated Search Gateway backend. First working draft.
maccabeelevine Oct 4, 2024
518de13
Support basic search indices
maccabeelevine Oct 4, 2024
e6c1cd1
Implement sorting
maccabeelevine Oct 4, 2024
300c51f
Add search result limits
maccabeelevine Oct 4, 2024
df23ce6
Display database facets
maccabeelevine Oct 4, 2024
0fb4406
Apply database filter
maccabeelevine Oct 7, 2024
0f3021c
Support checkbox filters
maccabeelevine Oct 7, 2024
ebab982
Add OOTB facet settings
maccabeelevine Oct 7, 2024
8305dd5
Add SearchCache support
maccabeelevine Oct 7, 2024
a23fefc
Fix NPE
maccabeelevine Oct 7, 2024
d77a08d
Fix single record load
maccabeelevine Oct 7, 2024
fc89632
Add next_prev_navigation
maccabeelevine Oct 8, 2024
5d5b9d0
Fix copy/paste typo
maccabeelevine Oct 8, 2024
896ba94
Remove dead code
maccabeelevine Oct 8, 2024
6ffeaa3
Add retain/reset filters configs that work OOTB
maccabeelevine Oct 8, 2024
d856c11
Add pagination and recommendation settings, work OOTB
maccabeelevine Oct 8, 2024
b5a7ddd
Simply constant backend params
maccabeelevine Oct 8, 2024
2d8b4c8
Support list view options
maccabeelevine Oct 8, 2024
f40b553
Fix typo
maccabeelevine Oct 8, 2024
a14a801
Remove dead code
maccabeelevine Oct 8, 2024
86abb67
Add advanced search support
maccabeelevine Oct 8, 2024
ec2dc6c
Fix styles and add missing comments
maccabeelevine Oct 8, 2024
4a0b7b5
Strip html from title
maccabeelevine Oct 9, 2024
190abbe
Merge branch 'dev' into proquest-federated-search-gateway
maccabeelevine Nov 6, 2024
d698830
Convert to bootstrap5
maccabeelevine Nov 6, 2024
9670ee6
Add recommendation module
maccabeelevine Nov 6, 2024
2e85a0f
Fix styles
maccabeelevine Nov 6, 2024
d22ff04
Fix copy/paste typos
maccabeelevine Nov 6, 2024
1741fb9
Fix more typos
maccabeelevine Nov 6, 2024
f58c0a4
Add tests
maccabeelevine Nov 8, 2024
f1c3776
Remove blank line
maccabeelevine Nov 8, 2024
e3824df
Perform search if necessary before getting facet list.
maccabeelevine Nov 8, 2024
645eb82
Simplify conditional ParamBag creation
maccabeelevine Nov 8, 2024
6b9ce46
Clean code.
maccabeelevine Nov 8, 2024
2a4c875
Protect missing docs in response
maccabeelevine Nov 8, 2024
f57b5ba
Merge branch 'dev' into proquest-federated-search-gateway
maccabeelevine Dec 2, 2024
5b9bebd
Add result_limit config
maccabeelevine Dec 2, 2024
c483813
Add import
maccabeelevine Dec 2, 2024
37c9675
Check if ProQuest is enabled first
maccabeelevine Dec 3, 2024
eb64924
Catch null sourceId
maccabeelevine Dec 3, 2024
337966f
Fix whitespace
maccabeelevine Dec 3, 2024
3de7007
Remove unneeded space
maccabeelevine Dec 3, 2024
8338a7b
Restored stupid ugly space that should be unnecessary
maccabeelevine Dec 3, 2024
6bd5782
Remove unused connector options
maccabeelevine Dec 3, 2024
2418fff
Remove deleted param comment
maccabeelevine Dec 3, 2024
651326b
Merge branch 'dev' into proquest-federated-search-gateway
maccabeelevine Dec 10, 2024
9c33653
Fix null typing
maccabeelevine Dec 10, 2024
87ef65d
Prevent an empty search, with useful message
maccabeelevine Dec 10, 2024
19184b6
Add missing include
maccabeelevine Dec 10, 2024
56c2058
Sort facets by result count
maccabeelevine Dec 10, 2024
09616b1
Strip HTML from titles on results list, for now
maccabeelevine Dec 10, 2024
ac20d8f
Display publication (journal-level) info and enable journal name search
maccabeelevine Dec 12, 2024
a5863d7
Add database/source name to description tab
maccabeelevine Dec 12, 2024
10075c4
Remove Holdings and Similar Items tabs
maccabeelevine Dec 12, 2024
f33dd28
Fix staff view tab.
maccabeelevine Dec 12, 2024
a6e7c4b
Change default operator to equals, not 'all', matching ProQuest UI
maccabeelevine Dec 13, 2024
68cb9eb
Merge branch 'dev' into proquest-federated-search-gateway
maccabeelevine Jan 21, 2025
b7cdf1b
Add DOI to record driver so it appears in the description tab
maccabeelevine Jan 21, 2025
7ecd9e4
Remove publication data from 'Published' field, so it displays only t…
maccabeelevine Jan 21, 2025
e7888af
Sync bootstrap3
maccabeelevine Jan 21, 2025
5ec7bfc
For books, truncate publication date to year
maccabeelevine Jan 27, 2025
4b36eb0
Check for SRU errors. This includes SRU error type 3 "Authentication…
maccabeelevine Jan 27, 2025
2a8fc17
Fix style
maccabeelevine Jan 27, 2025
0814acd
Be safer about the error message
maccabeelevine Jan 27, 2025
2f7531a
Merge branch 'dev' into proquest-federated-search-gateway
maccabeelevine Jan 28, 2025
35ce055
Suppress container info for books
maccabeelevine Jan 28, 2025
60de692
Decode html entities
maccabeelevine Jan 28, 2025
96cd4fb
Perform a second search for facets when needed
maccabeelevine Jan 29, 2025
11b8543
Merge branch 'dev' into proquest-federated-search-gateway
maccabeelevine Jan 29, 2025
ed7c53f
Define 'book' based on ProQuest data
maccabeelevine Jan 29, 2025
d42f516
Restore defaults for retain/reset filters
maccabeelevine Jan 29, 2025
5694042
Remove references to Primo
maccabeelevine Jan 29, 2025
039db68
Fix author / copyright
maccabeelevine Jan 29, 2025
cab3c23
Remove new bootstrap3 templates created before that theme was removed
maccabeelevine Jan 29, 2025
c2e6276
Save an API call if no facets are configured
maccabeelevine Jan 29, 2025
8248d16
Look to facet config for whether any should be displayed, since a nul…
maccabeelevine Jan 29, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
162 changes: 162 additions & 0 deletions config/vufind/ProQuestFSG.ini
Original file line number Diff line number Diff line change
@@ -0,0 +1,162 @@
; This section contains global settings affecting search behavior.
; For ProQuest Federated Search Gateway API documentation:
; https://support.proquest.com/s/article/ProQuest-Academic-Platform-Federated-Search-Options
[General]
; Enable the ProQuest Federated Search Gateway integration. Disabled by default.
;enabled = true

; This setting controls the default sort order of search results; the selected
; option should be one of the options present in the [Sorting] section below.
default_sort = relevance
; This section controls the result limit options for search results. default_limit
; sets the default number of results per page. limit_options is a comma-separated
; list of numbers to be presented to the end-user. If only one limit is required,
; set default_limit and leave limit_options commented out.
; WARNING: using large limits may cause problems due to a variety of limitations,
; especially if you support bulk operations (which can cause large URLs/requests).
; If you must support large page sizes, you may need to raise the PHP memory_limit
; and max_input_vars settings and/or adjust the Apache LimitRequestLine setting.
default_limit = 20
; limit_options = 10,20,40,60,80,100

demiankatz marked this conversation as resolved.
Show resolved Hide resolved
; This setting allows to limit pagination of a search result as deep pagination
; costs a lot of performance and most users are not very likely to navigate
; further down than 20 pages of a search result.
; This is especially useful to prevent webcrawlers from descending too deep and
; eating up search backend performance.
; Default is 400; the API supports a maximum limit of 1000.
;result_limit = 400

; These are the default recommendations modules to use when no specific setting
; are found in the [TopRecommendations], [SideRecommendations] or
; [NoResultsRecommendations] sections below.
; See the comments above those sections for details on legal settings. You may
; repeat these lines to load multiple recommendations.
;default_top_recommend[] = TopFacets:FacetsTop:Primo
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stray Primo reference here. Should we just remove this line?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one is not necessarily wrong (you could??? bring in Primo results at the top) but it's definitely confusing and shout not be there. Removed.

default_side_recommend[] = SideFacets:Facets:CheckboxFacets:ProQuestFSG
;default_noresults_recommend[] = SwitchTab
default_noresults_recommend[] = RemoveFilters

; When you filter a search using facets, should VuFind retain your current filters
; on the next search and provide a reset button to clear them (true), or should it
; always perform new searches unfiltered (false)?
; retain_filters_by_default = true
retain_filters_by_default = false

; Whether to always (when there are active filters) display the "Reset Filters"
; button regardless of the retain_filters_by_default setting. Default is false.
; always_display_reset_filters = false
always_display_reset_filters = true

; This controls whether results are loaded with JavaScript when paging or changing
; settings. Loading results this way improves performance and accessibility, and is
; enabled by default. Loading will automatically fall back to non-JS mode if
; JavaScript is not supported by the browser.
load_results_with_js = true

; This setting can be used to configure pagination control on top of results.
; Possible options are:
; empty string or false No top pagination (default)
; simple Simple next/prev button pagination
; full Full pagination alike to the one at the bottom of results
;top_paginator = simple

; The following two sections can be used to associate specific recommendations
; modules with specific search types defined in the [Basic_Searches] section
; below. For all the details on how these sections work, see the comments above
; the equivalent sections of searches.ini. Recommendations work the same in
; ProQuestFSG as they do in the regular Search module.
;[SideRecommendations]
; No search-specific settings by default -- add your own here.
;[TopRecommendations]
; No search-specific settings by default -- add your own here.

maccabeelevine marked this conversation as resolved.
Show resolved Hide resolved
; The order of display is as shown below
; The name of the index field is on the left
; The display name of the field is on the right
[Facets]
Databases = "Source"
sturkel89 marked this conversation as resolved.
Show resolved Hide resolved

; Checkbox facets are facets, which are getting displayed as checkboxes
; pcAvailability is a special facet. It's used to show all records found in
; PrimoCentral without checking the local availability against a holdingsfile.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment also seems to be copied from elsewhere and probably not relevant here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, yes clearly using Primo.ini as a source was not ideal. I've reverted this to (almost) what it says in facets.ini.

[CheckboxFacets]
limitFullText = "fulltext_limit"
limitScholarlyJournals = "scholarly_limit"
limitPeerReviewed = "peer_reviewed_limit"

; Facet display settings
[Results_Settings]
; By default, the side facets will only show 6 facets and then the "show more"
; button. This can get configured with the showMore settings.
; You can use the * to set a new default setting.
showMore[*] = 6
; Or you can set a facet specific value by using the facet name as index.
;showMore['rtype'] = 10
; Do we want any facets to be collapsed by default?
;collapsedFacets = *

; This section shows which search types will display in the basic search box at
; the top of Primo pages. The name of each setting below corresponds with an
; index defined in the Primo API. The value of each setting is the text to
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More Primo references here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed!

; display on screen. All on-screen text will be run through the translator, so
; be sure to update language files if necessary. The order of these settings
; will be maintained in the drop-down list in the UI.
[Basic_Searches]
cql.serverChoice = "All Fields"
title = Title
author = Author
subject = Subject
publication = Journal Title
;abstract = Abstract

; This section defines which search options will be included on the advanced
; search screen. All the notes above [Basic_Searches] also apply here.
[Advanced_Searches]
cql.serverChoice = "All Fields"
title = Title
author = Author
subject = Subject
publication = Journal Title
;abstract = Abstract

; This section defines the sort options available on search results.
[Sorting]
relevance = sort_relevance
date/descending = sort_year
date/ascending = sort_year_asc

; This section allows you to specify hidden sorting options. They can be used to create a

; This section controls the behavior of the ProQuestFSGRecord module.
[Record]
; Set this to true in order to enable "next" and "previous" links to navigate
; through the current result set from within the record view.
next_prev_navigation = false

; This section controls what happens when a record title in a search result list
; is clicked. VuFind can either embed the full result directly in the list using
; AJAX or can display it at its own separate URL as a full HTML page.
; full - separate page (default)
; tabs - embedded using tabs (see record/tabs.phtml)
; accordion - embedded using an accordion (see record/accordion.phtml)
; NOTE: To turn this feature on for favorite lists, see the lists_view setting
; in the [Social] section of config.ini.
; NOTE: This feature is incompatible with SyndeticsPlus content; please use
; regular Syndetics if necessary.
[List]
view=full
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Has anyone tested whether tab/accordion modes work here? Let us know if you'd like @sturkel89 or me to give it a try!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I definitely tested it during development (only added stuff to ProQuest.ini that I tested -- except obviously I copied over a lot of bad Primo comments :) ) but that's definitely not proof it works as designed.


; This section provides settings for optional caching of search requests.
[SearchCache]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm pretty sure that your connector here does not support search caching. Am I missing something, or should we remove these settings for now?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be working! It's actually the SRU Connector that uses ConnectorCacheTrait and uses it in search(); looks like I added that in this PR.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fantastic! Sorry for missing that detail -- didn't look high enough in the class hierarchy.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No worries, I had to check myself anyway, I wrote that part back in October!

; Supported adapters: Memcached, Filesystem. Others may also work, but have not been
; tested.
;adapter = Memcached
; Comma-separated list of servers for Memcached adapter:
;options[servers] = "localhost:11211"
; Cache directory for Filesystem adapter:
;options[cache_dir] = "/tmp/search-cache"
; Other options. See https://docs.laminas.dev/laminas-cache/storage/adapter/ for
; documentation on options for different adapters.
; ttl, Time to Live, i.e. cache entry life time in seconds. 300 seconds by default.
;options[ttl] = 300
7 changes: 7 additions & 0 deletions config/vufind/RecordTabs.ini
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,13 @@ tabs[Preview] = preview
tabs[Details] = StaffViewArray
defaultTab = null

[VuFind\RecordDriver\ProQuestFSG]
sturkel89 marked this conversation as resolved.
Show resolved Hide resolved
tabs[Description] = Description
tabs[UserComments] = UserComments
tabs[Reviews] = Reviews
tabs[Details] = StaffViewMARC
sturkel89 marked this conversation as resolved.
Show resolved Hide resolved
defaultTab = null

[VuFind\RecordDriver\SolrAuthDefault]
tabs[Details] = StaffViewArray
defaultTab = null
Expand Down
2 changes: 2 additions & 0 deletions languages/en.ini
Original file line number Diff line number Diff line change
Expand Up @@ -1150,6 +1150,8 @@ Production Credits = "Production Credits"
Profile = "Profile"
profile_update = "Your profile was updated as requested"
pronounced = "pronounced"
proquestfsg_recommendations = "ProQuest Results"
proquestfsg_recommendations_more = "More ProQuest Results"
maccabeelevine marked this conversation as resolved.
Show resolved Hide resolved
Provider = "Provider"
proxied_user = "Place request for proxied user"
proxy_hold_place_success_html = 'Your proxy request was successful. <a href="%%url%%">Your Holds and Recalls</a>.'
Expand Down
8 changes: 8 additions & 0 deletions module/VuFind/config/module.config.php
Original file line number Diff line number Diff line change
Expand Up @@ -211,6 +211,8 @@
'VuFind\Controller\Pazpar2Controller' => 'VuFind\Controller\AbstractBaseFactory',
'VuFind\Controller\PrimoController' => 'VuFind\Controller\AbstractBaseFactory',
'VuFind\Controller\PrimorecordController' => 'VuFind\Controller\AbstractBaseFactory',
'VuFind\Controller\ProQuestFSGController' => 'VuFind\Controller\AbstractBaseFactory',
'VuFind\Controller\ProQuestFSGrecordController' => 'VuFind\Controller\AbstractBaseFactory',
'VuFind\Controller\QRCodeController' => 'VuFind\Controller\QRCodeControllerFactory',
'VuFind\Controller\RecordController' => 'VuFind\Controller\AbstractBaseWithConfigFactory',
'VuFind\Controller\RecordsController' => 'VuFind\Controller\AbstractBaseFactory',
Expand Down Expand Up @@ -325,6 +327,10 @@
'primo' => 'VuFind\Controller\PrimoController',
'PrimoRecord' => 'VuFind\Controller\PrimorecordController',
'primorecord' => 'VuFind\Controller\PrimorecordController',
'ProQuestFSG' => 'VuFind\Controller\ProQuestFSGController',
'proquestfsg' => 'VuFind\Controller\ProQuestFSGController',
'ProQuestFSGRecord' => 'VuFind\Controller\ProQuestFSGrecordController',
'proquestfsgrecord' => 'VuFind\Controller\ProQuestFSGrecordController',
'QRCode' => 'VuFind\Controller\QRCodeController',
'qrcode' => 'VuFind\Controller\QRCodeController',
'Record' => 'VuFind\Controller\RecordController',
Expand Down Expand Up @@ -753,6 +759,7 @@
'epfrecord' => 'EPFRecord',
'missingrecord' => 'MissingRecord',
'primorecord' => 'PrimoRecord',
'proquestfsgrecord' => 'ProQuestFSGRecord',
'solrauthrecord' => 'AuthorityRecord',
'summonrecord' => 'SummonRecord',
'worldcatrecord' => 'WorldcatRecord',
Expand Down Expand Up @@ -823,6 +830,7 @@
'Overdrive/MyContent','Overdrive/Hold',
'Pazpar2/Home', 'Pazpar2/Search',
'Primo/Advanced', 'Primo/CitedBy', 'Primo/Cites', 'Primo/Home', 'Primo/Search',
'ProQuestFSG/Advanced', 'ProQuestFSG/Home', 'ProQuestFSG/Results',
'QRCode/Show', 'QRCode/Unavailable', 'Records/Home',
'Relais/Login', 'Relais/Request',
'Search/Advanced',
Expand Down
81 changes: 81 additions & 0 deletions module/VuFind/src/VuFind/Controller/Feature/CheckEnabledTrait.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
<?php

/**
* Check Enabled Trait
*
* PHP version 8
*
* Copyright (C) Villanova University 2024.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2,
* as published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
*
* @category VuFind
* @package Controller_Plugins
* @author Maccabee Levine <[email protected]>
* @license http://opensource.org/licenses/gpl-2.0.php GNU General Public License
* @link https://vufind.org Main Page
*/

namespace VuFind\Controller\Feature;

use Laminas\Mvc\MvcEvent;
use VuFind\Exception\Forbidden as ForbiddenException;

/**
* Check Enabled Trait
*
* @category VuFind
* @package Controller_Plugins
* @author Maccabee Levine <[email protected]>
* @license http://opensource.org/licenses/gpl-2.0.php GNU General Public License
* @link https://vufind.org Main Page
*/
trait CheckEnabledTrait
{
/**
* Check whether the controller is enabled
*
* @return void
*
* @throws ForbiddenException if the controller is not enabled
*/
protected function checkEnabled()
{
$configId = $this->searchClassId ?? $this->sourceId ?? null;
if (!$configId) {
return;
}

$config = $this->getConfig($configId);
if (!($config['General']['enabled'] ?? false)) {
throw new ForbiddenException($configId . ' is disabled');
}
}

/**
* Add to event listeners a check that the controller is enabled
*
* @return void
*/
protected function attachDefaultListeners()
{
parent::attachDefaultListeners();
$events = $this->getEventManager();
$events->attach(
MvcEvent::EVENT_DISPATCH,
fn () => $this->checkEnabled(),
1000
);
}
}
71 changes: 71 additions & 0 deletions module/VuFind/src/VuFind/Controller/ProQuestFSGController.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
<?php

/**
* ProQuest Federated Search Gateway Controller
*
* PHP version 8
*
* Copyright (C) Villanova University 2024.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2,
* as published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
*
* @category VuFind
* @package Controller
* @author Demian Katz <[email protected]>
* @author Maccabee Levine <[email protected]>
* @license http://opensource.org/licenses/gpl-2.0.php GNU General Public License
* @link https://vufind.org Main Site
*/

namespace VuFind\Controller;

use Laminas\ServiceManager\ServiceLocatorInterface;
use VuFind\Controller\Feature\CheckEnabledTrait;

/**
* ProQuest Federated Search Gateway Controller
*
* @category VuFind
* @package Controller
* @author Demian Katz <[email protected]>
* @author Maccabee Levine <[email protected]>
* @license http://opensource.org/licenses/gpl-2.0.php GNU General Public License
* @link https://vufind.org Main Site
*/
class ProQuestFSGController extends AbstractSearch
{
use CheckEnabledTrait;

/**
* Constructor
*
* @param ServiceLocatorInterface $sm Service locator
*/
public function __construct(ServiceLocatorInterface $sm)
{
$this->searchClassId = 'ProQuestFSG';
parent::__construct($sm);
}

/**
* Is the result scroller active?
*
* @return bool
*/
protected function resultScrollerActive()
{
$config = $this->getService(\VuFind\Config\PluginManager::class)->get('ProQuestFSG');
return $config->Record->next_prev_navigation ?? false;
}
}
Loading
Loading