Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Visconnect prototype base enrichment #771

Open
wants to merge 87 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
87 commits
Select commit Hold shift + click to select a range
f418068
wip
chreman Apr 24, 2024
b763365
wip
chreman Jun 13, 2024
cf9dfdb
deployment updates
chreman Jun 13, 2024
6401c1f
orcid wip
chreman Jun 20, 2024
6c98409
orcid wip
chreman Jun 20, 2024
62bcb36
gsheets cleanup
chreman Jun 20, 2024
70d64ec
orcid wip
chreman Jun 20, 2024
2960aa4
gsheets cleanup
chreman Jun 20, 2024
4cb0132
gsheets cleanup
chreman Jun 20, 2024
6e7a1d6
gsheets cleanup
chreman Jun 20, 2024
ed5206a
orcid metadata mapping
chreman Jun 20, 2024
27cc689
orcid rate limit bugfix
chreman Jun 20, 2024
df57617
orcid container deployment
chreman Jun 20, 2024
ec96de5
orcid client initiation bugfix
chreman Jun 20, 2024
15425f8
orcid integration bugfixes
chreman Jun 21, 2024
3bc9c3b
metadata sanitization
chreman Jun 21, 2024
3f7d3f2
deduplication wip
chreman Jun 25, 2024
ea0b0f3
doc type tags
chreman Jun 25, 2024
ad32c7f
date handling
chreman Jun 25, 2024
2a543f2
add today param to orcid
chreman Jun 25, 2024
bf9311d
logging cleanup
chreman Jun 25, 2024
a22e321
display doc types in orcid integration
chreman Jun 25, 2024
3f4c7b7
temporary change for dev process
chreman Jul 2, 2024
e597656
added missing files
chreman Jul 5, 2024
4075449
metadata enrichment wip
chreman Jul 6, 2024
9e341b8
metadata enrichment wip
chreman Jul 6, 2024
14ec80b
feat: extend orcid pipeline (data extraction)
Jul 8, 2024
1df2ec3
dataset bugfix; error handling bugfix
chreman Jul 8, 2024
f43c492
cleanup
chreman Jul 9, 2024
d586adb
Merge branch 'visconnect-prototype-updates' into visconnect-prototype
chreman Jul 9, 2024
9a1467c
feat: add reference to pyorcid repo
modsen-hedgehog Jul 9, 2024
d33d8ca
updates for new metadata
chreman Jul 9, 2024
d760588
Merge remote-tracking branch 'upstream/visconnect-prototype' into vis…
chreman Jul 9, 2024
c668ab7
added doc strings
chreman Jul 9, 2024
b217e25
bugfix handling missing metadata
chreman Jul 9, 2024
d7b5523
author name bugfix
chreman Jul 10, 2024
c7d7e7d
robustify author metadata handling
chreman Jul 10, 2024
d6636cc
feat: (orcid) improvements in error handling and refactoring
modsen-hedgehog Jul 15, 2024
3dccdf6
Merge pull request #762 from OpenKnowledgeMaps/visconnect-prototype-u…
modsen-hedgehog Jul 16, 2024
31b4f80
Merge remote-tracking branch 'upstream/visconnect-prototype' into vis…
chreman Jul 22, 2024
132ab1b
minor README update
chreman Jul 24, 2024
fa5ad31
improve orcid error handling
modsen-hedgehog Jul 24, 2024
890d8da
Merge remote-tracking branch 'upstream/visconnect-prototype' into vis…
chreman Jul 25, 2024
9b908c2
get additional metadata for ORCID work URLs
chreman Jul 26, 2024
7fca252
works id bugfix
chreman Jul 29, 2024
bd39d37
readme update
chreman Jul 29, 2024
b745364
orcid PDF OA state workaround
chreman Jul 30, 2024
9f87e5a
fix: orcid publication date parsing
modsen-hedgehog Jul 30, 2024
c79e6c4
Merge pull request #763 from OpenKnowledgeMaps/visconnect-prototype-u…
modsen-hedgehog Jul 31, 2024
aaa7969
fix: get_publication_date for orcid
modsen-hedgehog Aug 1, 2024
4bc06d2
Merge pull request #764 from OpenKnowledgeMaps/visconnect-prototype-u…
modsen-hedgehog Aug 1, 2024
022ff2e
Merge remote-tracking branch 'upstream/visconnect-prototype' into vis…
chreman Aug 4, 2024
01510c4
Visconnect prototype metrics (#768)
modsen-hedgehog Sep 2, 2024
695485c
Merge remote-tracking branch 'upstream/visconnect-prototype' into vis…
chreman Sep 2, 2024
7ddf2e2
academic_age_offset param
chreman Sep 2, 2024
dadf5a2
academic_age_offset param
chreman Sep 3, 2024
d877f0b
academic_age_offset param
chreman Sep 3, 2024
fcebe8e
academic_age_offset param
chreman Sep 3, 2024
e424268
fix: minor python
modsen-hedgehog Sep 4, 2024
de22c7c
fix: dropdown stylings
modsen-hedgehog Sep 4, 2024
917bdff
fix: metadata parsing and sanitizing, tooltips
modsen-hedgehog Sep 6, 2024
d0e61cd
deployment changes
chreman Sep 3, 2024
f96f161
temporary PDF deactivation
chreman Sep 6, 2024
e54c709
fix: bugs including logs
modsen-hedgehog Sep 10, 2024
e533bf6
fix: works
modsen-hedgehog Sep 10, 2024
fd51a04
fix: paper
modsen-hedgehog Sep 10, 2024
664523f
Merge remote-tracking branch 'upstream/visconnect-prototype' into vis…
chreman Sep 10, 2024
b949801
fix: paper n/a
modsen-hedgehog Sep 11, 2024
9d36728
R metrics metadata handling bugfix
chreman Sep 12, 2024
429052e
Merge remote-tracking branch 'upstream/visconnect-prototype' into vis…
chreman Sep 12, 2024
ec52852
feat: implement researcher info modal
modsen-hedgehog Sep 13, 2024
74e7f99
fix: enrich researcher retrics and add missed data
modsen-hedgehog Sep 13, 2024
57987cf
fix: add omitted pyorcid update
modsen-hedgehog Sep 17, 2024
c3fc897
feat: add some tests
modsen-hedgehog Sep 17, 2024
c8cc5fc
feat: add mac gitignore
modsen-hedgehog Sep 17, 2024
69d5548
fix: escaping
modsen-hedgehog Sep 17, 2024
dfa62af
fix: fallback and sorting
modsen-hedgehog Sep 18, 2024
8542131
Visconnect prototype typescript (#770)
modsen-hedgehog Oct 21, 2024
3a14bee
feat: base enrichment workflow
modsen-hedgehog Oct 23, 2024
efe8d22
feat: save fixes for base enrichment
modsen-hedgehog Oct 31, 2024
19f96d8
feat: implement enable_h_index
modsen-hedgehog Nov 4, 2024
550de50
additional enrichment for relation field
chreman Nov 5, 2024
650ccc5
typo
chreman Nov 5, 2024
e651dcc
fix: enable h index
modsen-hedgehog Nov 5, 2024
817097d
Merge remote-tracking branch 'upstream/visconnect-prototype-base-enri…
chreman Nov 5, 2024
0f78c20
metadata updates
chreman Nov 6, 2024
4e0e167
ORCID metrics timeout increases
chreman Nov 12, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix: enrich researcher retrics and add missed data
modsen-hedgehog committed Sep 13, 2024
commit 74e7f99e58b595351d5eeb01f6b718723a63fa19
19 changes: 17 additions & 2 deletions server/workers/orcid/src/model.py
Original file line number Diff line number Diff line change
@@ -19,15 +19,28 @@ class ExternalIdentifier(TypedDict):

@dataclass
class Employment:
id: str
organization: Optional[str] = None
organization_address: Optional[str] = None
department: Optional[str] = None
role: Optional[str] = None
start_date: Optional[str] = None
end_date: Optional[str] = None

@dataclass
class Distinction:
id: str
organization: Optional[str] = None
organization_address: Optional[str] = None
department: Optional[str] = None
role: Optional[str] = None
start_date: Optional[str] = None
end_date: Optional[str] = None
url: Optional[str] = None

@dataclass
class Membership:
id: str
organization: Optional[str] = None
organization_address: Optional[str] = None
department: Optional[str] = None
@@ -42,6 +55,7 @@ class Amount:

@dataclass
class Funding:
id: str
title: str
type: str
start_date: str
@@ -53,6 +67,7 @@ class Funding:

@dataclass
class Education:
id: str
department: Optional[str] = None
role: Optional[str] = None
start_date: Optional[str] = None
@@ -61,10 +76,9 @@ class Education:
organization_address: Optional[str] = None
url: Optional[str] = None

# @data

@dataclass
class PeerReview:
id: str
type: Optional[str] = None
role: Optional[str] = None
url: Optional[str] = None
@@ -100,6 +114,7 @@ class AuthorInfo:
educations: List[Education] = field(default_factory=list)
memberships: List[Membership] = field(default_factory=list)
peer_reviews: List[PeerReview] = field(default_factory=list)
distinctions: List[Distinction] = field(default_factory=list)

@dataclass
class Work:
35 changes: 34 additions & 1 deletion server/workers/orcid/src/repositories/author_info.py
Original file line number Diff line number Diff line change
@@ -5,10 +5,19 @@
import numpy as np
from common.utils import get_nested_value
from typing import List, Dict
from model import AuthorInfo, ExternalIdentifier, Website, Employment, Funding, Education, Membership, PeerReview
from model import AuthorInfo, ExternalIdentifier, Website, Employment, Funding, Education, Membership, PeerReview, Distinction
from typing import Optional, Any
import calendar

import hashlib
import time


def unique_id():
unique_string = str(time.time()).encode() # Encode a unique string (e.g., timestamp)
short_unique_id = hashlib.md5(unique_string).hexdigest()[:8] # Get first 8 characters of the hash
return short_unique_id

class AuthorInfoRepository:
logger = logging.getLogger(__name__)

@@ -57,6 +66,10 @@ def extract_author_info(self) -> AuthorInfo:
if peer_reviews:
author_info.peer_reviews = self.extract_peer_reviews(peer_reviews)

distinctions, _ = self.orcid.distinctions()
if distinctions:
author_info.distinctions = self.extract_distinctions(distinctions)

return author_info

def extract_peer_reviews(self, peer_reviews: Any) -> List[PeerReview]:
@@ -74,6 +87,7 @@ def extract_peer_reviews(self, peer_reviews: Any) -> List[PeerReview]:


peer_review_list.append(PeerReview(
id=unique_id(),
role=summary.get('reviewer-role', None),
type=summary.get('review-type', None),
url=summary.get('review-url', None),
@@ -87,6 +101,7 @@ def extract_peer_reviews(self, peer_reviews: Any) -> List[PeerReview]:
def extract_memberships(self, memberships: List[Dict[str, str]]) -> List[Membership]:
return [
Membership(
id=unique_id(),
organization=membership.get("organization", ""),
organization_address=membership.get("organization-address", ""),
department=membership.get("Department", ""),
@@ -100,6 +115,7 @@ def extract_memberships(self, memberships: List[Dict[str, str]]) -> List[Members
def extract_educations(self, educations: List[Dict[str, str]]) -> List[Education]:
return [
Education(
id=unique_id(),
department=education.get("Department", None),
role=education.get("Role", None),
start_date=education.get("start-date", ""),
@@ -110,9 +126,24 @@ def extract_educations(self, educations: List[Dict[str, str]]) -> List[Education
)
for education in educations]

def extract_distinctions(self, distinctions: List[Dict[str, str]]) -> List[Distinction]:
return [
Distinction(
id=unique_id(),
department=education.get("Department", None),
role=education.get("Role", None),
start_date=education.get("start-date", ""),
end_date=education.get("end-date", ""),
organization=education.get("organization", ""),
organization_address=education.get("organization-address", ""),
url=education.get("url", "")
)
for education in distinctions]

def extract_funds(self, funds: List[Dict[str, str]]) -> List[Funding]:
return [
Funding(
id=unique_id(),
title=funding.get("title", ""),
type=funding.get("type", ""),
start_date=funding.get("start-date", ""),
@@ -127,6 +158,7 @@ def extract_funds(self, funds: List[Dict[str, str]]) -> List[Funding]:
def extract_employment(self, employments: List[Dict[str, str]]) -> Optional[Employment]:
employment = employments[0] if employments else None
return Employment(
id=unique_id(),
organization=employment.get("organization", None),
organization_address=employment.get("organization-address", None),
department=employment.get("department", None),
@@ -138,6 +170,7 @@ def extract_employment(self, employments: List[Dict[str, str]]) -> Optional[Empl
def extract_employments(self, employments: List[Dict[str, str]]) -> List[Employment]:
return [
Employment(
id=unique_id(),
organization=employment.get("organization", None),
department=employment.get("department", None),
role=employment.get("Role", None),
1 change: 1 addition & 0 deletions vis/js/reducers/author.js
Original file line number Diff line number Diff line change
@@ -26,6 +26,7 @@ const author = (state = null, action) => {
funds: action.author?.funds,
educations: action.author?.educations,
memberships: action.author?.memberships,
distinctions: action.author?.distinctions,
};
default:
return state;
4 changes: 2 additions & 2 deletions vis/js/templates/contextfeatures/ResearcherInfo.jsx
Original file line number Diff line number Diff line change
@@ -3,7 +3,7 @@ import { connect } from "react-redux";

import useMatomo from "../../utils/useMatomo";
import { useLocalizationContext } from "../../components/LocalizationProvider";
import { openResearcherMetricsModal } from "../../actions";
import { openResearcherModal } from "../../actions";

const ResearcherInfo = ({ onClick }) => {
const loc = useLocalizationContext();
@@ -36,7 +36,7 @@ const ResearcherInfo = ({ onClick }) => {
};

const mapDispatchToProps = (dispatch) => ({
onClick: () => dispatch(openResearcherMetricsModal()),
onClick: () => dispatch(openResearcherModal()),
});

export default connect(null, mapDispatchToProps)(ResearcherInfo);
2 changes: 1 addition & 1 deletion vis/js/templates/modals/ResearcherInfoModal.jsx
Original file line number Diff line number Diff line change
@@ -30,7 +30,7 @@ const ResearcherInfoModal = ({open, onClose, params, service, isStreamgraph, mod


const mapStateToProps = (state) => ({
open: state.modals.openResearcherMetricsModal,
open: state.modals.openResearcherModal,
params: {
...state.modals.infoParams,
query: state.query.text,
6 changes: 3 additions & 3 deletions vis/js/templates/modals/ResearcherMetricsInfoModal.jsx
Original file line number Diff line number Diff line change
@@ -2,7 +2,7 @@ import React from "react";
import { connect } from "react-redux";
import { Modal } from "react-bootstrap";

import { closeResearcherModal } from "../../actions";
import { closeResearcherMetricsModal } from "../../actions";
import { STREAMGRAPH_MODE } from "../../reducers/chartType";

import ResearcherMetricsInfo from "./researcher-modal/OrcidResearcherMetricsInfo";
@@ -30,7 +30,7 @@ const ResearcherMetricsInfoModal = ({open, onClose, params, service, isStreamgra


const mapStateToProps = (state) => ({
open: state.modals.openResearcherModal,
open: state.modals.openResearcherMetricsModal,
params: {
...state.modals.infoParams,
query: state.query.text,
@@ -56,7 +56,7 @@ const mapStateToProps = (state) => ({
});

const mapDispatchToProps = (dispatch) => ({
onClose: () => dispatch(closeResearcherModal()),
onClose: () => dispatch(closeResearcherMetricsModal()),
});

export default connect(
15 changes: 11 additions & 4 deletions vis/js/templates/modals/researcher-modal/OrcidResearcherInfo.jsx
Original file line number Diff line number Diff line change
@@ -23,7 +23,7 @@ const ResearcherInfo = ({
{education.role} / {education.start_date} - {education.end_date} / {education.organization} / {education.organization_address}
</p>
))}
<h3>GRANTS ({params.funds?.length})</h3>
<h3>GRANTS ({params.funds?.length || 0})</h3>
{params.funds?.map((fund) => (
<p key={fund.id}>
{fund.title} / {fund.start_date} - {fund.end_date} / Funder: {fund.organization} / Amount: {fund.amount?.value} {fund.amount?.currency}
@@ -33,20 +33,26 @@ const ResearcherInfo = ({
<p>
{params.external_identifiers.map((external_id) => (
<p key={external_id["value"]}>
<i className="fas fa-solid fa-link" style={{ paddingRight: '3px'}}></i>
<a className="underline" href={external_id["url"]}>
{external_id["type"]}: {external_id["value"]}
</a>
</p>
))}
</p>
<h3>DISTINCTIONS / AWARDS</h3>
<h3>MEMBERSHIPS</h3>
<h3>DISTINCTIONS / AWARDS ({params.distinctions?.length || 0})</h3>
{params.distinctions?.map((distinction) => (
<p key={distinction.id}>
{distinction.title} / {distinction.start_date} - {distinction.end_date} / {distinction.organization} / {distinction.organization_address}
</p>
))}
<h3>MEMBERSHIPS ({params.memberships?.length || 0})</h3>
{params.memberships?.map((membership) => (
<p key={membership.id}>
{membership.role} / {membership.start_date} - {membership.end_date} / {membership.organization} / {membership.organization_address}
</p>
))}
<h3>PEER REVIEWS</h3>
<h3>PEER REVIEWS ({params.peer_reviews?.length || 0})</h3>
{params.peer_reviews?.map((peer_review) => (
<p key={peer_review.id}>
{peer_review.role} / {peer_review.completion_date} / {peer_review.organization} / {peer_review.organization_address}
@@ -66,6 +72,7 @@ const mapStateToProps = (state) => {
educations: state.author.educations,
employments: state.author.employments,
memberships: state.author.memberships,
distinctions: state.author.distinctions,
},
};
};