diff --git a/.gitignore b/.gitignore index 403288c7..12ce0bcc 100644 --- a/.gitignore +++ b/.gitignore @@ -21,5 +21,12 @@ build/ doi.db output/ transaction_history/ +.eggs/ +docs/Makefile +docs/design/PDS-technical-architecture.pdf +docs/make.bat +doi_temp.db +tests/aaDOI_production_submitted_labels.zip +tests/aaDOI_production_submitted_labels/ diff --git a/0.0.3-dev/REQUIREMENTS.md b/0.0.3-dev/REQUIREMENTS.md deleted file mode 100644 index 06fff3ce..00000000 --- a/0.0.3-dev/REQUIREMENTS.md +++ /dev/null @@ -1,73 +0,0 @@ - -Requirements Summary -==================== - -# DOI management - -## The software shall be capable of accepting a request to create a draft DOI. ([#5](https://github.com/NASA-PDS/pds-doi-service/issues/5)) - - -This requirement is not impacted by the current version -## The software shall be capable of accepting a request to reserve a DOI. ([#6](https://github.com/NASA-PDS/pds-doi-service/issues/6)) - - -This requirement is not impacted by the current version -## The software shall be capable of accepting a request to release a DOI. ([#7](https://github.com/NASA-PDS/pds-doi-service/issues/7)) - - -This requirement is not impacted by the current version -## The software shall be capable of accepting a request to deactivate a DOI. ([#8](https://github.com/NASA-PDS/pds-doi-service/issues/8)) - - -This requirement is not impacted by the current version -## The software shall be capable of accepting a request to update DOI metadata. ([#9](https://github.com/NASA-PDS/pds-doi-service/issues/9)) - - -This requirement is not impacted by the current version -## The software shall be capable of batch processing >1 DOI requests. ([#10](https://github.com/NASA-PDS/pds-doi-service/issues/10)) - - -This requirement is not impacted by the current version -# DOI metadata - -## The software shall be capable of autonomously generating the minimum set of DOI metadata from PDS4 Collection, Bundle, Document products. ([#11](https://github.com/NASA-PDS/pds-doi-service/issues/11)) - - -This requirement is not impacted by the current version -## The software shall validate a minimum set of metadata is provided when reserving, releasing, or updating a DOI. This minimum set of metadata will be defined by the PDS DOI Working Group. ([#12](https://github.com/NASA-PDS/pds-doi-service/issues/12)) :boom: - - -The enhancements which impact this requirements are: -- Develop DOI metadata automated validation component ([#18](https://github.com/NASA-PDS/pds-doi-service/issues/18)) -- Reserve a DOI capability ([#21](https://github.com/NASA-PDS/pds-doi-service/issues/21)) - -## The software shall validate the DOI metadata when reserving, releasing, or updating a DOI. ([#13](https://github.com/NASA-PDS/pds-doi-service/issues/13)) :boom: - - -The enhancements which impact this requirements are: -- Develop DOI metadata automated validation component ([#18](https://github.com/NASA-PDS/pds-doi-service/issues/18)) - -# DOI interface support - -## The software shall maintain a database of PDS DOIs and their current state. ([#14](https://github.com/NASA-PDS/pds-doi-service/issues/14)) - - -This requirement is not impacted by the current version -## The software shall maintain the ability to manage DOIs through OSTI ([#15](https://github.com/NASA-PDS/pds-doi-service/issues/15)) - - -This requirement is not impacted by the current version -## The software shall maintain the ability to manage DOIs through DataCite. ([#16](https://github.com/NASA-PDS/pds-doi-service/issues/16)) - - -This requirement is not impacted by the current version -# DOI-management - -## The software shall provide a Status capability that will allow a user to query for the current status of a DOI ([#30](https://github.com/NASA-PDS/pds-doi-service/issues/30)) - - -This requirement is not impacted by the current version -## The software shall provide the capability of producing a DOI Status Report based upon a user-specified query ([#35](https://github.com/NASA-PDS/pds-doi-service/issues/35)) - - -This requirement is not impacted by the current version \ No newline at end of file diff --git a/0.0.4-dev/REQUIREMENTS.md b/0.0.4-dev/REQUIREMENTS.md deleted file mode 100644 index 06fff3ce..00000000 --- a/0.0.4-dev/REQUIREMENTS.md +++ /dev/null @@ -1,73 +0,0 @@ - -Requirements Summary -==================== - -# DOI management - -## The software shall be capable of accepting a request to create a draft DOI. ([#5](https://github.com/NASA-PDS/pds-doi-service/issues/5)) - - -This requirement is not impacted by the current version -## The software shall be capable of accepting a request to reserve a DOI. ([#6](https://github.com/NASA-PDS/pds-doi-service/issues/6)) - - -This requirement is not impacted by the current version -## The software shall be capable of accepting a request to release a DOI. ([#7](https://github.com/NASA-PDS/pds-doi-service/issues/7)) - - -This requirement is not impacted by the current version -## The software shall be capable of accepting a request to deactivate a DOI. ([#8](https://github.com/NASA-PDS/pds-doi-service/issues/8)) - - -This requirement is not impacted by the current version -## The software shall be capable of accepting a request to update DOI metadata. ([#9](https://github.com/NASA-PDS/pds-doi-service/issues/9)) - - -This requirement is not impacted by the current version -## The software shall be capable of batch processing >1 DOI requests. ([#10](https://github.com/NASA-PDS/pds-doi-service/issues/10)) - - -This requirement is not impacted by the current version -# DOI metadata - -## The software shall be capable of autonomously generating the minimum set of DOI metadata from PDS4 Collection, Bundle, Document products. ([#11](https://github.com/NASA-PDS/pds-doi-service/issues/11)) - - -This requirement is not impacted by the current version -## The software shall validate a minimum set of metadata is provided when reserving, releasing, or updating a DOI. This minimum set of metadata will be defined by the PDS DOI Working Group. ([#12](https://github.com/NASA-PDS/pds-doi-service/issues/12)) :boom: - - -The enhancements which impact this requirements are: -- Develop DOI metadata automated validation component ([#18](https://github.com/NASA-PDS/pds-doi-service/issues/18)) -- Reserve a DOI capability ([#21](https://github.com/NASA-PDS/pds-doi-service/issues/21)) - -## The software shall validate the DOI metadata when reserving, releasing, or updating a DOI. ([#13](https://github.com/NASA-PDS/pds-doi-service/issues/13)) :boom: - - -The enhancements which impact this requirements are: -- Develop DOI metadata automated validation component ([#18](https://github.com/NASA-PDS/pds-doi-service/issues/18)) - -# DOI interface support - -## The software shall maintain a database of PDS DOIs and their current state. ([#14](https://github.com/NASA-PDS/pds-doi-service/issues/14)) - - -This requirement is not impacted by the current version -## The software shall maintain the ability to manage DOIs through OSTI ([#15](https://github.com/NASA-PDS/pds-doi-service/issues/15)) - - -This requirement is not impacted by the current version -## The software shall maintain the ability to manage DOIs through DataCite. ([#16](https://github.com/NASA-PDS/pds-doi-service/issues/16)) - - -This requirement is not impacted by the current version -# DOI-management - -## The software shall provide a Status capability that will allow a user to query for the current status of a DOI ([#30](https://github.com/NASA-PDS/pds-doi-service/issues/30)) - - -This requirement is not impacted by the current version -## The software shall provide the capability of producing a DOI Status Report based upon a user-specified query ([#35](https://github.com/NASA-PDS/pds-doi-service/issues/35)) - - -This requirement is not impacted by the current version \ No newline at end of file diff --git a/0.0.5-dev/REQUIREMENTS.md b/0.0.5-dev/REQUIREMENTS.md deleted file mode 100644 index 06fff3ce..00000000 --- a/0.0.5-dev/REQUIREMENTS.md +++ /dev/null @@ -1,73 +0,0 @@ - -Requirements Summary -==================== - -# DOI management - -## The software shall be capable of accepting a request to create a draft DOI. ([#5](https://github.com/NASA-PDS/pds-doi-service/issues/5)) - - -This requirement is not impacted by the current version -## The software shall be capable of accepting a request to reserve a DOI. ([#6](https://github.com/NASA-PDS/pds-doi-service/issues/6)) - - -This requirement is not impacted by the current version -## The software shall be capable of accepting a request to release a DOI. ([#7](https://github.com/NASA-PDS/pds-doi-service/issues/7)) - - -This requirement is not impacted by the current version -## The software shall be capable of accepting a request to deactivate a DOI. ([#8](https://github.com/NASA-PDS/pds-doi-service/issues/8)) - - -This requirement is not impacted by the current version -## The software shall be capable of accepting a request to update DOI metadata. ([#9](https://github.com/NASA-PDS/pds-doi-service/issues/9)) - - -This requirement is not impacted by the current version -## The software shall be capable of batch processing >1 DOI requests. ([#10](https://github.com/NASA-PDS/pds-doi-service/issues/10)) - - -This requirement is not impacted by the current version -# DOI metadata - -## The software shall be capable of autonomously generating the minimum set of DOI metadata from PDS4 Collection, Bundle, Document products. ([#11](https://github.com/NASA-PDS/pds-doi-service/issues/11)) - - -This requirement is not impacted by the current version -## The software shall validate a minimum set of metadata is provided when reserving, releasing, or updating a DOI. This minimum set of metadata will be defined by the PDS DOI Working Group. ([#12](https://github.com/NASA-PDS/pds-doi-service/issues/12)) :boom: - - -The enhancements which impact this requirements are: -- Develop DOI metadata automated validation component ([#18](https://github.com/NASA-PDS/pds-doi-service/issues/18)) -- Reserve a DOI capability ([#21](https://github.com/NASA-PDS/pds-doi-service/issues/21)) - -## The software shall validate the DOI metadata when reserving, releasing, or updating a DOI. ([#13](https://github.com/NASA-PDS/pds-doi-service/issues/13)) :boom: - - -The enhancements which impact this requirements are: -- Develop DOI metadata automated validation component ([#18](https://github.com/NASA-PDS/pds-doi-service/issues/18)) - -# DOI interface support - -## The software shall maintain a database of PDS DOIs and their current state. ([#14](https://github.com/NASA-PDS/pds-doi-service/issues/14)) - - -This requirement is not impacted by the current version -## The software shall maintain the ability to manage DOIs through OSTI ([#15](https://github.com/NASA-PDS/pds-doi-service/issues/15)) - - -This requirement is not impacted by the current version -## The software shall maintain the ability to manage DOIs through DataCite. ([#16](https://github.com/NASA-PDS/pds-doi-service/issues/16)) - - -This requirement is not impacted by the current version -# DOI-management - -## The software shall provide a Status capability that will allow a user to query for the current status of a DOI ([#30](https://github.com/NASA-PDS/pds-doi-service/issues/30)) - - -This requirement is not impacted by the current version -## The software shall provide the capability of producing a DOI Status Report based upon a user-specified query ([#35](https://github.com/NASA-PDS/pds-doi-service/issues/35)) - - -This requirement is not impacted by the current version \ No newline at end of file diff --git a/README.md b/README.md index b1fc230a..e6c258f5 100644 --- a/README.md +++ b/README.md @@ -63,6 +63,24 @@ You can also run them for a nicer reporting: behave -f allure_behave.formatter:AllureFormatter -o ./allure ./features allure service allure +#### To report to testrail + +Test report can be pushed to testrail: https://cae-testrail.jpl.nasa.gov/testrail/ + +Project: Planetary Data System (PDS) +Test suite: pds-doi-service + +Set you environment: + + export TESTRAIL_USER= + export TESTRAIL_KEY= + +Run the tests: + + behave + +See the results in https://cae-testrail.jpl.nasa.gov/testrail/index.php?/projects/overview/168 + ## Documentation management ### Design : diff --git a/config/DOI_IAD2_reserved_template_20200205-mustache.xml b/config/DOI_IAD2_reserved_template_20200205-mustache.xml index 01294416..ed8b7160 100644 --- a/config/DOI_IAD2_reserved_template_20200205-mustache.xml +++ b/config/DOI_IAD2_reserved_template_20200205-mustache.xml @@ -21,7 +21,7 @@ {{site_url}} {{/site_url}} {{^site_url}} - + N/A {{/site_url}} {{product_type}} {{product_type_specific}} diff --git a/features/create_draft_doi.feature b/features/create_draft_doi.feature index 95055c47..626f4f04 100644 --- a/features/create_draft_doi.feature +++ b/features/create_draft_doi.feature @@ -1,8 +1,9 @@ Feature: create a draft OSTI DOI + @testrail-C372600 @non-regression Scenario Outline: Create a draft DOI from a valid PDS4 label at - Given a valid PDS4 label at + Given a valid input at When create draft DOI for node Then produced osti record is similar to reference osti @@ -14,8 +15,8 @@ Feature: create a draft OSTI DOI | calibration collection | img | https://pds-imaging.jpl.nasa.gov/data/nsyt/insight_cameras/calibration/collection_calibration.xml | OSTI | tests/data/valid_calibcoll_doi.xml | | document collection | img | https://pds-imaging.jpl.nasa.gov/data/nsyt/insight_cameras/document/collection_document.xml | OSTI | tests/data/valid_docucoll_doi.xml | - - Scenario Outline: an invalid PDS4 is submitted for DOI draft input_type + @testrail-C372601 + Scenario Outline: an invalid PDS4 is submitted for DOI draft input_type Given an invalid PDS4 label at When create draft DOI for node Then a reading error report is generated for @@ -25,7 +26,7 @@ Feature: create a draft OSTI DOI | bundle |img | tests/data/invalid_bundle.xml | OSTI | tests/data/draft_error_report.txt | - + @testrail-C372602 Scenario Outline: Verify reference draft transactions match Given reference transactions in When reference record is drafted for node from @@ -33,31 +34,37 @@ Feature: create a draft OSTI DOI Examples: reference draft transactions | transaction_dir | node_value | input_subdir | ref_output_value | - | aaDOI_production_submitted_labels/ATMOS_mpf_irtf_Bundles_20200414 | atm | aaaSubmitted_by_ATMOS_active_2020414 | aaaRegistered_by_EN_active_20200330/DOI_registered_all_records.xml | - | aaDOI_production_submitted_labels/GEO_asurpif_phx_tega_20200428 | geo | aaaSubmitted_by_GEO_active_20200330 | aaaRegistered_by_EN_active_20200330/DOI_registered_all_records.xml | - | aaDOI_production_submitted_labels/GEO_Chand_Bundle_20200713 | geo | aaaSubmitted_by_GEO_active_20200713 | aaRegistered_by_EN/DOI_registered_all_records.xml | - | aaDOI_production_submitted_labels/GEO_Insight_cruise_20200611 | ppi | aaaSubmitted_by_GEO_2020611 | aaRegistered_by_EN/DOI_registered_all_records.xml | - - | aaDOI_production_submitted_labels/GEO_MER_cs_target_20200824 | geo | aaaSubmitted_by_GEO_active_20200824 | aaaRegistered_by_EN_active_20200813/DOI_registered_all_records.xml | - - | aaDOI_production_submitted_labels/GEO_Mer_updated_Bundles_20200629| geo | aaaSubmitted_by_GEO_active_2020629/MER_Bundles_2| aaRegistered_by_EN/DOI_registered_all_records.xml | - | aaDOI_production_submitted_labels/GEO_Messenger_20200327 | geo | aaaSubmitted_by_GEO_active_2020427/messenger_doi | aaaRegistered_by_EN_active_20200427/DOI_registered_all_records.xml | - | aaDOI_production_submitted_labels/GEO_phy_meca_Bundle_20200609 | geo | aaaSubmitted_by_GEO_active_2020609 | aaRegistered_by_EN/DOI_registered_all_records.xml | - | aaDOI_production_submitted_labels/IMG_MRO_Bundle_20200814 | img | aaaSubmitted_by_IMG_active_2020813 | aaaRegistered_by_EN_active_20200813/aaaSave_original/DOI_registered_all_records.xml | - | aaDOI_production_submitted_labels/PPI_InSight_Bundles_Collections_20200812 | ppi | aaaSubmitted_by_PPI_active_20200812 | aaRegistered_by_EN_active/DOI_registered_all_records_corrected.xml | - | aaDOI_production_submitted_labels/GEO_InSight_phx_ra_20200603 | geo | aaaSubmitted_by_GEO_active_2020603/bundle_phx_ra_EDR_raw_edited.xml | aaRegistered_by_EN/DOI-C_registered_bundle_phx_ra_edited.xml | - | aaDOI_production_submitted_labels/IMG_InSight_Bundle_20191216 | img | aaSubmitted_by_Insight/bundle.xml | aaRegistered_by_EN/DOI_bundle.xml | - | aaDOI_production_submitted_labels/IMG_InSight_Bundle_20191216 | img | aaSubmitted_by_Insight/collection_calibration.xml | aaRegistered_by_EN/DOI_collection_calibration.xml | - | aaDOI_production_submitted_labels/PPI_Cassini_RPWS_20200219 | ppi | aaSubmitted/collection_rpws-electron_density_data.xml | aaRegistered_by_EN/DOI_collection_data.xml | - - - -# TO BE ADDED -# Scenario Outline: Draft a DOI which has been previously reserved -# Given historical transaction -# When is reserved -# Then is draft -# Then the reserved and drafts doi have the same id + | tests/aaDOI_production_submitted_labels/ATMOS_mpf_irtf_Bundles_20200414 | atm | aaaSubmitted_by_ATMOS_active_2020414 | aaaRegistered_by_EN_active_20200330/DOI_registered_all_records.xml | + | tests/aaDOI_production_submitted_labels/GEO_asurpif_phx_tega_20200428 | geo | aaaSubmitted_by_GEO_active_20200330 | aaaRegistered_by_EN_active_20200330/DOI_registered_all_records.xml | + | tests/aaDOI_production_submitted_labels/GEO_Chand_Bundle_20200713 | geo | aaaSubmitted_by_GEO_active_20200713 | aaRegistered_by_EN/DOI_registered_all_records.xml | + | tests/aaDOI_production_submitted_labels/GEO_Insight_cruise_20200611 | ppi | aaaSubmitted_by_GEO_2020611 | aaRegistered_by_EN/DOI_registered_all_records.xml | + + | tests/aaDOI_production_submitted_labels/GEO_MER_cs_target_20200824 | geo | aaaSubmitted_by_GEO_active_20200824 | aaaRegistered_by_EN_active_20200813/DOI_registered_all_records.xml | + + | tests/aaDOI_production_submitted_labels/GEO_Mer_updated_Bundles_20200629| geo | aaaSubmitted_by_GEO_active_2020629/MER_Bundles_2| aaRegistered_by_EN/DOI_registered_all_records.xml | + | tests/aaDOI_production_submitted_labels/GEO_Messenger_20200327 | geo | aaaSubmitted_by_GEO_active_2020427/messenger_doi | aaaRegistered_by_EN_active_20200427/DOI_registered_all_records.xml | + | tests/aaDOI_production_submitted_labels/GEO_phy_meca_Bundle_20200609 | geo | aaaSubmitted_by_GEO_active_2020609 | aaRegistered_by_EN/DOI_registered_all_records.xml | + | tests/aaDOI_production_submitted_labels/IMG_MRO_Bundle_20200814 | img | aaaSubmitted_by_IMG_active_2020813 | aaaRegistered_by_EN_active_20200813/aaaSave_original/DOI_registered_all_records.xml | + | tests/aaDOI_production_submitted_labels/PPI_InSight_Bundles_Collections_20200812 | ppi | aaaSubmitted_by_PPI_active_20200812 | aaRegistered_by_EN_active/DOI_registered_all_records_corrected.xml | + | tests/aaDOI_production_submitted_labels/GEO_InSight_phx_ra_20200603 | geo | aaaSubmitted_by_GEO_active_2020603/bundle_phx_ra_EDR_raw_edited.xml | aaRegistered_by_EN/DOI-C_registered_bundle_phx_ra_edited.xml | + | tests/aaDOI_production_submitted_labels/IMG_InSight_Bundle_20191216 | img | aaSubmitted_by_Insight/bundle.xml | aaRegistered_by_EN/DOI_bundle.xml | + | tests/aaDOI_production_submitted_labels/IMG_InSight_Bundle_20191216 | img | aaSubmitted_by_Insight/collection_calibration.xml | aaRegistered_by_EN/DOI_collection_calibration.xml | + | tests/aaDOI_production_submitted_labels/PPI_Cassini_RPWS_20200219 | ppi | aaSubmitted/collection_rpws-electron_density_data.xml | aaRegistered_by_EN/DOI_collection_data.xml | + + + @testrail-C373834 + Scenario Outline: Draft a DOI which has been previously reserved + Given reference transactions in + Given random new lid + When reference record is reserved for node with + And submit osti record for + And reference record is drafted for node from + And submit osti record for + Then lidvid already submitted exception is raised + + Examples: reference reserve transactions + | transaction_dir |node_value | input_reserve | input_pds4 | + | tests/end_to_end | atm | reserve.csv | bundle_pds4.xml | # Some notes: The type of aaDOI_production_submitted_labels/GEO_Insight_cruise_20200611 should be geo diff --git a/features/environment.py b/features/environment.py new file mode 100644 index 00000000..0fe5461e --- /dev/null +++ b/features/environment.py @@ -0,0 +1,41 @@ +import os +import logging +import requests +import shutil +import zipfile +from behave_testrail_reporter import TestrailReporter + +logging.basicConfig(level=logging.DEBUG) +logger = logging.getLogger(__name__) + +def download_file(url): + local_filename = url.split('/')[-1] + tmp_dir = 'tests' + local_filepath = os.path.join(tmp_dir, + local_filename) + + dir_target = local_filepath[:-4] + if not os.path.exists(dir_target): + logger.info(f'Downloading reference test datasets {dir_target}') + with requests.get(url, stream=True) as r: + with open(local_filepath, 'wb') as f: + shutil.copyfileobj(r.raw, f) + + with zipfile.ZipFile(local_filepath, 'r') as zip_ref: + zip_ref.extractall(tmp_dir) + + return local_filepath[:-4] + + +def before_all(context): + + download_file('https://pds.nasa.gov/software/test-data/pds-doi-service/aaDOI_production_submitted_labels.zip') + + current_branch = 'master' # Change this to get the current build branch of your CI system + testrail_reporter = TestrailReporter(current_branch) + context.config.reporters.append(testrail_reporter) + + +#def after_scenario(context): +# if hasattr(context, 'random_lid'): +# del context.random_lid diff --git a/features/reserve_doi.feature b/features/reserve_doi.feature index f35c1ea8..83bec73c 100644 --- a/features/reserve_doi.feature +++ b/features/reserve_doi.feature @@ -1,15 +1,16 @@ Feature: reserve a OSTI DOI + + @testrail-C372603 @non-regression Scenario Outline: Reserve an OSTI DOI with valid PDS4 label - Given a valid PDS4 label at + Given a valid input at When reserve DOI in OSTI format at - Then OSTI DOI label is created at input_value,node_value , - Then The OSTI DOI is submitted to the OSTI server - Examples: Valid PDS4 labels - | input_type | node_value | input_value | - | bundle | img | input/DOI_Reserved_GEO_200318.csv | - + Then produced osti record is similar to reference osti + Examples: Valid reserve spreadsheets + | input_type | node_value | input_value | ref_output_value | + | bundle | img | input/DOI_Reserved_GEO_200318.csv | tests/reserve_ok/output.xml | + @testrail-C372604 Scenario Outline: Reserve an OSTI DOI with an invalid PDS4 label Given an invalid PDS4 label at When reserve DOI in OSTI format at @@ -18,20 +19,21 @@ Feature: reserve a OSTI DOI | input_type | node_value | input_value | error_report | | bundle | img | tests/data/invalid_bundle.xml | tests/data/reserve_error_report.txt | + @testrail-C372605 Scenario Outline: Verify reference reserve transactions match Given reference transactions in - When reference record is reserved with node with + When reference record is reserved for node with Then produced osti record is similar to reference osti Examples: reference reserve transactions | transaction_dir |node_value | input_value | output_value | - | aaDOI_production_submitted_labels | atm | ATMOS_reserve_Insight_Bundle_20200624/aaaSubmitted_by_ATMOS_reserve_2020624/DOI_Requests_ATM-2020-06-30.xlsx | ATMOS_reserve_Insight_Bundle_20200624/aaRegistered_by_EN/DOI_reserved_all_records.xml | - | aaDOI_production_submitted_labels | geo | GEO_APXS_reserve_Bundle_20200710/aaaSubmitted_by_GEO_reserve_20200710/DOI_Reserved_template_20200709.xlsx | GEO_APXS_reserve_Bundle_20200710/aaRegistered_by_EN/DOI_reserved_all_records.xml | - | aaDOI_production_submitted_labels | geo | GEO_Apollo_Bundles_reserve_20200316/aaaSubmitted_by_GEO_reserve_20200316/DOI_GEO_Apollo_Reserved_Bundles_20200316.xlsx | GEO_Apollo_Bundles_reserve_20200316/aaaRegistered_by_EN_reserve_allrecords_20200416/DOI_reserved_all_records.xml | - | aaDOI_production_submitted_labels | geo | GEO_reserve_Bundle_20200706/aaaSubmitted_by_GEO_reserve_20200706/DOI_Reserved_template_20200702.xlsx | GEO_reserve_Bundle_20200706/aaRegistered_by_EN/DOI_reserved_all_records.xml | - | aaDOI_production_submitted_labels | geo | GEO_reserve_Lunar_Space_weather_20200730/aaaSubmitted_by_GEO_reserve_20200730/DOI_Reserved_template_20200729.xlsx | GEO_reserve_Lunar_Space_weather_20200730/aaRegistered_by_EN_reserve/DOI_reserved_all_records.xml | - | aaDOI_production_submitted_labels | rms | RINGS_Jupiter_occs_20200609/aaaSubmitted_by_RINGS_reserve_2020609/DOI_RMS_U-occs-Reserved-2020-06-08.xlsx | RINGS_Jupiter_occs_20200609/aaRegistered_by_EN/DOI_reserved_all_records.xml | + | tests/aaDOI_production_submitted_labels | atm | ATMOS_reserve_Insight_Bundle_20200624/aaaSubmitted_by_ATMOS_reserve_2020624/DOI_Requests_ATM-2020-06-30.xlsx | ATMOS_reserve_Insight_Bundle_20200624/aaRegistered_by_EN/DOI_reserved_all_records.xml | + | tests/aaDOI_production_submitted_labels | geo | GEO_APXS_reserve_Bundle_20200710/aaaSubmitted_by_GEO_reserve_20200710/DOI_Reserved_template_20200709.xlsx | GEO_APXS_reserve_Bundle_20200710/aaRegistered_by_EN/DOI_reserved_all_records.xml | + | tests/aaDOI_production_submitted_labels | geo | GEO_Apollo_Bundles_reserve_20200316/aaaSubmitted_by_GEO_reserve_20200316/DOI_GEO_Apollo_Reserved_Bundles_20200316.xlsx | GEO_Apollo_Bundles_reserve_20200316/aaaRegistered_by_EN_reserve_allrecords_20200416/DOI_reserved_all_records.xml | + | tests/aaDOI_production_submitted_labels | geo | GEO_reserve_Bundle_20200706/aaaSubmitted_by_GEO_reserve_20200706/DOI_Reserved_template_20200702.xlsx | GEO_reserve_Bundle_20200706/aaRegistered_by_EN/DOI_reserved_all_records.xml | + | tests/aaDOI_production_submitted_labels | geo | GEO_reserve_Lunar_Space_weather_20200730/aaaSubmitted_by_GEO_reserve_20200730/DOI_Reserved_template_20200729.xlsx | GEO_reserve_Lunar_Space_weather_20200730/aaRegistered_by_EN_reserve/DOI_reserved_all_records.xml | + | tests/aaDOI_production_submitted_labels | rms | RINGS_Jupiter_occs_20200609/aaaSubmitted_by_RINGS_reserve_2020609/DOI_RMS_U-occs-Reserved-2020-06-08.xlsx | RINGS_Jupiter_occs_20200609/aaRegistered_by_EN/DOI_reserved_all_records.xml | # Future tests: diff --git a/features/steps/steps.py b/features/steps/steps.py index eaadcc25..2b94cb7e 100644 --- a/features/steps/steps.py +++ b/features/steps/steps.py @@ -1,50 +1,57 @@ import os -import sys from behave import * from copy import deepcopy -from datetime import datetime -from enum import Enum from lxml import etree -from io import StringIO +import tempfile +import uuid + import logging -from pds_doi_core.input.exceptions import DuplicatedTitleDOIException, TitleDoesNotMatchProductTypeException, \ - UnexpectedDOIActionException, InputFormatException, WarningDOIException, CriticalDOIException -from pds_doi_core.util.general_util import get_logger +from pds_doi_core.input.exceptions import InputFormatException, CriticalDOIException, IllegalDOIActionException from pds_doi_core.util.doi_xml_differ import DOIDiffer from pds_doi_core.actions.draft import DOICoreActionDraft from pds_doi_core.actions.reserve import DOICoreActionReserve -from pds_doi_core.outputs.osti import DOIOutputOsti +from pds_doi_core.actions.release import DOICoreActionRelease from pds_doi_core.outputs.osti_web_client import DOIOstiWebClient -from pds_doi_core.outputs.osti_web_parser import DOIOstiWebParser from pds_doi_core.util.config_parser import DOIConfigUtil -logging.basicConfig(level=logging.DEBUG) +logging.basicConfig(level=logging.INFO) logger = logging.getLogger(__name__) # Global flag to submit the DOI to OSTI or not after it has been built. g_submit_flag = True g_submit_flag = False -def get_temporary_output_filename(): - return 'temp_doi_label.xml' + +def get_temporary_output_filename(extension='xml'): + return os.path.join(tempfile.gettempdir(), f'{str(uuid.uuid4())}.{extension}') def save_doi_to_temporary_file(doi_label): # Save doi_label to disk so it can be compared to reference in next step. temporary_file_name = get_temporary_output_filename() - temporary_file_ptr = open(temporary_file_name,"w+") - temporary_file_ptr.write(doi_label + "\n") - temporary_file_ptr.close() + with open(temporary_file_name,"w+") as f: + f.write(doi_label + "\n") return temporary_file_name -def draft_action_run(node_value,input_value): +def replace_lidvid_in_file(input_file, lid, extension='csv'): + input_value_with_random_lidvid = get_temporary_output_filename(extension=extension) + with open(input_file, 'r') as f_in: + with open(input_value_with_random_lidvid, 'w') as f_out: + for line in f_in.readlines(): + f_out.write(line.replace('{{random_lid}}', lid)) + return input_value_with_random_lidvid + +def draft_action_run(node_value,input_value, lid=None): # Helper function to 'draft' a given input_value and write the DOI to a temporary file. # This file will be available for other validation functions. db_name = 'doi_temp.db' _action = DOICoreActionDraft(db_name=db_name) logger.info(f"input_value {input_value}") + if lid: + input_value = replace_lidvid_in_file(input_value, lid, extension='xml') + o_doi_label = _action.run(input=input_value, node=node_value, submitter='my_user@my_node.gov',force=True) @@ -52,21 +59,36 @@ def draft_action_run(node_value,input_value): logger.info(f"success input_value {input_value}") return save_doi_to_temporary_file(o_doi_label) -def reserve_action_run(node_value,input_value): + +def reserve_action_run(node_value,input_value, lid=None): # Helper function to 'reserve' a given input_value. logger.info(f"when node_value,input_value {node_value,input_value}") - o_doi_label = None db_name = 'doi_temp.db' _action = DOICoreActionReserve(db_name=db_name) + if lid: + input_value = replace_lidvid_in_file(input_value, lid, extension='csv') + o_doi_label = _action.run( input=input_value, node=node_value, submitter='my_user@my_node.gov', - dry_run=True,force=True) + dry_run=True, force=True) return save_doi_to_temporary_file(o_doi_label) + +def release_action_run(node_value, input_value): + try: + db_name = 'doi_temp.db' + release_action = DOICoreActionRelease(db_name=db_name) + released_doi_str = release_action.run(input=input_value, node=node_value, + submitter='my_user@my_node.gov', force=True) + return save_doi_to_temporary_file(released_doi_str) + except Exception as e: + raise + + def file_output_compare(output_file, ref_output_value): # Function compare two XML files created from 'draft' or 'reserve' actions. # Assumption(s): @@ -77,7 +99,6 @@ def file_output_compare(output_file, ref_output_value): o_fields_differ_list, o_values_differ_list, o_record_index_differ_list = DOIDiffer.doi_xml_differ(ref_output_value, output_file) - logger.info(f'different fields are {o_fields_differ_list}') logger.info(f'o_fields_differ_list {o_fields_differ_list}') logger.info(f'o_values_differ_list {o_values_differ_list}') @@ -87,7 +108,7 @@ def file_output_compare(output_file, ref_output_value): return 1 -@given('a valid PDS4 label at {input_value}') +@given('a valid input at {input_value}') def given_valid_action_input(context, input_value): logger.info(f"given {input_value}") context.input_value = input_value # Don't forget to set the input_value in context to be available for other functions. @@ -97,13 +118,22 @@ def given_invalid_pds4(context, input_value): logger.info(f'an invalid reserve PDS4 label at input_value {input_value}') context.input_value = input_value # Don't forget to set the input_value in context to be available for other functions. +@given('random new lid') +def given_random_new_lid(context): + context.random_lid = f'urn:nasa:pds:{uuid.uuid4()}' + @when('create draft DOI for node {node_value}') def when_create_draft_impl(context, node_value): logger.info(f"when create DOI draft ") logger.info(f"input_value {context.input_value}") try: - context.output_file = draft_action_run(node_value,context.input_value) + if not hasattr(context, 'output_files'): + context.output_files = [] + new_draft_output = draft_action_run(node_value, + context.input_value, + lid=context.random_lid if hasattr(context, 'random_lid') else None) + context.output_files.append(new_draft_output) except CriticalDOIException as e: logger.info(str(e)) @@ -121,7 +151,10 @@ def step_when_reserve_doi_in_osti_format_impl(context, node_value): logger.info(f"when context {context}") logger.info(f"when input_value {input_value}") try: - context.output_file = reserve_action_run(node_value,input_value) + if not hasattr(context, 'output_files'): + context.output_files = [] + new_reserve_file = reserve_action_run(node_value,input_value) + context.output_files.append(new_reserve_file) except InputFormatException as e: # Save the error message to context.exception_msg so the function step_an_error_report_is_generated_impl has something to check logger.info(f"Expecting InputFormatException from input_value {input_value}") @@ -135,13 +168,18 @@ def step_when_reserve_doi_in_osti_format_impl(context, node_value): context.exception_msg = str(e) logger.info(f"context.failed {context.failed}") -@then('OSTI DOI label is created at input_value,node_value {input_value},{node_value}') +@then('OSTI DOI label is created from {input_value} for node {node_value}') def step_then_osti_doi_label_is_created_impl(context,node_value,input_value): logger.info(f"when context {context}") logger.info(f"when input_value {input_value}") try: - context.output_file = reserve_action_run(node_value,input_value) + if not hasattr(context, 'output_files'): + context.output_files = [] + reserve_ouput_file = reserve_action_run(node_value, + input_value, + lid=context.random_lid if hasattr(context, 'random_lid') else None) + context.output_files.append(reserve_ouput_file) except InputFormatException as e: logger.error(e) except CriticalDOIException as e: @@ -154,12 +192,9 @@ def step_doi_label_is_submitted_impl(context): doi_config_util = DOIConfigUtil() m_config = doi_config_util.get_config() - payload_filename = get_temporary_output_filename() - # Fetch the content of payload_filename into memory and change the status from status="reserved_not_submitted" # to status="Reserved". - - payload_doc = etree.parse(payload_filename) + payload_doc = etree.parse(context.output_files[0]) payload_root = payload_doc.getroot() # Make a new root with modified 'status' attribute to 'Reserved' @@ -181,25 +216,59 @@ def step_doi_label_is_submitted_impl(context): logger.info(f"g_submit_flag is False") @when('reference record is drafted for node {node_value} from {input_subdir}') -def when_reference_is_drafted_from_impl(context,node_value,input_subdir): +def when_reference_is_drafted_from_impl(context, node_value, input_subdir): input_dir = os.path.join(context.transaction_dir, input_subdir) - context.output_file = draft_action_run(node_value, input_dir) + if not hasattr(context, 'output_files'): + context.output_files = [] + new_draft_file = draft_action_run(node_value, + input_dir, + lid=context.random_lid if hasattr(context, 'random_lid') else None) + context.output_files.append(new_draft_file) @given('reference transactions in {transaction_dir}') def given_reference_dir_impl(context,transaction_dir): context.transaction_dir = transaction_dir -@when('reference record is reserved with node {node_value} with {input_value}') -def step_reference_is_reserved_at_input_impl(context,node_value,input_value): +@when('reference record is reserved for node {node_value} with {input_value}') +def step_reference_is_reserved_at_input_impl(context, node_value, input_value): transaction_dir = context.transaction_dir input_dir = os.path.join(transaction_dir,input_value) - context.output_file = reserve_action_run(node_value,input_dir) + if not hasattr(context, 'output_files'): + context.output_files = [] + context.output_files.append(reserve_action_run(node_value,input_dir, + lid=context.random_lid if hasattr(context, 'random_lid') else None)) @then('produced osti record is similar to reference osti {ref_output_value}') -def step_produced_osti_record_is_similiar_to_reference_osti_impl(context,ref_output_value): +def step_produced_osti_record_is_similiar_to_reference_osti_impl(context, ref_output_value): if hasattr(context, 'transaction_dir'): ref_output_value = os.path.join(context.transaction_dir, ref_output_value) logger.info(f"context.transaction_dir {context.transaction_dir}") - logger.info(f"context.output_file {context.output_file}") + logger.info(f"context.output_files {context.output_files}") logger.info(f"ref_output_value {ref_output_value}") - file_output_compare(context.output_file, ref_output_value) + file_output_compare(context.output_files[0], ref_output_value) + + +@when('submit osti record for {node_value}') +def submit_osti_record(context, node_value): + try: + context.output_files[-1] = release_action_run(node_value, context.output_files[-1]) + logger.info(f'record in file {context.output_files[-1]} submitted from output index {len(context.output_files)}') + except CriticalDOIException as e: + context.exception_msg = str(e) + +@then('lidvid already submitted exception is raised') +def step_lidvid_already_submitted_exception_is_raised(context): + assert hasattr(context, 'exception_msg') + logger.info(f'grab first created doi from file {context.output_files}') + reserved_xml = etree.parse(context.output_files[0]) + reserved_doi = reserved_xml.xpath('/records/record/doi')[0].text + + excepted_exception_msg = f'There is already a DOI {reserved_doi} submitted for this lidvid {context.random_lid}::1.0 (status=Pending). You cannot submit a new DOI for the same lidvid.' + logger.info(f'expected message {excepted_exception_msg}') + logger.info(f'found msg is {context.exception_msg}') + assert context.exception_msg == excepted_exception_msg + + + + + diff --git a/outputs/ATMOS_reserve_Insight_Bundle_20200624/aaaSubmitted_by_ATMOS_reserve_2020624/DOI_reserved_all_records.xml b/outputs/ATMOS_reserve_Insight_Bundle_20200624/aaaSubmitted_by_ATMOS_reserve_2020624/DOI_reserved_all_records.xml deleted file mode 100644 index bf429a29..00000000 --- a/outputs/ATMOS_reserve_Insight_Bundle_20200624/aaaSubmitted_by_ATMOS_reserve_2020624/DOI_reserved_all_records.xml +++ /dev/null @@ -1,1390 +0,0 @@ - - - - - - - - - - - Cassini Orbiter Radio Science Subsystem Occultation and Electron Density Bundle - - - P.; L. - Withers; Huber - - - - - Atmospheres Node - DataCurator - - - - 2020-08-01 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:corss_occul_el_dens::1.0 - Cites - - - - - - - - - - - - Juno MWR Bundle - - - M.; L. - Janssen; Huber - - - - - Atmospheres Node - DataCurator - - - - 2017-05-19 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:juno_mwr_bundle::1.0 - Cites - - - - - - - - - - - - Mars Atmosphere and Volatile Evolution (MAVEN) Imaging Ultraviolet Spectrograph (IUVS) Raw-level Data Product Bundle - - - J. - Deighan - - - - - Atmospheres Node - DataCurator - - - - 2018-06-28 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:maven.iuvs.raw::13.0 - Cites - - - - - - - - - - - - Viking Orbiters 1 and 2 Radio Occultation Electron Density Bundle - - - P.; L. - Withers; Huber - - - - - Atmospheres Node - DataCurator - - - - 2020-08-01 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:voroelden::1.0 - Cites - - - - - - - - - - - - MAVEN Neutral Gas and Ion Mass Spectrometer Data - - - M.: E. - Benni; Lyness - - - - - Atmospheres Node - DataCurator - - - - 2016-03-15 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:maven_ngims::1.0 - Cites - - - - - - - - - - - - Messenger MASCS 1001 Bundle - - - L. - Huber - - - - - Atmospheres Node - DataCurator - - - - 2019-07-31 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:mess_mascs_1001_bundle::1.0 - Cites - - - - - - - - - - - - Messenger MASCS 2001 Bundle - - - L. - Huber - - - - - Atmospheres Node - DataCurator - - - - 2019-10-14 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:mess_mascs_2001_bundle::1.0 - Cites - - - - - - - - - - - - Messenger MASCS 2101 Bundle - - - L. - Huber - - - - - Atmospheres Node - DataCurator - - - - 2019-02-20 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:mess_mascs_2101_bundle::1.0 - Cites - - - - - - - - - - - - InSight EDL Bundle - - - J. - Ashley - - - - - Atmospheres Node - DataCurator - - - - 2020-08-01 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:insight_edl::1.0 - Cites - - - - - - - - - - - - ODYA Bundle - - - L. - Huber - - - - - Atmospheres Node - DataCurator - - - - 2015-01-16 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:ody_accel_bundle::1.0 - Cites - - - - - - - - - - - - MRO Accel Bundle - - - L. - Huber - - - - - Atmospheres Node - DataCurator - - - - 2016-12-08 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:mro_accel_bundle::1.0 - Cites - - - - - - - - - - - - Juno Gravity Bundle - - - D.; L.; A. - Buccino; Huber; Verma - - - - - Atmospheres Node - DataCurator - - - - 2020-07-22 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:jno.rss.raw.jugr::1.0 - Cites - - - - - - - - - - - - APSS PS Data - - - L. - Mora - - - - - Atmospheres Node - DataCurator - - - - 2019-09-10 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:insight_ps::3.0 - Cites - - - - - - - - - - - - MGS Accel Bundle - - - L. - Huber - - - - - Atmospheres Node - DataCurator - - - - 2015-06-24 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:mgs_accel_bundle::1.0 - Cites - - - - - - - - - - - - Mars Exploration Rover IMU Bundle - - - L. - Huber - - - - - Atmospheres Node - DataCurator - - - - 2015-08-17 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:mer_imu_bundle::1.0 - Cites - - - - - - - - - - - - Titan Ionosphere Bundle - - - A. - Kliore; et al - - - - - Atmospheres Node - DataCurator - - - - 2019-05-07 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:corss_titan_ionosphere::1.0 - Cites - - - - - - - - - - - - LADEE Neutral Mass Spectrometer Data - - - M.: E. - Benni; Lyness - - - - - Atmospheres Node - DataCurator - - - - 2015-11-02 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:ladee_nms::1.0 - Cites - - - - - - - - - - - - MSLEDL Bundle - - - C.; P. - Holstein-Rathlou; Withers - - - - - Atmospheres Node - DataCurator - - - - 2015-08-10 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:msledl::1.0 - Cites - - - - - - - - - - - - Lunar Atmosphere and Dust Environment Explorer (LADEE) Ultraviolet-Visible Spectrometer (UVS) Archive Bundle - - - M. - Shirley - - - - - Atmospheres Node - DataCurator - - - - 2015-09-29 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:ladee_uvs::2.0 - Cites - - - - - - - - - - - - Mars Atmosphere and Volatile Evolution (MAVEN) Imaging Ultraviolet Spectrograph (IUVS) Calibrated-level Data Product Bundle - - - J. - Deighan - - - - - Atmospheres Node - DataCurator - - - - 2018-06-28 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:maven.iuvs.calibrated::13.0 - Cites - - - - - - - - - - - - MAVEN Accelerometer Data - - - R.; R. - Tolson; Lugo - - - - - Atmospheres Node - DataCurator - - - - 2016-03-15 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:maven_accel::1.1 - Cites - - - - - - - - - - - - Viking Lander MET Bundle - - - L. - Huber - - - - - Atmospheres Node - DataCurator - - - - 2019-09-26 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:bundle_vl_met::1.0 - Cites - - - - - - - - - - - - Laboratory Study of Hydrocarbon IR Spectra Bundle - - - P. - Bernath - - - - - Atmospheres Node - DataCurator - - - - 2020-08-01 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:lab.hydrocarbon_spectra::1.0 - Cites - - - - - - - - - - - - APSS TWINS Data - - - L. - Mora - - - - - Atmospheres Node - DataCurator - - - - 2019-09-10 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:insight_twins::3.0 - Cites - - - - - - - - - - - - Juno UVS Bundle - - - B.; L. - Trantham; Huber - - - - - Atmospheres Node - DataCurator - - - - 2017-05-19 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:juno_uvs_bundle::1.0 - Cites - - - - - - - - - - - - Viking Lander FTPD Bundle - - - L. - Huber - - - - - Atmospheres Node - DataCurator - - - - 2019-09-26 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:bundle_vl_ftpd::1.0 - Cites - - - - - - - - - - - - InfraRed Telescope Facility Io Photometry Bundle - - - A.; G. - Davies; Veeder - - - - - Atmospheres Node - DataCurator - - - - 2019-02-26 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:irtf_io_photometry::1.0 - Cites - - - - - - - - - - - - LADEE LUNAR DUST EXPERIMENT - - - C. - Neese - - - - - Atmospheres Node - DataCurator - - - - 2014-09-17 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:ladee_ldex::1.0 - Cites - - - - - - - - - - - - Mars Atmosphere and Volatile Evolution (MAVEN) Imaging Ultraviolet Spectrograph (IUVS) Key Parameter Data Product Bundle - - - J. - Deighan - - - - - Atmospheres Node - DataCurator - - - - 2018-06-28 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:maven.iuvs.kp::13.0 - Cites - - - - - - - - - - - - Mars Atmosphere and Volatile Evolution (MAVEN) Imaging Ultraviolet Spectrograph (IUVS) Derived-level Data Product Bundle - - - J. - Deighan - - - - - Atmospheres Node - DataCurator - - - - 2018-06-28 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:maven.iuvs.derived::13.0 - Cites - - - - - - - - - - - - Lowell Observatory Uranus and Neptune b (472 nm) and y (551 nm) Photometry (1972-2016) Bundle - - - G. - Lockwood - - - - - Atmospheres Node - DataCurator - - - - 2020-04-25 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:lowell_uranus-neptune-photometry::1.0 - Cites - - - - - - - - - - - - Hot Ion Atmospheric Escape Data - - - R.; H. - Johnson; Williamson - - - - - Atmospheres Node - DataCurator - - - - 2020-05-27 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:hot_ion_atmos_escape::1.0 - Cites - - - - - - - - - - - - Wind Tunnel Threshold Speed Bundle - - - D.; E.; S. - Burr; Nield; Sutton - - - - - Atmospheres Node - DataCurator - - - - 2020-03-25 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:wt_threshold_speed::1.0 - Cites - - - - - - - - - - - - Cassini Orbiter RSS Neutral Atmosphere T-P profiles for Titan Bundle - - - P. - Schinder - - - - - Atmospheres Node - DataCurator - - - - 2020-01-04 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:corsstpp::2.0 - Cites - - - - - - - - - - - - Saturn Ionosphere Bundle - - - A. - Kliore; et al - - - - - Atmospheres Node - DataCurator - - - - 2019-05-07 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:corss_saturn_ionosphere::1.0 - Cites - - - - - - - - - - - - Saturn Zonal Winds Bundle - - - E.; A. - Garcia-Melendo; Sanchez-Lavega - - - - - Atmospheres Node - DataCurator - - - - 2020-03-10 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:coiss_zonal_winds::1.0 - Cites - - - - - - - - - - - - Mars Global Surveyor Thermal Emission Spectrometer Atmospheric Recalibration Bundle - - - A. - Pankine - - - - - Atmospheres Node - DataCurator - - - - 2020-08-01 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:mgs_tes_recalib_atmos::1.0 - Cites - - - - - - - - - - - - Mars Atmosphere and Volatile Evolution (MAVEN) Imaging Ultraviolet Spectrograph (IUVS) Processed-level Data Product Bundle - - - J. - Deighan - - - - - Atmospheres Node - DataCurator - - - - 2018-06-28 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:maven.iuvs.processed::13.0 - Cites - - - - - - - - - - - - C2H4 mole fraction and temperature profiles after the 2010 Saturn Storm - - - B. - Hesman - - - - - Atmospheres Node - DataCurator - - - - 2020-03-27 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:cocirs_c2h4abund::1.0 - Cites - - - - - - - - - - - - IMP Opacities Bundle - - - P.; M.; R. - Smith; Lemmon; Beebe - - - - - Atmospheres Node - DataCurator - - - - 2015-04-03 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:mpf_mpim::1.0 - Cites - - - - - - - - - - - - Juno JIRAM Bundle - - - R.; L. - Noschese; Huber - - - - - Atmospheres Node - DataCurator - - - - 2020-04-29 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:juno_jiram_bundle::1.0 - Cites - - - - - - - - - - - - Mars Pathfinder ASI/MET Bundle - - - L. - Huber - - - - - Atmospheres Node - DataCurator - - - - 2019-02-01 - Collection - PDS4 Bundle - - - URL - urn:nasa:pds:bundle_mpf_asimet::1.0 - Cites - - - - - diff --git a/pds_doi_core/actions/release.py b/pds_doi_core/actions/release.py index da6e848e..ae691d5e 100644 --- a/pds_doi_core/actions/release.py +++ b/pds_doi_core/actions/release.py @@ -94,7 +94,7 @@ def _raise_warn_exceptions(self, exception_classes, exception_messages): message_to_raise = message_to_raise + ', ' + exception_classes[ii] + ':' + exception_messages[ii] raise WarningDOIException(message_to_raise) - def _validate_doi(self, doi_label): + def _validate_doi(self, doi_label, force=False): """ Before submitting the user input, it has to be validated against the database so a Doi object need to be built. Since the format of o_doi_label is same as a response from OSTI, the same parser can be used. @@ -104,24 +104,24 @@ def _validate_doi(self, doi_label): exception_classes = [] exception_messages = [] try: + dois = DOIOstiWebParser().response_get_parse_osti_xml(doi_label) for doi in dois: - doi.status = 'Registered' # Add 'status' field so the ranking in the workflow can be determined. - single_doi_label = DOIOutputOsti().create_osti_doi_release_record(doi) - if self._config.get('OTHER', 'release_validate_against_xsd_flag').lower() == 'true': - self._doi_validator.validate_against_xsd(single_doi_label) - - # Validate the label to ensure that no rules are violated. - # Put validate_osti_submission() within a try/except clause to catch all the exceptions instead of - # exiting after the first exception. This allow the user to see all the things that are wrong with - # all the DOIs instead of just the first one. try: + doi.status = 'Registered' # Add 'status' field so the ranking in the workflow can be determined. + single_doi_label = DOIOutputOsti().create_osti_doi_release_record(doi) + if self._config.get('OTHER', 'release_validate_against_xsd_flag').lower() == 'true': + self._doi_validator.validate_against_xsd(single_doi_label) + self._doi_validator.validate_osti_submission(doi) + except (DuplicatedTitleDOIException, UnexpectedDOIActionException, - TitleDoesNotMatchProductTypeException, SiteURNotExistException, WarningDOIException) as e: - (exception_classes, exception_messages) = self._collect_exception_classes_and_messages(e, - exception_classes, exception_messages) + TitleDoesNotMatchProductTypeException, SiteURNotExistException) as e: + (exception_classes, exception_messages) = \ + self._collect_exception_classes_and_messages(e, + exception_classes, + exception_messages) except Exception as e: raise # Re-raise all exceptions. diff --git a/pds_doi_core/actions/reserve.py b/pds_doi_core/actions/reserve.py index fc6a6ade..816a02e3 100644 --- a/pds_doi_core/actions/reserve.py +++ b/pds_doi_core/actions/reserve.py @@ -175,9 +175,10 @@ def complete_and_validate_dois(self, dois, contributor, publisher, dry_run): dois_out = [] for doi in dois: if dry_run: - self._doi_validator.validate_osti_submission(doi) - else: self._doi_validator.validate(doi) + else: + self._doi_validator.validate_osti_submission(doi) + dois_out.append(doi) return dois_out @@ -247,7 +248,6 @@ def run(self, **kwargs): except (UnknownNodeException, InputFormatException) as e: raise CriticalDOIException(e) - o_doi_label = DOIOutputOsti().create_osti_doi_reserved_record(dois) if not self._dry_run: diff --git a/pds_doi_core/input/input_util.py b/pds_doi_core/input/input_util.py index 80fdb2d2..d0bc1753 100644 --- a/pds_doi_core/input/input_util.py +++ b/pds_doi_core/input/input_util.py @@ -24,9 +24,6 @@ def parse_sxls_file(self, i_filepath): logger.info("i_filepath" + " " + i_filepath) - doi_directory_pathname = os.path.join('.', 'output') - os.makedirs(doi_directory_pathname, exist_ok=True) - xl_wb = pd.ExcelFile(i_filepath) actual_sheet_name = xl_wb.sheet_names[0] # We only want the first sheet. xl_sheet = pd.read_excel(i_filepath, actual_sheet_name, converters={'publication_date': str, @@ -89,9 +86,6 @@ def parse_csv_file(self, i_filepath): logger.info("i_filepath" + " " + i_filepath) - doi_directory_pathname = os.path.join('.', 'output') - os.makedirs(doi_directory_pathname, exist_ok=True) - # Read the CSV file into memory. csv_sheet = pd.read_csv(i_filepath) diff --git a/pds_doi_core/util/doi_validator.py b/pds_doi_core/util/doi_validator.py index b1d0fb36..71199b78 100644 --- a/pds_doi_core/util/doi_validator.py +++ b/pds_doi_core/util/doi_validator.py @@ -225,6 +225,7 @@ def validate(self, doi: Doi): return 1 def validate_osti_submission(self, doi: Doi): - - self.validate(doi) + # do first the critical error check self._check_field_lidvid_update(doi) + self.validate(doi) + diff --git a/pds_doi_core/util/general_util.py b/pds_doi_core/util/general_util.py index 5b9f9506..f251febb 100644 --- a/pds_doi_core/util/general_util.py +++ b/pds_doi_core/util/general_util.py @@ -20,6 +20,6 @@ def get_logger(module_name=''): logging.basicConfig(format=my_format, filemode='a') - logger.setLevel(logging.DEBUG) + logger.setLevel(logging.INFO) return logger diff --git a/testrail.yml b/testrail.yml new file mode 100644 index 00000000..b50087e1 --- /dev/null +++ b/testrail.yml @@ -0,0 +1,7 @@ +base_url: https://cae-testrail.jpl.nasa.gov/testrail +projects: + - + name: 'Test run {branch}' + id: 168 + suite_id: 4412 + allowed_branch_pattern: '^(master|release\/\d+([\.\d]+)?)$' diff --git a/tests/end_to_end/bundle_pds4.xml b/tests/end_to_end/bundle_pds4.xml new file mode 100644 index 00000000..580dbd25 --- /dev/null +++ b/tests/end_to_end/bundle_pds4.xml @@ -0,0 +1,124 @@ + + +{{random_lid}} +1.0 +InSight Cameras Bundle +1.11.1.0 +Product_Bundle + +R. Deen, H. Abarca, P. Zamani, J.Maki + + + +Smith, P. H.; Lemmon, M.; Beebe, R. F. + + + +2019 + +InSight Cameras Experiment Data Record (EDR) and Reduced Data Record (RDR) Data Products + + + + +Observational Intent + +2019-10-02T04:36:06.728Z +2020-01-01T14:57:01.214Z + + +Science +Raw + +Visible +Surface +Imaging + + + +Insight +Mission + +urn:nasa:pds:context:investigation:mission.insight +bundle_to_investigation + + + + +Insight Lander +Spacecraft + + +urn:nasa:pds:context:instrument_host:spacecraft.insight + +is_instrument_host +Reference to the Insight spacecraft. + + + +Insight Context Camera +Instrument + +urn:nasa:pds:context:instrument:icc.insight +is_instrument + +Reference to the InSight Context Camera instrument onboard the InSight spacecraft. + + + + +Insight Deployment Camera +Instrument + +urn:nasa:pds:context:instrument:idc.insight +is_instrument + +Reference to the InSight Deployment Camera instrument onboard the InsSight spacecraft. + + + + + +Mars +Planet + +urn:nasa:pds:context:target:planet.mars +document_to_target +Reference to the Planet - Mars target + + + + +Archive +This Bundle contains InSight camera data. + + +urn:nasa:pds:insight_cameras:browse +Primary +bundle_has_browse_collection + + +urn:nasa:pds:insight_cameras:calibration +Primary +bundle_has_calibration_collection + + +urn:nasa:pds:insight_cameras:data +Primary +bundle_has_data_collection + + + + +urn:nasa:pds:insight_cameras:miscellaneous +Primary +bundle_has_document_collection + + +urn:nasa:pds:insight_cameras:xml_schema +Primary +bundle_has_schema_collection + + diff --git a/tests/end_to_end/reserve.csv b/tests/end_to_end/reserve.csv new file mode 100644 index 00000000..34cfafeb --- /dev/null +++ b/tests/end_to_end/reserve.csv @@ -0,0 +1,2 @@ +status,title,publication_date,product_type_specific,author_last_name,author_first_name,related_resource +Reserved,Laboratory Shocked Feldspars Collection #1,2020-03-11,PDS4 Collection,Johnson,J. R.,{{random_lid}}::1.0 \ No newline at end of file diff --git a/tests/reserve_ok/output.xml b/tests/reserve_ok/output.xml new file mode 100644 index 00000000..2e28219b --- /dev/null +++ b/tests/reserve_ok/output.xml @@ -0,0 +1,93 @@ + + + + + Laboratory Shocked Feldspars Collection #1 + National Aeronautics and Space Administration (NASA) + urn:nasa:pds:lab_shocked_feldspars::2.0 + NASA Planetary Data System + + 2020-03-11 + N/A + Collection + PDS4 Collection + 2020-10-01 + PDS; PDS4; + + + Johnson + J. R. + + + + + Planetary Data System: Cartography and Imaging Sciences Discipline Node + DataCurator + + + PDS Operator + PDS + pds-operator@jpl.nasa.gov + 818.393.7165 + + + + Laboratory Shocked Feldspars Collection #2 + National Aeronautics and Space Administration (NASA) + urn:nasa:pds:lab_shocked_feldspars_2::2.0 + NASA Planetary Data System + + 2020-03-12 + N/A + Collection + PDS4 Collection + 2020-10-01 + PDS; PDS4; + + + Johnson_2 + J2. R2. + + + + + Planetary Data System: Cartography and Imaging Sciences Discipline Node + DataCurator + + + PDS Operator + PDS + pds-operator@jpl.nasa.gov + 818.393.7165 + + + + Laboratory Shocked Feldspars Collection #3 + National Aeronautics and Space Administration (NASA) + urn:nasa:pds:lab_shocked_feldspars_3::2.0 + NASA Planetary Data System + + 2020-03-12 + N/A + Collection + PDS4 Collection + 2020-10-01 + PDS; PDS4; + + + Johnson_3 + J3. R3. + + + + + Planetary Data System: Cartography and Imaging Sciences Discipline Node + DataCurator + + + PDS Operator + PDS + pds-operator@jpl.nasa.gov + 818.393.7165 + + \ No newline at end of file