-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
pds-api-125: get the XML MIME types working again #20
Conversation
Generated a patch from NASA-PDS/registry-api-service branch pds-api-125 of the code changes along the branch thus far. The file locations in the new repository have changed so used emacs to replace all /src/main/ with /api/src/main/. With a functional patch, used git to apply it in this repository to create the branch pds-api-125.
Something is still wrong with this repo and my permissions. If I do the git fetch and git merge for main, it says up to date. However this PR seems to think there are conflicts but it seems to be more about my permissions to not write or merge to main via the PR than actual conflicts. |
Hi @al-niessner , Di you also make |
Yup, that worked and nothing in eclipse works again. Another day lost. |
service/src/main/java/gov/nasa/pds/api/engineering/serializer/PdsProductXMLSerializer.java
Outdated
Show resolved
Hide resolved
service/src/main/java/gov/nasa/pds/api/engineering/serializer/PdsProductsXMLSerializer.java
Outdated
Show resolved
Hide resolved
service/src/main/java/gov/nasa/pds/api/engineering/serializer/Pds4XmlProductSerializer.java
Outdated
Show resolved
Hide resolved
service/src/main/java/gov/nasa/pds/api/engineering/serializer/PdsProductXMLSerializer.java
Outdated
Show resolved
Hide resolved
service/src/main/java/gov/nasa/pds/api/engineering/serializer/PdsProductsXMLSerializer.java
Outdated
Show resolved
Hide resolved
Okay, back to where I was. At least it was quicker. Failing to map pds4+xml again. Thought it would be a problem with the model (previously known as pds-api-javalib) as in the past so I upped the version in model/pom.xml to 0.5.1-SNAPSHOT and then made services/pom.xml match:
I thought that since I was doing 'mvn clean install' at registry-api repo root that it would use the fresh build. Instead I am seeing this:
What is/was the idea for building all these items as one consistent set? |
Tried doing 'mvn clean install' in the registry-api/model directory followed by same command in registry-api and have same failure. Interestingly it builds the correct artifacts and installs them, I presume locally, but then does not use them:
|
Hi @al-niessner sorry about these issues. I will try to reproduce that tomorrow in the morning. |
@tloubrieu-jpl Thanks, but I think I have it. Just catching some of the looser ends. |
To have an independent build, the artifacts in service pom have to match those in the other directories: lexer and model. Once all the artifacts were referenced correctly, the build is independent of what is deployed at the maven archives so that what is developed is consistent.
<artifactId>api</artifactId> | ||
<version>0.5.0-SNAPSHOT</version> | ||
<artifactId>registry-api-model</artifactId> | ||
<version>0.5.0-SNAPSHOT</version> | ||
</dependency> | ||
|
||
<dependency> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Critical OSS Vulnerability:
pkg:maven/gov.nasa.pds/[email protected]
4 Critical, 0 Severe, 2 Moderate, 0 Unknown vulnerabilities have been found across 2 dependencies
Components
- The configuration file specifies an allowed third-party
JNDI
lookup host for theJMSAppender
- the
javax.jms.*
API is included in the application'sCLASSPATH
- Restrict LDAP access via JNDI apache/logging-log4j2#608 (comment)
- https://logging.apache.org/log4j/1.2/
- The configuration file specifies an allowed third-party
JNDI
lookup host for theJMSAppender
- the
javax.jms.*
API is included in the application'sCLASSPATH
- Restrict LDAP access via JNDI apache/logging-log4j2#608 (comment)
- https://logging.apache.org/log4j/1.2/
pkg:maven/log4j/[email protected]
CRITICAL Vulnerabilities (2)
CVE-2019-17571
[CVE-2019-17571] Included in Log4j 1.2 is a SocketServer class that is vulnerable to deserializat...
Included in Log4j 1.2 is a SocketServer class that is vulnerable to deserialization of untrusted data which can be exploited to remotely execute arbitrary code when combined with a deserialization gadget when listening to untrusted network traffic for log data. This affects Log4j versions up to 1.2 up to 1.2.17.
CVSS Score: 9.8
CVSS Vector: CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H
CVE-2021-4104
[CVE-2021-4104] JMSAppender in Log4j 1.2 is vulnerable to deserialization of untrusted data when...
JMSAppender in Log4j 1.2 is vulnerable to deserialization of untrusted data when the attacker has write access to the Log4j configuration. The attacker can provide TopicBindingName and TopicConnectionFactoryBindingName configurations causing JMSAppender to perform JNDI requests that result in remote code execution in a similar fashion to CVE-2021-44228. Note this issue only affects Log4j 1.2 when specifically configured to use JMSAppender, which is not the default. Apache Log4j 1.2 reached end of life in August 2015. Users should upgrade to Log4j 2 as it addresses numerous other issues from the previous versions.
===================================================
The following information is provided by Sonatype Nexus Intelligence. Nexus Intelligence is the only security research service that performs "secondary expansion" to determine if newly discovered vulnerabilities are also present in other components.
Learn more about Nexus Intelligence -- https://www.sonatype.com/products/intelligence
Explanation
The log4j:log4j
package is vulnerable to Deserialization of Untrusted Data. The lookup()
and activateOptions()
methods in the JMSAppender
class allow JNDI
lookup requests to be made when the TopicBindingName
and TopicConnectionFactoryBindingName
specify a trusted host. Lookups made to this host may be used by attackers to request a serialized malicious Java Object that can be deserialized and executed, leading to Remote Code Execution (RCE).
Note that this vulnerability is different from CVE-2021-44228 and requires the attacker to be in control of the third party host that is specified in the configuration, or write access to the Log4j configuration file in order to specify a malicious lookup host directly. This vulnerability also only affects the 1.x.x component of Log4j
released under the log4j:log4j
group and artifact IDs.
Advisory Deviation Notice: The Sonatype security research team discovered that the root cause of the vulnerability is in all versions of log4j:log4j, not just in the 1.2.x branch as the advisory states.
Detection
The application is vulnerable by using this component under the following circumstances:
Reference: https://bugzilla.redhat.com/show_bug.cgi?id=2031667#c28
Recommendation
The 1.x.x component has reach End of Life
, and users should upgrade to a non-vulnerable version of org.apache.logging.log4j:log4j-core
as this component includes other security vulnerabilities that are not fixed.
References:
CVSS Score: 8.1
CVSS Vector: CVSS:3.0/AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:H/A:H
MODERATE Vulnerabilities (1)
[CVE-2020-9488] Improper validation of certificate with host mismatch in Apache Log4j SMTP appen...
Improper validation of certificate with host mismatch in Apache Log4j SMTP appender. This could allow an SMTPS connection to be intercepted by a man-in-the-middle attack which could leak any log messages sent through that appender.
CVSS Score: 3.7
CVSS Vector: CVSS:3.0/AV:N/AC:H/PR:N/UI:N/S:U/C:L/I:N/A:N
pkg:maven/log4j/[email protected]
CRITICAL Vulnerabilities (2)
CVE-2019-17571
[CVE-2019-17571] Included in Log4j 1.2 is a SocketServer class that is vulnerable to deserializat...
Included in Log4j 1.2 is a SocketServer class that is vulnerable to deserialization of untrusted data which can be exploited to remotely execute arbitrary code when combined with a deserialization gadget when listening to untrusted network traffic for log data. This affects Log4j versions up to 1.2 up to 1.2.17.
CVSS Score: 9.8
CVSS Vector: CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:H/I:H/A:H
CVE-2021-4104
[CVE-2021-4104] JMSAppender in Log4j 1.2 is vulnerable to deserialization of untrusted data when...
JMSAppender in Log4j 1.2 is vulnerable to deserialization of untrusted data when the attacker has write access to the Log4j configuration. The attacker can provide TopicBindingName and TopicConnectionFactoryBindingName configurations causing JMSAppender to perform JNDI requests that result in remote code execution in a similar fashion to CVE-2021-44228. Note this issue only affects Log4j 1.2 when specifically configured to use JMSAppender, which is not the default. Apache Log4j 1.2 reached end of life in August 2015. Users should upgrade to Log4j 2 as it addresses numerous other issues from the previous versions.
===================================================
The following information is provided by Sonatype Nexus Intelligence. Nexus Intelligence is the only security research service that performs "secondary expansion" to determine if newly discovered vulnerabilities are also present in other components.
Learn more about Nexus Intelligence -- https://www.sonatype.com/products/intelligence
Explanation
The log4j:log4j
package is vulnerable to Deserialization of Untrusted Data. The lookup()
and activateOptions()
methods in the JMSAppender
class allow JNDI
lookup requests to be made when the TopicBindingName
and TopicConnectionFactoryBindingName
specify a trusted host. Lookups made to this host may be used by attackers to request a serialized malicious Java Object that can be deserialized and executed, leading to Remote Code Execution (RCE).
Note that this vulnerability is different from CVE-2021-44228 and requires the attacker to be in control of the third party host that is specified in the configuration, or write access to the Log4j configuration file in order to specify a malicious lookup host directly. This vulnerability also only affects the 1.x.x component of Log4j
released under the log4j:log4j
group and artifact IDs.
Advisory Deviation Notice: The Sonatype security research team discovered that the root cause of the vulnerability is in all versions of log4j:log4j, not just in the 1.2.x branch as the advisory states.
Detection
The application is vulnerable by using this component under the following circumstances:
Reference: https://bugzilla.redhat.com/show_bug.cgi?id=2031667#c28
Recommendation
The 1.x.x component has reach End of Life
, and users should upgrade to a non-vulnerable version of org.apache.logging.log4j:log4j-core
as this component includes other security vulnerabilities that are not fixed.
References:
CVSS Score: 8.1
CVSS Vector: CVSS:3.0/AV:N/AC:H/PR:N/UI:N/S:U/C:H/I:H/A:H
MODERATE Vulnerabilities (1)
[CVE-2020-9488] Improper validation of certificate with host mismatch in Apache Log4j SMTP appen...
Improper validation of certificate with host mismatch in Apache Log4j SMTP appender. This could allow an SMTPS connection to be intercepted by a man-in-the-middle attack which could leak any log messages sent through that appender.
CVSS Score: 3.7
CVSS Vector: CVSS:3.0/AV:N/AC:H/PR:N/UI:N/S:U/C:L/I:N/A:N
Hi @al-niessner , thanks for solving that. I tried anyway this morning to have a 'stable' method to import. So I did in eclipse:
And I think I have all errors removed. I will write that in the README. |
Reviewing this data again because it has been bugging me what PDS-4 blob needs to be decoded when it seems none are found in my searches. It seems from reviewing NASA-PDS/pds-api#18 that @jordanpadams is not getting his wish about never seeing |
@al-niessner those comments were a bit dated, but basically, I don't want to ever see per the blob not being found in your searches, not sure. i don't have this deployed locally so I cannot reproduce. at least from the version on pds-gamma, I do see the json_blob for the JSON response |
Should |
@al-niessner I am fairly certain the blob should only be decoded for |
for the remaining response types, |
Turns out there is a lot of duplicate code that wanted to do the same thing but had different behaviors and end results. This really made testing and comparing odd because it would fail for one user but not another mostly because of endpoints used when there should have been no difference. This change cleans most of that up with respect to getting data and using the fields to include and those to exclude. The blob has successfully been removed once again.
First, there are now two blob possibilities and they are encoded. Added the second blob type and adjusted the searching to handle either or both or neither. The Pds4Prduct JSON serializer did the decoding of the blob. Moved it from the single serializer to the Pds4ProductFactory where all of the creation happens. Now the blob is decoded for PDS4 only but for any PDS4 output format.
All four issues at the top of this PR are now fixed. PDS4 outputs for JSON and XML both return the decoded JSON block now. Both blobs are now being filtered out by default but if explicitly specified will show up encoded in any other output structure. All XML output work as well. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @al-niessner ,
Thanks for the update.
When I do the following request:
curl --location --request A 'http://localhost:8080/products' \
--header 'Accept: application/json'
I am getting this error:
{
"timestamp": 1643069228942,
"status": 405,
"error": "Method Not Allowed",
"message": "",
"path": "/products"
}
The request
curl --location --request GET 'http://localhost:8080/bundles' \
--header 'Accept: application/kvp+json'
and same with 'text/csv'
Give 500 error (I think that is only when fields is not specified)
I did not have that before.
Regarding the application/pds4+xml output, there is no xml namespace definition available starting from tag Pds4Product
. Do you know why it is ?
At last, the decoded part of the XML label looks weird (more json than xml):
<pds4>{"Product_Observational":{"Identification_Area":{"product_class":"Product_Observational","Modification_History":{"Modification_Detail":{"modification_date":"2020-06-02","description":"Release 5","version_id":1}},"information_model_version":"1.10.1.0","logical_identifier":"urn:nasa:pds:insight_rad:data_raw:hp3_rad_raw_00390_20200101_120222","version_id":1,"title":"InSight HP3 Radiometer Experiment Raw Product:hp3_rad_raw_00390_20200101_120222"},"xmlns":"http://pds.nasa.gov/pds4/pds/v1","Reference_List":{"Internal_Reference":[{"lid_reference":"urn:nasa:pds:insight_documents:document_mission","reference_type":"data_to_document"},{"lid_reference":"urn:nasa:pds:insight_documents:document_hp3rad:hp3_rad_sis","reference_type":"data_to_document"}]},"xsi:schemaLocation":"http://pds.nasa.gov/pds4/pds/v1 \r\n https://pds.nasa.gov/pds4/pds/v1/PDS4_PDS_1A10.xsd\r\n \r\n http://pds.nasa.gov/pds4/mission/insight/v1 \r\n https://pds.nasa.gov/pds4/mission/insight/v1
...
With regards to the "method not allowed" and 500 error with some output types, that is because `api-0.5.0-SNAPSHOT' is being used instead of the model that comes with this repository. Took me a while to get the POMs right to make those errors go away. |
With regards to PDS4 blob. At the breakout it was decode the blob. That is what the blob looks like decoded -- a JSON object. When doing XML should the other blob be decoded or should the JSON object be mapped to XML? |
With regards to build being broken, maybe not. Try doing those searches with |
All fields results should be fixed. The elastic search error was from searching for Sorry for the mistake, but my searches were with |
@al-niessner I still have this error, I am not understanding why... i will try |
The command
I am building the applcation by running, at the root of the project:
Then in
Any idea of what I should do ? Thanks |
You dependency tree, I am not seeing registry-api-service. Did not cut-n-paste it or something more nefarious? |
What curl(s) is failing? All the products without fields are now working for me or were. I am double checking now. |
To have an up-to-date environment, I do the following:
There is a lot of output missing, but these two commands let me do all that I need to get done. I also have an elasticsearch docker running too but you need one of those too. |
Thanks @al-niessner , I had a typo in one of my requests which led me to think the other request would not work either. Let me retest them all, that should work. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The pr is good to merge. We'll work on the pds4+xml format in a new ticket
🗒️ Summary
Reworked the XML serialization.
⚙️ Test Data and/or Report
♻️ Related Issues
#456
#440
NASA-PDS/pds-api#18
NASA-PDS/pds-api#73