issue [pds-api-54]: wildcard searching #15

al-niessner · 2021-04-07T22:10:22Z

up to date

Merge from parent repo

Updated the pom to use the correct version of ANTLR. Updated ElasticSearchRegistrySearchRequestBuilder to make log output clearer for isolating current problems. Reworked Antlr4SearchListener to get the wildcard request right. Gave it state and a stack which it processes more or less uniformly. Groups are the same as the parent such that if there are no () there are no sub boolean query groups. Then added the groups which add the current stack level onto the group and then shift up one when the groups ends. The and/or expressions similarly handle the conjunction in use the same way. However their stack levels need be the same. This allows (x) to be processed the same as (x and y and (w or u or v)) without and special checking or branching. Antlr4SearchListenerTest was added to verify that the stack works correctly with respect to tranforming the grammar to a set of QueryBuilder(s). The only one being tested currently is the wildcard but many more should be added.

al-niessner · 2021-04-07T22:25:58Z

@tloubrieu-jpl

I have to start the testing but the wildcard testing works and I will then work through the rest to see if it comes together correctly. I will also add the fail tests for the acceptance criteria back on the original ticket. However, this will allow you to see the changes in the code.

al-niessner · 2021-04-08T14:04:50Z

@tloubrieu-jpl @jordanpadams

Behavior:

lid eq *pdart14_meap* - works
(lid eq *pdart14_meap*) - fails grammar with "line 1:0 mismatched input '(lid' expecting {'not', '(', FIELD}"
( lid eq *pdart14_meap* ) - works

Questions:

Do you want to force the user to include the spaces which will be %20 on the browser?
Is this a problem with grammar definition or something that I am missing?

Fix erroneous stack handling in Antlr4SearchListener. Fix up the second test to show that the group does work.

jordanpadams · 2021-04-08T18:24:27Z

@tloubrieu-jpl ☝️

tloubrieu-jpl · 2021-04-09T02:11:18Z

@tloubrieu-jpl @jordanpadams

Behavior:

lid eq *pdart14_meap* - works
(lid eq *pdart14_meap*) - fails grammar with "line 1:0 mismatched input '(lid' expecting {'not', '(', FIELD}"
( lid eq *pdart14_meap* ) - works

Questions:

Do you want to force the user to include the spaces which will be %20 on the browser?

Is this a problem with grammar definition or something that I am missing?

@al-niessner How does that work with " around the string value with wildcard ? That might be unrelated but the grammar was able to parse it before. There was no complaint on the API side although you know that does not mean the request was actually understood, but I guess the grammar parsed it since there is no error in the previous version of the code.

Tested the bulk of the grammar but need to do nested groups and find a way to verify the and/or statements are correctly stacked.

al-niessner · 2021-04-13T19:29:58Z

Actually, the previous code has same problem but it fails silently in the sense that the parser does not throw an exception but does not alter the boolean query object either. Unless the actual content of the search is verified, the request likely succeeded from an http perspective but might have all the wrong content.

al-niessner · 2021-04-13T19:48:12Z

@tloubrieu-jpl @jordanpadams

Spaces are required after ( and before ) as defined in the 'group' parser rule. If we want to get rid of them, then going to have to do a big rework of the grammar. I think we can change queryTerm to be:

"queryTerm: andStatement | comparison | group | orStatement ;"

The parser rule 'expersion' can be removed and the parser rule 'group' would become:

"group: GROUP ;"

requiring the new lexer rule:

"GROUP: '(' .* ')' ;"'

While this is probably not perfect and will require tweaking to make it actually work, it gives an idea on the scope. However, if the user will never be typing these items and an interface will generate it, then these changes are going to cost a lot for no gain.

tloubrieu-jpl · 2021-04-13T20:26:03Z

@tloubrieu-jpl @jordanpadams

Spaces are required after ( and before ) as defined in the 'group' parser rule. If we want to get rid of them, then going to have to do a big rework of the grammar. I think we can change queryTerm to be:

"queryTerm: andStatement | comparison | group | orStatement ;"

The parser rule 'expersion' can be removed and the parser rule 'group' would become:

"group: GROUP ;"

requiring the new lexer rule:

"GROUP: '(' .* ')' ;"'

While this is probably not perfect and will require tweaking to make it actually work, it gives an idea on the scope. However, if the user will never be typing these items and an interface will generate it, then these changes are going to cost a lot for no gain.

@al-niessner

That would make sense to me to keep the space around terms so that the parser can separate them without us having to define a specific lexer.

By any chance, did you also try to have explicit whitespaces in the parser rule, like:

group : WS* NOT? WS* LPAREN WS* expression WS* RPAREN WS*;

with:
WS : [ \t\r\n]+ -> skip ;

The WS expression was not used in the parser initially which might be wrong.

I found that on https://tomassetti.me/antlr-mega-tutorial/#chapter23

The example is:
operation : WHITESPACE* NUMBER WHITESPACE* '+' WHITESPACE* NUMBER;

Completed the testing with nested conditionals and a not group. Also added negative checks that should result in bad parsing. They are handled by forcing ANTLR to bail when there is a problem rather than doing the best it can.

al-niessner · 2021-04-13T23:04:54Z

@tloubrieu-jpl

Completed the unit testing only in the sense that it seems to cover the big bits like wildcarding, escaping the wildcarding, grouping, nested grouping, nots, and all operators. Here is the output:

success - lid eq *pdart14_meap
success - lid ne pdart14_meap?
success - lid eq "*pdart14_meap?"
success - ( lid eq *pdart14_meap* )
success - ( timestamp gt 12 and timestamp lt 27 )
success - ( timestamp ge 12 and timestamp le 27 )
success - not ( timestamp ge 12 and timestamp le 27 )
success - ( ( timestamp ge 12 and timestamp le 27 ) or ( timestamp gt 12 and timestamp lt 27 ) )
line 1:7 no viable alternative at input 'aeqb'
success - ( a eq b
success - a eq b )
line 1:0 no viable alternative at input 'not('
success - not( a eq b )
success - a eq b and c eq d and
line 1:7 no viable alternative at input 'aeqb'
success - ( a eq b and c eq d and )
line 1:7 no viable alternative at input 'aeqb'
success - ( a eq b and c eq d or e eq f )
test is a success

Added catch block to change any parsing error into an HTTP 422 (unprocessable entity). Searching on the web indicated that this error is the most appropriate in that that the syntax is sufficient to pass all URL tests and get routed to the proper processing unit (the lexer) but it cannot understand it semantically (from the standpoint of the URL). Changed the testing to JUnit test.

al-niessner · 2021-04-22T21:23:43Z

@jordanpadams @tloubrieu-jpl

Please add anyone else that should review this.

To show that the acceptance criteria are met, developed and included verify/issue_54.py.

The standard test database contains this family of lidvids:

urn:nasa:pds:izenberg_pdart14_meap:data_eetable::1.0
urn:nasa:pds:izenberg_pdart14_meap:data_imagecube::1.0
urn:nasa:pds:izenberg_pdart14_meap:data_tnmap::1.0
urn:nasa:pds:izenberg_pdart14_meap:document::1.0
urn:nasa:pds:izenberg_pdart14_meap:data_eetable:ele_evt_12hr_orbit_2011-2012::1.0
urn:nasa:pds:izenberg_pdart14_meap:data_eetable:ele_evt_8hr_orbit_2012-2013::1.0
urn:nasa:pds:izenberg_pdart14_meap:data_eetable:ele_evt_8hr_orbit_2014-2015::1.0
urn:nasa:pds:izenberg_pdart14_meap:data_imagecube:virs_cube_64ppd_h01np::1.0
urn:nasa:pds:izenberg_pdart14_meap:data_imagecube:virs_wavelengths::1.0
urn:nasa:pds:izenberg_pdart14_meap:data_tnmap:thermal_neutron_map::1.0
urn:nasa:pds:izenberg_pdart14_meap:document:grs_inst::1.0
urn:nasa:pds:izenberg_pdart14_meap:document:instrument_host::1.0
urn:nasa:pds:izenberg_pdart14_meap:document:mascs_inst::1.0
urn:nasa:pds:izenberg_pdart14_meap:document:meap_spec::1.0
urn:nasa:pds:izenberg_pdart14_meap:document:mission::1.0
urn:nasa:pds:izenberg_pdart14_meap:document:ns_inst::1.0

The simple script does two checks:

lid eq *pdart14_me*
( lid eq *pdarat14_me* and lid eq *_orbit_* )

It then checks that the correct number of items is returned and that they contain the wildcards being looked for. The output from the script is:

$ ./verify/issue_54.py 
success 200 http://localhost:8080/products?q=lid eq *pdart14_me*
  success all data parameters match
success 200 http://localhost:8080/products?q=( lid eq *pdart14_me* and lid eq *_orbit_* )
  success all data parameters match

Hence, it should be clear that the acceptance criteria has been met.

src/main/java/gov/nasa/pds/api/engineering/controllers/MyProductsApiBareController.java

jordanpadams · 2021-04-27T18:07:35Z

@tloubrieu-jpl where are we at with testing this? should we get Eugene to take a look if you don't have the time?

tloubrieu-jpl · 2021-04-27T20:13:56Z

@jordanpadams I made some comments to @al-niessner but did not get feedback yet. I will email him.

I have my test suite in postman that I would like to test against the changes. Eugene could also test but I would like to keep an eye on that.

Fixing "a eq b" required updating the api-search-query-lexer that had some fallout here. Maining that ctx.FIELD() now requires a parameter for which of them is being requested. The quoting option of a eq "b" should not carry quotes into search request.

al-niessner · 2021-04-28T23:43:20Z

@tloubrieu-jpl @jordanpadams

Should be fixed with a lexer/parser update. Your requests should now work.

tloubrieu-jpl

Hi @al-niessner ,

The request I was trying before (mercury target) seems to work. Now I am trying a request on a different property.
It shows as follow in one of the record in my elastissearch database:
... "properties": { "ops:Data_File_Info.ops:file_size": "4395", ...

Si I am trying:

curl --location --request GET 'http://localhost:8080/collections?q=ops:Data_File_Info.ops:file_size%20eq%204395'

or

curl --location --request GET 'http://localhost:8080/collections?q=ops:Data_File_Info.ops:file_size%20eq%20%224395%22'

The status is 200 but I don't get a result. I am expecting at least the one I found in my database.

request elasticSearch :SearchRequest{searchType=QUERY_THEN_FETCH, indices=[registry], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, expand_wildcards_hidden=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=null, allowPartialSearchResults=null, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={"from":0,"size":100,"timeout":"60s","query":{"bool":{"must":[{"match":{"ops:Data_File_Info.ops:file_size":{"query":"4395","operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}},{"term":{"product_class":{"value":"Product_Collection","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}}}

The error comes from the fact that the API properties must be transformed by replacing the '.' by a '/' in the elasticsearch properties.

Thanks,

Thomas

al-niessner · 2021-05-04T17:51:52Z

@tloubrieu-jpl

The error comes from the fact that the API properties must be transformed by replacing the '.' by a '/' in the elasticsearch properties.

What dot to what slash? There are lots of dots and I cannot tell what you may or may not be trying to show as now a slash.

Do we always turn dots to slash or is this a particular identifiable case? In other words if a user ever types a dot, like foo.bar, how does one know it should or should not be foo./bar?

al-niessner · 2021-05-04T18:32:23Z

@tloubrieu-jpl

When I look my instance of test data from harvest I find this: 'ops:Data_File_Info/ops:file_size'. Are you saying that we want to turn / to . to become 'ops:Data_File_Info.ops:file_size'? Or are you saying the 'ops:Data_File_Info.ops:file_size' needs to become 'ops:Data_File_Info/ops:file_size' in the search?

What is the rule for doing the substitution? Always when in a label? There are labels that have dots in the name so how does one know to use . over / in an algorithmic sense? Why is the user not typing what they are really looking for? Are we (PDS) doing an irreversible / to . conversion for display and you are trying to make it easy? Since dots are used in other labels it makes / to . conversion irreversible as dots are boson not fermion.

al-niessner · 2021-05-04T18:38:47Z

If I type it right it works so I am really confused:

niessner@elysium:~$ curl --location --request GET 'http://localhost:8080/collections?q=ops:Data_File_Info.ops:file_size%20eq%204395'
{"summary":{"q":"ops:Data_File_Info.ops:file_size eq 4395","start":0,"limit":100,"sort":[],"properties":[]}}
niessner@elysium:~$ 

niessner@elysium:~$ curl --location --request GET 'http://localhost:8080/collections?q=ops:Data_File_Info/ops:file_size%20eq%204395'
{"summary":{"q":"ops:Data_File_Info/ops:file_size eq 4395","start":0,"limit":100,"sort":[],"properties":["ops:Data_File_Info.ops:file_size","ref_lid_instrument_host","ops:Data_File_Info.ops:mime_type","lid","ref_lid_investigation","lidvid","ops:Data_File_Info.ops:md5_checksum","ops:Label_File_Info.ops:file_size","ops:Label_File_Info.ops:blob","title","ops:Label_File_Info.ops:md5_checksum","ops:Data_File_Info.ops:file_ref","_package_id","ref_lid_instrument","vid","product_class","ops:Data_File_Info.ops:creation_date_time","ops:Data_File_Info.ops:file_name","ops:Label_File_Info.ops:file_name","ops:Label_File_Info.ops:creation_date_time","ref_lid_target","ops:Label_File_Info.ops:file_ref"]},"data":[{"id":"urn:nasa:pds:izenberg_pdart14_meap:data_imagecube::1.0","type":"Product_Collection","title":"Izenberg PDART 2014 MESSENGER Advanced Products Image Cube Collection","investigations":[{"id":"urn:nasa:pds:context:investigation:mission.messenger","href":"http://localhost:8080/products/urn:nasa:pds:context:investigation:mission.messenger"}],"observing_system_components":[{"id":"urn:nasa:pds:context:instrument_host:spacecraft.mess","href":"http://localhost:8080/products/urn:nasa:pds:context:instrument_host:spacecraft.mess"},{"id":"urn:nasa:pds:context:instrument:mascs.mess","href":"http://localhost:8080/products/urn:nasa:pds:context:instrument:mascs.mess"}],"targets":[{"id":"urn:nasa:pds:context:target:planet.mercury","href":"http://localhost:8080/products/urn:nasa:pds:context:target:planet.mercury"}],"metadata":{"version":"1.0","label_url":"/var/local/harvest/archive/data_imagecube/collection_imagecube_inventory.xml"},"properties":{"ops:Data_File_Info.ops:file_size":"4395","ref_lid_instrument_host":"urn:nasa:pds:context:instrument_host:spacecraft.mess","ops:Data_File_Info.ops:mime_type":"text/plain","lid":"urn:nasa:pds:izenberg_pdart14_meap:data_imagecube","ref_lid_investigation":"urn:nasa:pds:context:investigation:mission.messenger","lidvid":"urn:nasa:pds:izenberg_pdart14_meap:data_imagecube::1.0","ops:Data_File_Info.ops:md5_checksum":"7cb459d9bbc1c4ab63a35aad69f2c5f0","ops:Label_File_Info.ops:file_size":"5406","ops:Label_File_Info.ops:blob":"eJy9WE1v2zgQvRfofyB8aoHastNkURiKisBOtgaSbmBli8VeCFoa28RKpEBSTtxfv0NKViRZttM2aA6OpXmcGb754ND+56c0IRtQmktx2RsNhj0CIpIxF6vL3t8PN/1Pvc/B2ze+xfVTGUNC1gqWl721MdnY87JYDwTTbLCSG/twbj+8zci7n4bnFD/o6Ho4HOho3SP4ASkz2wyEflaQq2QgoyQaSLXyYh0nXolTUjjbaP1eyTiPDJ3IJIHIoK8E/alrOeBG7+2bAjl+0rxCPz4+Dh4/OoNnw+HI++fuNnQ2+1xow0QEbp3m48KVWxkx4wg6YY78GCtPOu7hBgn++bMYhOFLXliiVwpYKXLiRK5QlFBewkAFuRJja2SMusf8O4gFqBXNYqbM6JymwLJxzAyjPGUriPIF+F6HlpqNMg1QGmAi+F7tuYYy3CQQzEp75H56NX8gZ8PRObm7DsPrr39ez8lVvLEsxqQMnCYz6wSZoBfkOYi+VyiraediKVVacODSjZZeoEuj88HQ+nUYU1OUlSkTJUzrYD+BfK+JqC2dcFMonz0bqskdhuVmLRVNuDYVFx/I14Hv1SWtRVm+SHYB3gJTAdL2BzrSft1a9h9sH6WKg4pf39u9OoS8CifhSdS32fwwKAYdKZ65vb9SsOsqa3R7J/j277AdVYXxBXmVatt2t4GZgmE8aUEcLK3DsDrAhaA/PO8PP/nevrRDxYuqpJtGwQ1nCbFNYNdzD7FSMHN8Vy15kxnfO9xR/IkUBp5M8W6nrqb4gaeAtYJpwQXSoNuOYZNUxjFEDUKRRMdgf/TxX99rC/fWyqyxdHRh+f84dEsbsvpWj7iExwP2OLWlc9B5YmiYp/Zxv/pUJjUEYcTxiANbd8WLNk7JCLTGE5AmsIEkmILiG4hdx2hKjtTMwxqILTASZpj/CsPu6kEuyR2oKFdbwjVhZDS86C+YiMkqkQsE6R06KtH7OVWu/0BivsFGHhMujCQX5yTBQJs8hn4ixcp9I4YnoBFZ7IAslUyJsZ5xzRcJELS8bwDLUDGF8MJ1uVIsW5N3djfvSSTTTArMLOucVbXbzpVJpc7WoJxWEuZqySLY1z6xCjR3h3hpIAUDirxzXev9gBzuFCcD7c/EBrThq85T1CEEw8yqtVL33MLYGSW447qoUPfUQsywgJTAo3QOS9wy5lNX9Sc8pqoCNA7sqCjBMa87PE4Lm4MU0wzEChSe2Q0dHUYqIXWORlXHpUbShnrfa2Hb/eborqz4CLn+XwsNamOLI9xqA+nPEN/WQSe7dOva+Qs0Pofzi9SmM5Yvj+fLY6qNylP0mq7R6lhnWAmRYkvjAns6pl1x5Zq29J4I5wtCWkBeTPpPhacYRY6GZlbt67cHaJwyHenXCsvvjcg+rl6OD0ytwNDmFNBdk0UHP9IK7xMmoDs4r9cJjXN4nDlbGJHSqV/rf4XSX218x7jE+bU+TVUT1s6LuqK6b9aNKd7PfK/99llvS4d/g8e5M0NtKxatQdiJ21tb2iUusDUz1ZXQnQ9OzyDSG997RrfURGi0Gourya2anDvEdf5ajvld3juBXC41GJLjpHzZW2wN9AIcsYu3e3MaU24Uszd2vPTGdgLH0ZpMw29khMNah7ilQUGE86QOLi5sghTfOyEUL5k85ZgjwYQpxZG9/hwMpjG55QL6N2DHwz3sXiggaaiSOL5YzpuvW6vmhdZpKe+8ZTgVOjgrlbV34TArJfNMWzrLbx2YG+fKMVMOVnaNFO+EJMS7W64PNvjaxkVuFwSj3Y7L5wNr3E8XrhzwAJnNaGgURhNHw+r9gYUpe+JpntLCSIJTlFk3Egod6MJ09urThPwQZ7ez6bfZlOK/HyDs7OcIq9v6ddLObI28Em2+dyKn987YXbuga6Zp6hKPlj/fHGntxcAqGpfi7g5qbxbtH4jw9f8vH0t3","title":"Izenberg PDART 2014 MESSENGER Advanced Products Image Cube Collection","ops:Label_File_Info.ops:md5_checksum":"ab72eb17bac95af9194d63dc7b26da50","ops:Data_File_Info.ops:file_ref":"/var/local/harvest/archive/data_imagecube/collection_imagecube_inventory.csv","_package_id":"98d573f6-ae08-4b2f-bc2a-e550a6a8bf02","ref_lid_instrument":"urn:nasa:pds:context:instrument:mascs.mess","vid":"1.0","product_class":"Product_Collection","ops:Data_File_Info.ops:creation_date_time":"2021-02-19T04:11:19Z","ops:Data_File_Info.ops:file_name":"collection_imagecube_inventory.csv","ops:Label_File_Info.ops:file_name":"collection_imagecube_inventory.xml","ops:Label_File_Info.ops:creation_date_time":"2021-02-19T04:11:19Z","ref_lid_target":"urn:nasa:pds:context:target:planet.mercury","ops:Label_File_Info.ops:file_ref":"/var/local/harvest/archive/data_imagecube/collection_imagecube_inventory.xml"}}]}
niessner@elysium:~$

tloubrieu-jpl · 2021-05-04T18:39:01Z

@tloubrieu-jpl

When I look my instance of test data from harvest I find this: 'ops:Data_File_Info/ops:file_size'. Are you saying that we want to turn / to . to become 'ops:Data_File_Info.ops:file_size'? Or are you saying the 'ops:Data_File_Info.ops:file_size' needs to become 'ops:Data_File_Info/ops:file_size' in the search?

What is the rule for doing the substitution? Always when in a label? There are labels that have dots in the name so how does one know to use . over / in an algorithmic sense? Why is the user not typing what they are really looking for? Are we (PDS) doing an irreversible / to . conversion for display and you are trying to make it easy? Since dots are used in other labels it makes / to . conversion irreversible as dots are boson not fermion.

Hi @al-niessner ,

That is the second case:
the 'ops:Data_File_Info.ops:file_size' needs to become 'ops:Data_File_Info/ops:file_size' in the search?

For the conversion, you should use the methods available in /pds-api-service/src/main/java/gov/nasa/pds/api/engineering/elasticsearch/ElasticSearchUtil.java, line 23, jsonPropertyToElasticProperty

Although the conversion is super-simple, this is safest to use this method in case, we want to change the syntax of properties again (this has been discussed).

tloubrieu-jpl · 2021-05-04T18:45:32Z

@al-niessner ,

One question, which label with . are you refering to in elasticSearch ? The conversion should be reversible, let me know where you think it is an issue. Thanks.

Regarding the request you are testeing, in my case they returned status 200 which is fine but the result is empty whereas I should have one result.

al-niessner · 2021-05-04T18:47:24Z

For the conversion, you should use the methods available in /pds-api-service/src/main/java/gov/nasa/pds/api/engineering/elasticsearch/ElasticSearchUtil.java, line 23, jsonPropertyToElasticProperty

For all FIELDs? All values too? What part of the Search.g4 comparison should be converted, as in left side of operator, all FIELDs independent which side of the operator it appears on, anything the user types, etc? When, as in always, just when it is a property but how is that determined, etc?

tloubrieu-jpl · 2021-05-04T18:54:00Z

For the conversion, you should use the methods available in /pds-api-service/src/main/java/gov/nasa/pds/api/engineering/elasticsearch/ElasticSearchUtil.java, line 23, jsonPropertyToElasticProperty

For all FIELDs? All values too? What part of the Search.g4 comparison should be converted, as in left side of operator, all FIELDs independent which side of the operator it appears on, anything the user types, etc? When, as in always, just when it is a property but how is that determined, etc?

@al-niessner
The conversion need to be applied for the field names inside the 'properties' structure. We only support in the q parameter fields coming from this properties structure unless you changed that. We don't want to convert values.

Let me know if that is still unclear.

Thanks,

al-niessner · 2021-05-04T19:13:45Z

@tloubrieu-jpl

If I understood all of this correctly, the conversion is now being done.

tloubrieu-jpl · 2021-05-05T21:06:57Z

Hi @al-niessner ,

do you have a working example of request with "not" operator ? I tried

curl --location --request GET 'http://localhost:8080/products?q=not(%20ops:Data_File_Info.ops:file_size%20le%201050%20)%20'

But that did not work,

Thanks,

Thomas

al-niessner · 2021-05-05T21:35:09Z

Hi @al-niessner ,

do you have a working example of request with "not" operator ? I tried

curl --location --request GET 'http://localhost:8080/products?q=not(%20ops:Data_File_Info.ops:file_size%20le%201050%20)%20'

But that did not work,

Yes, there is a test for not.

'not( a eq b )' does not work. It has to be 'not ( a eq b )'. It is that whole white space parenthesis thing.

Use curl --location --request GET 'http://localhost:8080/products?q=not%20(%20ops:Data_File_Info.ops:file_size%20le%201050%20)%20'

tloubrieu-jpl

@al-niessner thanks. The new antlr4 listener looks good and more importantly it works. I updated my postman test suite.

I appreciate the work you've done on that.

I still have one question on the management of the right term of the query (in my code comments), let me know what you think about that.

Thanks

src/main/java/gov/nasa/pds/api/engineering/elasticsearch/Antlr4SearchListener.java

src/main/java/gov/nasa/pds/api/engineering/controllers/MyProductsApiBareController.java

src/main/java/gov/nasa/pds/api/engineering/elasticsearch/Antlr4SearchListener.java

tloubrieu-jpl

Your latest update does not work, you can remove it. It is not something we need now. Thanks

src/main/java/gov/nasa/pds/api/engineering/controllers/MyProductsApiBareController.java

al-niessner · 2021-05-06T19:38:08Z

Yeah, well, what I thought was fix is not. Ah well fix it on another ticket.

al-niessner · 2021-05-06T19:43:34Z

@tloubrieu-jpl comment added.

al-niessner and others added 5 commits February 25, 2021 10:48

Merge pull request #1 from NASA-PDS/master

01c6a98

up to date

Merge pull request #2 from NASA-PDS/master

c4beaf7

Merge from parent repo

Merge branch 'master' into pds-api-54

4e6b8cf

tiny cleanup

b0658ac

al-niessner changed the title ~~issue [pds-api-54](https://github.com/NASA-PDS/pds-api/issues/54): wildcard searchiing~~ issue [pds-api-54](https://github.com/NASA-PDS/pds-api/issues/54) : wildcard searching Apr 8, 2021

al-niessner changed the title ~~issue [pds-api-54](https://github.com/NASA-PDS/pds-api/issues/54) : wildcard searching~~ issue [pds-api-54]: wildcard searching Apr 8, 2021

step forward on testing

2382d1e

Fix erroneous stack handling in Antlr4SearchListener. Fix up the second test to show that the group does work.

test updates

5012a49

Tested the bulk of the grammar but need to do nested groups and find a way to verify the and/or statements are correctly stacked.

update unit testing

ab196ea

Completed the testing with nested conditionals and a not group. Also added negative checks that should result in bad parsing. They are handled by forcing ANTLR to bail when there is a problem rather than doing the best it can.

removed clutter

e3ab1e8

al-niessner mentioned this pull request Apr 20, 2021

API server crashes with OutOfMemoryError if invalid query is used #16

Closed

Al Niessner added 5 commits April 22, 2021 09:59

updated POM to latest lexer

9bb0d36

make errors uniform and add logging

d5ae142

fix not group

9eb5c44

verification tool for acceptance criteria

a1687f2

al-niessner requested review from jordanpadams and tloubrieu-jpl April 22, 2021 21:17

make URL more readable because requests helps

3201409

tloubrieu-jpl reviewed Apr 26, 2021

View reviewed changes

src/main/java/gov/nasa/pds/api/engineering/controllers/MyProductsApiBareController.java Outdated Show resolved Hide resolved

fix details

4893710

Fixing "a eq b" required updating the api-search-query-lexer that had some fallout here. Maining that ctx.FIELD() now requires a parameter for which of them is being requested. The quoting option of a eq "b" should not carry quotes into search request.

change http error code

d21e3ce

tloubrieu-jpl suggested changes May 4, 2021

View reviewed changes

odd FIELD conversion as requested

ed80031

tloubrieu-jpl reviewed May 6, 2021

View reviewed changes

src/main/java/gov/nasa/pds/api/engineering/elasticsearch/Antlr4SearchListener.java Show resolved Hide resolved

src/main/java/gov/nasa/pds/api/engineering/controllers/MyProductsApiBareController.java Show resolved Hide resolved

put error message in response

83c99f4

tloubrieu-jpl suggested changes May 6, 2021

View reviewed changes

src/main/java/gov/nasa/pds/api/engineering/elasticsearch/Antlr4SearchListener.java Show resolved Hide resolved

tloubrieu-jpl suggested changes May 6, 2021

View reviewed changes

src/main/java/gov/nasa/pds/api/engineering/controllers/MyProductsApiBareController.java Show resolved Hide resolved

not right for this ResponseEntity<Products>

14aeb32

add requested comment

a49b330

tloubrieu-jpl merged commit fa60dc8 into master May 6, 2021

jordanpadams mentioned this pull request May 7, 2021

As a user, I want to know why my query syntax is invalid #26

Closed

tdddblog deleted the pds-api-54 branch September 21, 2021 20:27

tloubrieu-jpl mentioned this pull request Jan 5, 2022

As a user, I want to know why my query syntax is invalid NASA-PDS/registry-api#13

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

issue [pds-api-54]: wildcard searching #15

issue [pds-api-54]: wildcard searching #15

al-niessner commented Apr 7, 2021

al-niessner commented Apr 7, 2021

al-niessner commented Apr 8, 2021

jordanpadams commented Apr 8, 2021

tloubrieu-jpl commented Apr 9, 2021 •

edited

Loading

al-niessner commented Apr 13, 2021

al-niessner commented Apr 13, 2021

tloubrieu-jpl commented Apr 13, 2021

al-niessner commented Apr 13, 2021

al-niessner commented Apr 22, 2021 •

edited

Loading

jordanpadams commented Apr 27, 2021 •

edited by tloubrieu-jpl

Loading

tloubrieu-jpl commented Apr 27, 2021 •

edited

Loading

al-niessner commented Apr 28, 2021

tloubrieu-jpl left a comment

al-niessner commented May 4, 2021 •

edited

Loading

al-niessner commented May 4, 2021

al-niessner commented May 4, 2021

tloubrieu-jpl commented May 4, 2021

tloubrieu-jpl commented May 4, 2021

al-niessner commented May 4, 2021

tloubrieu-jpl commented May 4, 2021

al-niessner commented May 4, 2021

tloubrieu-jpl commented May 5, 2021

al-niessner commented May 5, 2021

tloubrieu-jpl left a comment

tloubrieu-jpl left a comment

al-niessner commented May 6, 2021

al-niessner commented May 6, 2021

issue [pds-api-54]: wildcard searching #15

issue [pds-api-54]: wildcard searching #15

Conversation

al-niessner commented Apr 7, 2021

al-niessner commented Apr 7, 2021

al-niessner commented Apr 8, 2021

jordanpadams commented Apr 8, 2021

tloubrieu-jpl commented Apr 9, 2021 • edited Loading

al-niessner commented Apr 13, 2021

al-niessner commented Apr 13, 2021

tloubrieu-jpl commented Apr 13, 2021

al-niessner commented Apr 13, 2021

al-niessner commented Apr 22, 2021 • edited Loading

jordanpadams commented Apr 27, 2021 • edited by tloubrieu-jpl Loading

tloubrieu-jpl commented Apr 27, 2021 • edited Loading

al-niessner commented Apr 28, 2021

tloubrieu-jpl left a comment

Choose a reason for hiding this comment

al-niessner commented May 4, 2021 • edited Loading

al-niessner commented May 4, 2021

al-niessner commented May 4, 2021

tloubrieu-jpl commented May 4, 2021

tloubrieu-jpl commented May 4, 2021

al-niessner commented May 4, 2021

tloubrieu-jpl commented May 4, 2021

al-niessner commented May 4, 2021

tloubrieu-jpl commented May 5, 2021

al-niessner commented May 5, 2021

tloubrieu-jpl left a comment

Choose a reason for hiding this comment

tloubrieu-jpl left a comment

Choose a reason for hiding this comment

al-niessner commented May 6, 2021

al-niessner commented May 6, 2021

tloubrieu-jpl commented Apr 9, 2021 •

edited

Loading

al-niessner commented Apr 22, 2021 •

edited

Loading

jordanpadams commented Apr 27, 2021 •

edited by tloubrieu-jpl

Loading

tloubrieu-jpl commented Apr 27, 2021 •

edited

Loading

al-niessner commented May 4, 2021 •

edited

Loading