Remove Vocab.oov_prob, fix lexeme_ lookups table creation

[adrianeboyd]

Description

Remove Vocab.oov_prob and Vocab.cfg (both unused) and fix how lexeme_prob and lexeme_cluster tables are added dynamically when an attribute is set on a vocab where these tables don't already exist.

Types of change

?

Checklist

• I confirm that I have the right to submit this contribution under the project's MIT license.
• I ran the tests, and all new and existing tests passed.
• My changes don't require a change to the documentation, or if they do, I've added all required information.

[adrianeboyd]

The original test did not make sense. lex1 < lex2 is comparing ORTH and not PROB. Replaced with a test for the lookups table creation.

[adrianeboyd]

This is meant to be a teeny step in the right direction in case these attributes are preserved in this format.

It may not make sense to continue to keep the Lexeme properties that have been kept for backwards compatibility with lookups tables instead of values stored on LexemeC.

They would be a good candidate for a refactored lex_attr_getters where these can be added flexibly and the data structures and defaults and backoffs would come from user code rather than being hard-coded here.

[adrianeboyd]

Actually, I'm not sure this is helpful for lex_attr_getters because those are only for the values in the struct. It is a mess at this point!

[honnibal]

You'll maybe hate me for saying this because it's indeed been pretty broken / messy. But:

I don't feel great about having the OOV_PROB be a constant value. This is probably pedantic because nobody really makes these unigram frequency distributions anymore, but what we're really trying to do here is say if n=0 in some sample, what probability should we assume that the item has of occurring? The value that it's sensible to say for this will depend on how many items we've sampled. (As a probably irrelevant technical note, in the early versions of the lexeme data these probabilities were estimated with Simple Good Turing estimation, so the probability was a projection based on the probability of items with count 2 and the probability of items with count 1).

Is there a version of this that lets us keep the oov probability as a value in the vocab, but handles it correctly?

[svlandeg]

Temporarily closing this PR as we currently don't have the bandwidth to finish this.