Struct Lexicon 

pub struct Lexicon<T: Eq, Category: Eq> { /* private fields */ }

The struct which represents a specific MG. They can be created by parsing:

let grammar: Lexicon<&str, &str> = Lexicon::from_string("John::d\nsaw::d= =d v\nMary::d")?;

It is also possible to make a Lexicon which use different types for categories or lemmas, such as integers or Strings. These can be made with Lexicon::remap_lexicon or by passing a vector of LexicalEntry using Lexicon::new.

Lemmas

Lemmas are representated as Option<T>, so empty strings are None rather than an empty string to allow for arbitrary lemma type T.

Useful functions

• Lexicon::parse parse a string.
• Lexicon::parse_multiple parse multiple strings simultaneously.
• Lexicon::generate generate some of the strings of a grammar.
• Lexicon::valid_continuations find the valid next tokens of a grammar given a prefix

Representation

The struct represents a lexicon as a graph, as per Stabler (2013). This means certain functions exploit this, such as Lexicon::n_nodes which doesn’t reference the number of features of the grammar as written out, but rather the number of nodes in the graph representation.

• Stabler, E. (2013). Two Models of Minimalist, Incremental Syntactic Analysis. Topics in Cognitive Science, 5(3), 611–633. https://doi.org/10.1111/tops.12031

Implementations

impl<T: Eq + Debug + Clone + SymbolCost, Category: Eq + Debug + Clone + Hash> Lexicon<T, Category>

pub fn mdl_score_fixed_category_size( &self, n_phonemes: u16, n_categories: u16, ) -> Result<f64, LexiconError>

Returns the MDL score of a lexicon assuming the number of phonemes is fixed.

pub fn mdl_score(&self, n_phonemes: u16) -> Result<f64, LexiconError>

Returns the MDL Score of the lexicon

Arguments

• n_phonemes - The size of required to encode a symbol of the phonology. e.g in English orthography, it would be 26.

impl<T, C> Lexicon<T, C>

where T: Eq + Debug + Clone, C: Eq + Debug + Clone,

pub fn prune(&mut self, start: &C)

Prunes a grammar of any inaccessible grammatical categories that can’t be found given a base category of start

impl<T, C> Lexicon<T, C>

where T: Eq + Debug + Clone + Hash, C: Eq + Debug + Clone + FreshCategory + Hash,

pub fn uniform_distribution(&mut self)

Remap the weights of all lexical items to be the uniform distribution.

pub fn unify(&mut self, other: &Self) -> Vec<LexemeDetails>

Combines two lexicons and returns a vector of the leaves from other as LexemeDetails consisting of the index of the old leaves in other and their new index in the self.

pub fn add_new_lexeme_from_sibling( &mut self, lemma: T, rng: &mut impl Rng, ) -> Option<NewLexeme>

Adds a new lexeme with the same features but a different lemma as another lexeme. Returns the node of the new lexeme as a NewLexeme with data about its copied sibling.

pub fn add_new_lexeme( &mut self, lemma: Option<T>, config: Option<LexicalProbConfig>, rng: &mut impl Rng, ) -> Option<LexemeId>

Adds a new lexeme and return its node index if it is novel.

pub fn delete_lexeme(&mut self, lexeme: LexemeId) -> Result<(), MutationError>

Deletes a lexeme from the grammar. Returns MutationError if the grammar has only one lexeme or if the NodeIndex passed is not a leaf.

pub fn delete_from_node(&mut self, rng: &mut impl Rng)

Delets a random branch of the grammar

pub fn delete_node(&mut self, rng: &mut impl Rng) -> Option<LexemeId>

Find a random node and deletes it, and then restitches its parent and children.

pub fn random( base_category: &C, lemmas: &[T], config: Option<LexicalProbConfig>, rng: &mut impl Rng, ) -> Self

Samples an entirely random grammar

pub fn change_feature( &mut self, lemmas: &[T], config: Option<LexicalProbConfig>, rng: &mut impl Rng, )

Chose a random feature and change it to something else.

pub fn resample_below_node( &mut self, lemmas: &[T], config: Option<LexicalProbConfig>, rng: &mut impl Rng, )

Finds a random node and deletes its children and samples from scratch below that node.

impl<T, Category> Lexicon<T, Category>

where T: Eq + Debug + Clone + Display, Category: Eq + Debug + Clone + Display,

pub fn graphviz(&self) -> String

Prints a lexicon as a GraphViz dot file.

impl<T: Eq + Debug, Category: Eq + Hash + Clone> Lexicon<T, Category>

pub fn obligatory_movers(&self) -> Vec<ObligatoryMover<Category>>

This function goes through a lexicon and returns all licensees which are always correlated with a specific category.

 let lex = Lexicon::from_string("John::d -k -q\nwho::d -k -q -w\nsaw::d= +k v")?;
 assert_eq!(
     lex.obligatory_movers(),
     vec![
         ObligatoryMover {
             licensee: "k",
             category: "d",
         },
         ObligatoryMover {
             licensee: "q",
             category: "d",
         },
     ]
 );
 let lex = Lexicon::from_string("John::d -k -q\nwho::d -k -w\nsaw::d= +k v")?;
 assert_eq!(
     lex.obligatory_movers(),
     vec![ObligatoryMover {
         licensee: "k",
         category: "d",
     }]
 );
 let lex = Lexicon::from_string("John::d -q\nwho::d -k -q -w\nsaw::d= +k v")?;
 assert_eq!(lex.obligatory_movers(), vec![]);

 let lex = Lexicon::from_string("John::d -k -q\nwho::d -k -q -w\nsaw::d= +k v -q")?;
 assert_eq!(
     lex.obligatory_movers(),
     vec![ObligatoryMover {
         licensee: "k",
         category: "d",
     }]
 );

impl<T: Eq, Category: Eq> Lexicon<T, Category>

pub fn root_categories(&self) -> impl Iterator<Item = &Category>

Goes over all categories than can be parsed. Crucially, will exclude any category that may must necessarily be moved.

pub fn leaf_to_features( &self, lexeme: LexemeId, ) -> Option<Climber<'_, T, Category>>

Climb up a lexeme over all of its features

pub fn leaf_to_lemma(&self, lexeme_id: LexemeId) -> Option<&Option<T>>

Gets the lemma of a lexeme.

pub fn category(&self, lexeme_id: LexemeId) -> Option<&Category>

Get the category of a lexeme.

impl<T: Eq, Category: Eq> Lexicon<T, Category>

pub fn is_complement(&self, nx: NodeIndex) -> bool

Checks if nx is a complement

pub fn n_nodes(&self) -> usize

The number of nodes in a lexicon.

pub fn leaves(&self) -> &[LexemeId]

Returns the leaves of a grammar

pub fn lemmas(&self) -> impl Iterator<Item = &Option<T>>

Get all lemmas of a grammar

impl<T: Eq + Debug + Clone, Category: Eq + Debug + Clone> Lexicon<T, Category>

pub fn sibling_leaves(&self) -> Vec<Vec<LexemeId>>

Returns all leaves with their sibling nodes (e.g. exemes that are identical except for their lemma)

pub fn lexemes_and_ids( &self, ) -> Result<impl Iterator<Item = (LexemeId, LexicalEntry<T, Category>)>, LexiconError>

Gets each lexical entry along with its ID.

pub fn lexemes(&self) -> Result<Vec<LexicalEntry<T, Category>>, LexiconError>

Turns a lexicon into a Vec<LexicalEntry<T, Category>> which can be useful for printing or investigating individual lexical entries.

pub fn get_lexical_entry( &self, lexeme_id: LexemeId, ) -> Result<LexicalEntry<T, Category>, LexiconError>

Given a leaf node, return its LexicalEntry

pub fn new(items: Vec<LexicalEntry<T, Category>>, collapse_lemmas: bool) -> Self

Create a new grammar from a Vec of LexicalEntry

pub fn new_with_weights( items: Vec<LexicalEntry<T, Category>>, weights: Vec<f64>, collapse_lemmas: bool, ) -> Self

Create a new grammar from a Vec of LexicalEntry where lexical entries are weighted in probability

pub fn categories(&self) -> impl Iterator<Item = &Category>

Iterate over all categories of a grammar whether as selectors or categories. This goes over the entire lexicon so it will repeat elements multiple times.

pub fn licensor_types(&self) -> impl Iterator<Item = &Category>

Iterate over all licensors of a grammar

impl<'src> Lexicon<&'src str, &'src str>

pub fn from_string(s: &'src str) -> Result<Self, LexiconParsingError<'src>>

Parse a grammar and return a Lexicon<&str, &str>

impl<T: Eq, C: Eq> Lexicon<T, C>

pub fn remap_lexicon<'lex, T2: Eq, C2: Eq>( &'lex self, lemma_map: impl Fn(&'lex T) -> T2, category_map: impl Fn(&'lex C) -> C2, ) -> Lexicon<T2, C2>

Converts a Lexicon<T,C> to a Lexicon<T2, C2> according to two functions which remap them.

impl<'a> Lexicon<&'a str, &'a str>

pub fn to_owned_values(&self) -> Lexicon<String, String>

Converts from Lexicon<&str, &str> to Lexicon<String, String>

impl<T, C> Lexicon<T, C>

where T: Eq + Debug + Clone + Hash, C: Eq + Clone + Debug + Hash,

pub fn valid_continuations( &self, initial_category: C, prefix: &[PhonContent<T>], config: &ParsingConfig, ) -> Result<HashSet<Continuation<T>>, ParsingError<C>>

Given a grammar and a prefix string, return a [HashSet] of the possible Continuations (i.e. next words) that are valid in the grammar. Returns an ParsingError if there is no node with the category of initial_category.

let lex = Lexicon::from_string("a::S= b= S\n::S\nb::b")?;
let continuations = lex.valid_continuations("S", &PhonContent::from(["a"]), &ParsingConfig::default())?;
assert_eq!(continuations, HashSet::from_iter([Continuation::Word("b"), Continuation::Word("a")].into_iter()));
let continuations = lex.valid_continuations("S", &PhonContent::from(["a", "b"]), &ParsingConfig::default())?;
assert_eq!(continuations, HashSet::from_iter([Continuation::EndOfSentence]));

impl<T: Eq + Clone, C: Eq + Clone + Display> Lexicon<T, C>

pub fn derivation(&self, rules: RulePool) -> Derivation<'static, T, C>

Converts a RulePool into a Derivation which allows the construction of Trees which can be used to plot syntactic trees throughout the derivation of the parse.

impl<T, Category> Lexicon<T, Category>

where T: Eq + Debug + Clone, Category: Eq + Clone + Debug,

pub fn generate( &self, category: Category, config: &ParsingConfig, ) -> Result<Generator<&Self, T, Category>, ParsingError<Category>>

Generates the strings in a grammar that are findable according to the parsing config.

Returns an iterator, Parser which has items of type ([LogProb<f64>], Vec<T>, RulePool) where the middle items are the generated strings and the RulePool has the structure of each genrerated string.

pub fn into_generate( self, category: Category, config: &ParsingConfig, ) -> Result<Generator<Self, T, Category>, ParsingError<Category>>

Like Lexicon::generate but consumes the lexicon.

pub fn random_parse<'a, 'b, R: Rng>( &'a self, s: &'b [PhonContent<T>], category: Category, config: &'a ParsingConfig, rng: &'a mut R, ) -> Result<Option<(LogProb<f64>, &'b [PhonContent<T>], RulePool)>, ParsingError<Category>>

Parses a sentence under the assumption it is has the category, category, but will randomly return only one possible parse.

Returns None if no parse can be found.

pub fn random_parse_multiple<'a, 'b, U, R: Rng>( &'a self, sentences: &'b [U], category: Category, config: &'a ParsingConfig, rng: &'a mut R, ) -> Result<RandomParser<'a, 'b, T, Category, R>, ParsingError<Category>>

where U: AsRef<[PhonContent<T>]>,

Parses sentences under the assumption it is has the category, category, but will randomly return only one possible parse for each sentence.

Returns an iterator, Parser which has items of type ([LogProb<f64>], &'a [T], RulePool)

pub fn parse<'a, 'b>( &'a self, s: &'b [PhonContent<T>], category: Category, config: &'a ParsingConfig, ) -> Result<Parser<'a, 'b, T, Category>, ParsingError<Category>>

Parses a sentence under the assumption it is has the category, category.

Returns an iterator, Parser which has items of type ([LogProb<f64>], &'a [T], RulePool)

pub fn parse_multiple<'a, 'b, U>( &'a self, sentences: &'b [U], category: Category, config: &'a ParsingConfig, ) -> Result<Parser<'a, 'b, T, Category>, ParsingError<Category>>

where U: AsRef<[PhonContent<T>]>,

Functions like Lexicon::parse but parsing multiple grammars at once.

pub fn fuzzy_parse<'a, 'b, U>( &'a self, sentences: &'b [U], category: Category, config: &'a ParsingConfig, ) -> Result<FuzzyParser<'a, 'b, T, Category>, ParsingError<Category>>

where U: AsRef<[PhonContent<T>]>,

Functions like Lexicon::parse or Lexicon::generate but will return parses that are close to the provided sentences, but are not necessarily the same.

Trait Implementations

impl<T: Clone + Eq, Category: Clone + Eq> Clone for Lexicon<T, Category>

fn clone(&self) -> Lexicon<T, Category>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<T: Debug + Eq, Category: Debug + Eq> Debug for Lexicon<T, Category>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<T, C> Display for Lexicon<T, C>

where T: Eq + Display + Debug + Clone, C: Eq + Display + Debug + Clone,

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<T: Eq, C: Eq> From<SemanticLexicon<'_, T, C>> for Lexicon<T, C>

fn from(value: SemanticLexicon<'_, T, C>) -> Self

Converts to this type from the input type.

impl<T: Eq, Category: Eq> PartialEq for Lexicon<T, Category>

fn eq(&self, other: &Self) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl<T: Eq, Category: Eq> Eq for Lexicon<T, Category>