Struct Lexicon 

pub struct Lexicon<T: Eq, Category: Eq> { /* private fields */ }

The struct which represents a specific MG. They can be created by parsing:

let grammar: Lexicon<&str, &str> = Lexicon::from_string("John::d\nsaw::d= =d v\nMary::d")?;

It is also possible to make a Lexicon which use different types for categories or lemmas, such as integers or Strings. These can be made with Lexicon::remap_lexicon or by passing a vector of LexicalEntry using Lexicon::new.

Lemmas

Lemmas are representated as Option<T>, so empty strings are None rather than an empty string to allow for arbitrary lemma type T.

Useful functions

• Lexicon::parse parse a string.
• Lexicon::parse_multiple parse multiple strings simultaneously.
• Lexicon::generate generate some of the strings of a grammar.
• Lexicon::valid_continuations find the valid next tokens of a grammar given a prefix

Representation

The struct represents a lexicon as a graph, as per Stabler (2013). This means certain functions exploit this, such as Lexicon::n_nodes which doesn’t reference the number of features of the grammar as written out, but rather the number of nodes in the graph representation.

• Stabler, E. (2013). Two Models of Minimalist, Incremental Syntactic Analysis. Topics in Cognitive Science, 5(3), 611–633. https://doi.org/10.1111/tops.12031

Implementations

impl<T: Eq + Debug + Clone + SymbolCost, Category: Eq + Debug + Clone + Hash> Lexicon<T, Category>

pub fn mdl_score_fixed_category_size( &self, n_phonemes: u16, n_categories: u16, ) -> Result<f64, LexiconError>

Returns the MDL score of a lexicon assuming the number of phonemes is fixed.

pub fn mdl_score(&self, n_phonemes: u16) -> Result<f64, LexiconError>

Returns the MDL Score of the lexicon

Arguments

• n_phonemes - The size of required to encode a symbol of the phonology. e.g in English orthography, it would be 26.

impl<T, C> Lexicon<T, C>

where T: Eq + Debug + Clone, C: Eq + Debug + Clone,

pub fn prune(&mut self, start: &C)

Prunes a grammar of any inaccessible grammatical categories that can’t be found given a base category of start

impl<T, C> Lexicon<T, C>

where T: Eq + Debug + Clone + Hash, C: Eq + Debug + Clone + FreshCategory + Hash,

pub fn uniform_distribution(&mut self)

Remap the weights of all lexical items to be the uniform distribution.

pub fn unify(&mut self, other: &Self) -> Vec<LexemeDetails>

Combines two lexicons and returns a vector of the leaves from other as LexemeDetails consisting of the index of the old leaves in other and their new index in the self.

pub fn add_new_lexeme_from_sibling( &mut self, lemma: T, rng: &mut impl Rng, ) -> Option<NewLexeme>

Adds a new lexeme with the same features but a different lemma as another lexeme. Returns the node of the new lexeme as a NewLexeme with data about its copied sibling.

pub fn add_new_lexeme( &mut self, lemma: Option<T>, config: Option<LexicalProbConfig>, rng: &mut impl Rng, ) -> Option<LexemeId>

Adds a new lexeme and return its node index if it is novel.

pub fn delete_lexeme(&mut self, lexeme: LexemeId) -> Result<(), MutationError>

Deletes a lexeme from the grammar. Returns MutationError if the grammar has only one lexeme or if the NodeIndex passed is not a leaf.

pub fn delete_from_node(&mut self, rng: &mut impl Rng)

Delets a random branch of the grammar

pub fn delete_node(&mut self, rng: &mut impl Rng) -> Option<LexemeId>

Find a random node and deletes it, and then restitches its parent and children.

pub fn random( base_category: &C, lemmas: &[T], config: Option<LexicalProbConfig>, rng: &mut impl Rng, ) -> Self

Samples an entirely random grammar

pub fn change_feature( &mut self, lemmas: &[T], config: Option<LexicalProbConfig>, rng: &mut impl Rng, )

Chose a random feature and change it to something else.

pub fn resample_below_node( &mut self, lemmas: &[T], config: Option<LexicalProbConfig>, rng: &mut impl Rng, )

Finds a random node and deletes its children and samples from scratch below that node.

impl<T, Category> Lexicon<T, Category>

where T: Eq + Debug + Clone + Display, Category: Eq + Debug + Clone + Display,

pub fn graphviz(&self) -> String

Prints a lexicon as a GraphViz dot file.

impl<T: Eq, Category: Eq> Lexicon<T, Category>

pub fn leaf_to_features<'a>( &'a self, lexeme: LexemeId, ) -> Option<Climber<'a, T, Category>>

Climb up a lexeme over all of its features

pub fn leaf_to_lemma(&self, lexeme_id: LexemeId) -> Option<&Option<T>>

Gets the lemma of a lexeme.

pub fn category(&self, lexeme_id: LexemeId) -> Option<&Category>

Get the category of a lexeme.

impl<T: Eq + Debug + Clone, Category: Eq + Debug + Clone> Lexicon<T, Category>

pub fn is_complement(&self, nx: NodeIndex) -> bool

Checks if nx is a complement

pub fn n_nodes(&self) -> usize

The number of nodes in a lexicon.

pub fn leaves(&self) -> &[LexemeId]

Returns the leaves of a grammar

pub fn sibling_leaves(&self) -> Vec<Vec<LexemeId>>

Returns all leaves with their sibling nodes (e.g. exemes that are identical except for their lemma)

pub fn lexemes_and_ids( &self, ) -> Result<impl Iterator<Item = (LexemeId, LexicalEntry<T, Category>)>, LexiconError>

Gets each lexical entry along with its ID.

pub fn lexemes(&self) -> Result<Vec<LexicalEntry<T, Category>>, LexiconError>

Turns a lexicon into a Vec<LexicalEntry<T, Category>> which can be useful for printing or investigating individual lexical entries.

pub fn get_lexical_entry( &self, lexeme_id: LexemeId, ) -> Result<LexicalEntry<T, Category>, LexiconError>

Given a leaf node, return its LexicalEntry

pub fn lemmas(&self) -> impl Iterator<Item = &Option<T>>

Get all lemmas of a grammar

pub fn new(items: Vec<LexicalEntry<T, Category>>, collapse_lemmas: bool) -> Self

Create a new grammar from a Vec of LexicalEntry

pub fn new_with_weights( items: Vec<LexicalEntry<T, Category>>, weights: Vec<f64>, collapse_lemmas: bool, ) -> Self

Create a new grammar from a Vec of LexicalEntry where lexical entries are weighted in probability

pub fn categories(&self) -> impl Iterator<Item = &Category>

Iterate over all categories of a grammar

pub fn licensor_types(&self) -> impl Iterator<Item = &Category>

Iterate over all licensors of a grammar

impl<'src> Lexicon<&'src str, &'src str>

pub fn from_string(s: &'src str) -> Result<Self, LexiconParsingError<'src>>

Parse a grammar and return a Lexicon<&str, &str>

impl<T: Eq, C: Eq> Lexicon<T, C>

pub fn remap_lexicon<'lex, T2: Eq, C2: Eq>( &'lex self, lemma_map: impl Fn(&'lex T) -> T2, category_map: impl Fn(&'lex C) -> C2, ) -> Lexicon<T2, C2>

Converts a Lexicon<T,C> to a Lexicon<T2, C2> according to two functions which remap them.

impl<'a> Lexicon<&'a str, &'a str>

pub fn to_owned_values(&self) -> Lexicon<String, String>

Converts from Lexicon<&str, &str> to Lexicon<String, String>

impl<T, C> Lexicon<T, C>

where T: Eq + Debug + Clone + Hash, C: Eq + Clone + Debug + Hash,

pub fn valid_continuations( &self, initial_category: C, prefix: &[PhonContent<T>], config: &ParsingConfig, ) -> Result<HashSet<Continuation<T>>, ParsingError<C>>

Given a grammar and a prefix string, return a [HashSet] of the possible Continuations (i.e. next words) that are valid in the grammar. Returns an ParsingError if there is no node with the category of initial_category.

let lex = Lexicon::from_string("a::S= b= S\n::S\nb::b")?;
let continuations = lex.valid_continuations("S", &PhonContent::from(["a"]), &ParsingConfig::default())?;
assert_eq!(continuations, HashSet::from_iter([Continuation::Word("b"), Continuation::Word("a")].into_iter()));
let continuations = lex.valid_continuations("S", &PhonContent::from(["a", "b"]), &ParsingConfig::default())?;
assert_eq!(continuations, HashSet::from_iter([Continuation::EndOfSentence]));

impl<T: Eq + Clone, C: Eq + Clone + Display> Lexicon<T, C>

pub fn derivation(&self, rules: RulePool) -> Derivation<'static, T, C>

Converts a RulePool into a Derivation which allows the construction of Trees which can be used to plot syntactic trees throughout the derivation of the parse.

impl<T, Category> Lexicon<T, Category>

where T: Eq + Debug + Clone, Category: Eq + Clone + Debug,

pub fn generate( &self, category: Category, config: &ParsingConfig, ) -> Result<Generator<&Self, T, Category>, ParsingError<Category>>

Generates the strings in a grammar that are findable according to the parsing config.

Returns an iterator, Parser which has items of type ([LogProb<f64>], Vec<T>, RulePool) where the middle items are the generated strings and the RulePool has the structure of each genrerated string.

pub fn into_generate( self, category: Category, config: &ParsingConfig, ) -> Result<Generator<Self, T, Category>, ParsingError<Category>>

Like Lexicon::generate but consumes the lexicon.

pub fn parse<'a, 'b>( &'a self, s: &'b [PhonContent<T>], category: Category, config: &'a ParsingConfig, ) -> Result<Parser<'a, 'b, T, Category>, ParsingError<Category>>

Parses a sentence under the assumption it is has the category, category.

Returns an iterator, Parser which has items of type ([LogProb<f64>], &'a [T], RulePool)

pub fn parse_multiple<'a, 'b, U>( &'a self, sentences: &'b [U], category: Category, config: &'a ParsingConfig, ) -> Result<Parser<'a, 'b, T, Category>, ParsingError<Category>>

where U: AsRef<[PhonContent<T>]>,

Functions like Lexicon::parse but parsing multiple grammars at once.

pub fn fuzzy_parse<'a, 'b, U>( &'a self, sentences: &'b [U], category: Category, config: &'a ParsingConfig, ) -> Result<FuzzyParser<'a, 'b, T, Category>, ParsingError<Category>>

where U: AsRef<[PhonContent<T>]>,

Functions like Lexicon::parse or Lexicon::generate but will return parses that are close to the provided sentences, but are not necessarily the same.

Trait Implementations

impl<T: Clone + Eq, Category: Clone + Eq> Clone for Lexicon<T, Category>

fn clone(&self) -> Lexicon<T, Category>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<T: Debug + Eq, Category: Debug + Eq> Debug for Lexicon<T, Category>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<T, C> Display for Lexicon<T, C>

where T: Eq + Display + Debug + Clone, C: Eq + Display + Debug + Clone,

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<T: Eq, C: Eq> From<SemanticLexicon<'_, T, C>> for Lexicon<T, C>

fn from(value: SemanticLexicon<'_, T, C>) -> Self

Converts to this type from the input type.

impl<T: Eq, Category: Eq> PartialEq for Lexicon<T, Category>

fn eq(&self, other: &Self) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl<T: Eq, Category: Eq> Eq for Lexicon<T, Category>