How to Read Machinese Analysers' Output

In the following sections of this manual, you will find answers on how to read Machinese analysers' output. In the examples, the command line programs are used, but for most part, you can archive the same using the Machinese COM or library application programming interfaces.

Tokenisation

The most basic information Machinese analysers return is the boundaries of words and sentences. Tokenisation rules in Machinese analysers may divide some of the words in the input text into two tokens - for example, "isn't" is analysed to contain two tokens "is" and "n't". Sentence breaks are marked with special tags.

In Machinese Phrase Tagger, the sentence boundaries are marked with tag SB or self-explaining phrase "sentence boundary" if using the prose output format.

Machinese Syntax marks sentence boundaries with the tag <s> and detects paragraph ends and shows these using the tag <p>.

For example, Machinese Phrase Tagger gives the following analysis for the sentence "Isn't Dr. Spock here?":

The first two columns in this default output format show the start position of the token and its length in characters. From this information, you can see that the first two tokens are written together as a single word in the input text. Also, you can see that the dot in "Dr." does not mark the end of the sentence, but the question mark does as it is marked with the tag SB.

Base Forms

In most languages, words are inflected in various case forms to form sensible phrases. However, in many applications it is desirable to find and match all occurrences of the same word despite the fact that the inflected word forms in the text are quite different - take for example irregular verbs like "be", which can also appear in forms like "is", "was" and "were". Luckily, Machinese analysers return the base form (also known as lemma) for any word.

All Machinese output formats will show the base form information, although there are small differences in the display properties. For example, in the analysis in the default output format of Machinese Phrase Tagger, the base form is shown in the fourth column.

One noteworthy detail is also that Machinese analysers mark the word boundaries of compound words (like the word bussiyhtiö in the above Finnish language example) with the non-breaking space character (U+00A0).

Part of Speech

Part of speech (PoS) or word class information describes what role the word has in the phrase or sentence. As there are several words which can occur in text in multiple PoS categories, Machinese analysers will use the context information to determine which role the word has in the sentence.

All Machinese output formats will show the part-of-speech information, although there are small differences in the display properties.

For example the following Machinese Phrase Tagger analysis for sentence 'I saw him walking with a saw.' demonstrates, that the analyser detects the two different senses of word 'saw':

For more detailed description of PoS analysis in Machinese analysers, please see the Machinese Language Model manual.

Morphology

Morphological information tells the details of the word forms used in the text.

Please note that Machinese Phrase Tagger shows only a limited number of morphological features while Machinese Syntax offers much more detailed analysis. For example, Machinese Phrase Tagger gives in the prose output mode the following analysis for the French language sentence 'C'est la vie.':

While Machinese Syntax provides the following analysis, which includes many morphological features which Machinese Phrase Tagger omitted:

Detailed description of these analyses can be found in the Machinese Language Model manual.

Syntax

Whereas part of speech and morphology give information on individual words, syntax describes the information about relations between words within phrases or sentences. Machinese Phrase Tagger tells what syntactic function each word has. Machinese Syntax gives more detailed description, which provides details on what role the syntactic relation has and which words are adjoined by this relation.

If you compare the analyses shown in the morphology chapter above, you can see that the fourth field in the Machinese Syntax output lists the abbreviated syntax relation name and number designating which word the relation points to.

Detailed description of what syntactic analyses Machinese analysers will produce can be found in the Machinese Language Model manual.

Noun Phrases

In information retrieval applications, the interesting part of the text is usually nouns or noun phrases, as these describe which subjects and objects the text is about. Single nouns are obviously easy to pick from text (see information on part of speech), but often noun phrases tell more as they combine the information of adjacent words. Machinese Phrase Tagger makes finding noun phrases easy as it includes noun phrase detection, which marks where a noun phrase starts, ends and which words in between belong to that noun phrase.

Take for example sentence 'The bloody failure of the police state run by Ceausescu could have become a drawn-out civil war.' Now, when analysed with Machinese Phrase Tagger, the default output format looks like following:

In the output, you can see the following tags, which denote parts of noun phrase constructs:

NP-Single

a single noun.

NP-First

first word of multi-word noun phrase

NP-Last

last word of multi-word noun phrase

NP-Internal

if a noun phrase is longer than three words, the words between the first and the last of the noun phrase are marked with this tag.

Following these tags, you can extract the three noun phrases of the above sentence: 'bloody failure of the police state', 'Ceausescu', and 'drawn-out civil war'.

Proper Noun Detection

Often it is desirable to find names from a text. The solution is to use the information from Machinese Phrase Tagger's proper noun detection. The output formats will include the proper noun information alongside with the morphological information. The tag used to mark proper nouns is Prop and in the prose output the same is described with the denotation "proper noun". When combined with the noun phrase information, it is possible to pick up multiword names, for example pairs of first and last names.