razdel

— rule-based system for Russian sentence and word tokenization.

Usage

Installation

supports Python 3.7+ and PyPy 3.

Documentation

Materials are in Russian:

• Razdel page on natasha.github.io
• Razdel section of Datafest 2020 talk

Evaluation

Unfortunately, there is no single correct way to split text into sentences and tokens. For example, one may split

into three sentences

["«Как же так?!", "Захар...»", "— воскликнут Пронин."]

while

razdel

splits it into two

["«Как же так?!", "Захар...» — воскликнут Пронин."]

. What would be the correct way to tokenizer

т.е.

? One may split in into

т.|е.

,

razdel

splits into

т|.|е|.

.

tries to mimic segmentation of these 4 datasets: SynTagRus, OpenCorpora, GICRYA and RNC. These datasets mainly consist of news and fiction.

razdel

rules are optimized for these kinds of texts. Library may perform worse on other domains like social media, scientific articles, legal documents.

We measure absolute number of errors. There are a lot of trivial cases in the tokenization task. For example, text

is not non-trivial, one may split it into

чуть|-|чуть|?|!

while the correct tokenization is

чуть-чуть|?!

, such examples are rare. Vast majority of cases are trivial, for example text

в 5 часов ...

is correctly tokenized even via Python native

str.split

into

в| |5| |часов| |...

. Due to the large number of trivial case overall quality of all segmenators is high, it is hard to compare differentiate between for examlpe 99.33%, 99.95% and 99.88%, so we report the absolute number of errors.

— number of errors per 1000 tokens/sentencies. For example, consider etalon segmentation is

что-то|?

, prediction is

что|-|то?

, then the number of errors is 3: 1 for missing split

то?

+ 2 for extra splits

что|-|то

.

— seconds taken to process whole dataset.

,

aatimofeev

and others a defined in naeval/segment/models.py, for links to models see Naeval registry. Tables are computed in naeval/segment/main.ipynb.

Tokens

Sentences

Support

• Chat — https://telegram.me/natural_language_processing
• Issues — https://github.com/natasha/razdel/issues
• Commercial support — https://lab.alexkuk.ru

Development

Dev env

Test

Release

errors on

syntag

Non-trivial token tests

Update integration tests

and

moses

diff

performance