master
some old changes to bad sentences list
(migration) renamed text_forms to tokens
added some exceptions to sentence split checker, suddenly
optimize tokenizer learning, should be about 2x faster
cronjob wrapper, take 2
set --data_dir explicitly argument instead of using buggy FindBin
fixed bug in jobs dispatcher
use tabs in tokenize.php
[tokenizer] add some bad sentences and exceptions
[tokenizer] add reading limit in train.php