master
[geodisambig] C msgpack serialization dependency
[fix] memcpy in geohash, closes #707
#501. Fixed build issues on MSYS on Windows.
Updated linenoise to be MSys2/MinGW compatible. Updated address_parser app to use the defined libpostal api and not include internal components directly. Removed windows src Makefile as it is now the same as the standard one.
Removing console colors for Windows builds.
[geodisambig] adding MurmurHash3, used by the Bloom filter
[build] pulling in change from parser-data that allows user to pass CFLAGS
Removing EXPORT statements from all source files and most header files, leaving only the exports for the main API in libpostal.h. Modified Makefiles so that all the test apps build without having extra functions exported from libpostal.
[build] adding new source files to Makefile for the lieu APIs
[fix] possible acronym for single toke phrases if it's a directional
[dedupe] adding a function to acronyms module to detect existing/known acronyms like MS for middle school, HS for high school, etc. Forms like MS have to be deined in the dictionaries specifically but any acronym written like M.S. will be detected as such by the tokenizer
[expand] adding method for checking phrase is in multiple dictionaries, and a helper method for determining whether an address phrase has a canonical interpretation
[expansion] using easier-to-access data structure for address dictionaries
[auto][ci skip] Adding data files from Travis build #381
[fix] max dictionaries is now 5, weird that that wasn't committed by the Travis build
[fix] clear context->separators on each parse before adding to it, fixes #676
[parser] adding label constants to address_parser header
Modifed the libpostal API to add an extra function libpostal_parser_print_features to toggle debugging info. Updated address_parser app to use the new function.
[fix] handle multiple separators (like parens used in https://www.openstreetmap.org/node/244081449). Creates bad trie entries otherwise, which affect more than just that toponym
[parser] header changes for the data set struct
[api] doing this now since we're bumping a major version. Using a libpostal prefixes for all public header functions and definitions
Add portable way of formatting khint_t type (from klib)
[parser] using new zeros API in averaged_perceptron.c
[parser] Averaged perceptron training
[parser] moving feature printing to averaged perceptron tagger, taking advantage of trie prefix-sharing in feature incorporating previous tags
Fix log_* formats which expect long long uint but receive uint64_t.
[parser] learning a sparser averaged perceptron model for the parser using the following method:
[fix] calloc instead of malloc when performing initialization on structs that may fail halfway and need to clean up while partially initialized (calloc will set all the bytes to zero so the member pointers are NULL instead of garbage memory)
[fix] Adding stdint.h include to most of the header files for portability
[utils] cartesian product iterator for grid search during model selection
[utils] adding a chunked shuffle as the concatenated file sizes may get larger than memory
[utils] hash_get is no longer a string-only function, can be used for generic hashtables
[constants] max country code length
[fix] removing log error for sequences of length 0
[parser/crf] adding runtime CRF tagger, which can be loaded/used once trained. Currently only does Viterbi inference, can add top-N and/or sequence probabilities later
[revert] reverting the commits from #578, leaving header file in repo for the moment
[parser/crf] adding the beginnings of a linear-chain Conditional Random Field implementation for the address parser.
[parser/crf] adding crf_trainer, which can be thought of as a "base class" as much as that's possible in C, for creating trainers for the CRF. It doesn't deal with the weights or their representation, just provides an interface for keeping track of string features and label names, and holds the crf_context
[utils] adding a function for checking if files exists (yay C), or at least the closest agreed-upon method for it (may return false if the user doesn't have permissions, but that's ok for our purposes here)
[parser/crf] adding an initial training algorithm for CRFs, the averaged perceptron (FTW!)
[dedupe] account for missing ordinal suffixes in Soft-TFIDF deduping i.e. to count 1st Place and 1 Plce as the same where there might be a misspelling and the phrase wouldn't match under exact expansions
[api] adding APIs for getting default options and using a consistent naming convention
[similarity] using NULL-terminated varargs in double metaphone instead of specifying the number of arguments, should be more maintainable
[similarity] adding basic double metaphone implementation
[fix] ifdef includes around stndup
[expand] adding a few of the address phrase checks to the expand header
avoid UB in bit shifts
[utils] adding non-branching sign functions
[optimization] for the FTRL and SGD optimizers, use the new *_array_sum_sq function to do L2 regularization, vs. the L2 norm which will use the linear algebra meaning
[optimization] implemented Google's FTRL-Proximal, adapted for the multiclass/multinomial case. It is L1 and L2 regularized, and should both encourage sparsity with the L1 penalty while being robust to collinearity of features due to the L2 penalty. Ref: https://research.google.com/pubs/archive/41159.pdf
[fix] DICTIONARY_STREET_NAME applies only to the street component, unlike synonyms, which would apply to any compnoent. This makes street names a good place to add synonyms found in streets that are not exactly thoroughfare types i.e. could not be removed from the string and have it retain more or less the same meaning.
[dictionaries] Making new component for near/nearby prepositions
[geodisambig] Shaving a few hundred more megabytes off of the geodb by only adding a single geohash prefix and not indexing the neighbors (query can use its neighbors)
[geodb] Additional filenames and struct members in geodb.h
[merge] merging master
[fix] Check the return of malloc() in geonames.c
[geodb] Adding separate bitset for geonames place types and using NFC normalization instead of NFD (requires retraining)
[geonames] Re-generating postal/geonames fields headers
[parser] using a bipartite graph (indptr + indices) to represent postal code<=>admin relationships instead of a set of 64-bit ints. Requires |V(postal codes)| + |E| 32 bit ints instead of |E| 64 bit ints. Saves several hundred MB in file size and even more space in memory because of the hashtable overhead
[similarity] adding a string array version of Jaccard similarity that creates the string sets internally for convenience
[api] Simple JSON encoding for strings, UTF-8 rather than Unicode
[api/compat] PR #465 redefined the language classifier response struct in the API and was casting between incompatible pointer types. Using the exported struct throughout.
[log] log the offending line if token count does not match in language_classifier_io
[language_classification] Automatic hyperparameter optimization using either the cross-validation set or two distinct subsets of the training set
Fix log_* formats which expect size_t but receive uint32_t.
[languages] adding replace_hyphens and split_alpha_from_numeric in language classifier input normalization
[language_classifier] Features for address languages classification, quadgrams for most languages, unigrams for ideographic characters, script for single-script languages like Thai, Hebrew, etc.
Merge pull request #391 from edding/fix_memory_leak
[near_dupes] exposing name_word_hashes directly in the API
[build] adding libpostal_setup_datadir, libpostal_setup_parser_datadir, libpostal_setup_language_classifier_datadir functions for configuring the datadir at runtime
Fix dash syntax error in libpostal_data
[math] using new matrix methods in softmax
[math] Generic dense matrix implementation using BLAS calls for matrix-matrix multiplication if available
Revert format regression introduced in ecd07b1
[classification] removing regularization update from gradient computation in logistic regression, as that's now handled by the optimizer
[classification] correcting cost functions in SGD and FTRL for use in parameter sweeps
[expand] adding a --root flag to the libpostal script for root-only expansion
[fix] memcpy in aligned vector/matrix resize needs to know the original size. Since this is an internal static function and does not affect client APIs, changing it to aligned_malloc, aligned_free, and aligned_resize, which takes the original pointer size as well as the new. On Windows it simply passes the pointer, new size, and alignment through _aligned_realloc, whereas on other platforms only the aligned_malloc is for new_size bytes and uses old_size bytes for memcpy
[fix] NULL check
[features] Functions for dealing with minibatches
[utils] cstring_array_terminate, moving msgpack_utils to separate file
[dedupe/test] adding remaining options to near_dupe_test
[ngrams] changing args to ngrams
[fix/utf8] reviewed and fixed all points where utf8proc_iterate is called and may return an error which can cause the iteration not to make forward progress. This includes fixing a bug where injecting invalid UTF-8 through a series of HTML-encoded codepoints can cause the C library to hang. Note: we're not fixing all the garbage encoding in the world, so if encoding is bad the output of expand_address may not be useful but it won't hang. Fixes #448
[normalize/api] exposing normalize_string_languages and normalized_tokens_languages to the API for pre-normalizing numeric expressions at tokenization time
[numex] helper function to retrieve ordinal suffix lengths from a tokenized string for use in deduping
[auto][ci skip] Adding data files from Travis build #284
[numex] Separating rules from keys for Linux gcc compilation
[numex] adding ordinal suffixes themselves to the numex trie so they can be removed from strings
[optimization] moving regularization methods to their own module
#511. Fixed C compilation errors for the latest versions of Mac OS X
[utils] adding default chunk size to shuffle.h
Fix segmentation fault in soft_tfidf.c
[fix] sparse_matrix_utils also had an incompatible pointer
[utils] sparse_matrix_add_unique_columns_alias, adds the actual column indices to hashtable/array and aliases those in the table from 1 to N (where N is the number of unique columns in this batch). This way it's compatible with smaller matrices of batch weights.
[similarity] adding a multi-word alignmnet algorithm for streets and names like "de la cruz" vs. "dela cruz" or "Oceanwalk Ter" vs. "Ocean Walk Ter"
[dedupe] for near-dupe hashing, remove whitespace from root expansions so something like "Ocean Walk Dr" and "Oceanwalk Dr" will have a chance of matching downstream
[fix] conditional compilation for strndup and drand48 for Windows, using config.h
[fix] add #include guard to tagger.h
[merge] merging commit from v1.1
Fix bug in strndup fix for windows. Move all includes out of headers and into code for strndup.h and move it to be the last include.
[expand] adding improvements to root expansions (using possible phrase roots even if they're abbreviated e.g. "E Ctr St", adding special valid components check for root expansions beyond what's stored in the build address dictionaries), removing spaces before checking unique strings, only splitting numeric from alpha in the case of non-ordinals, using cstring_array internally and char ** in the public API
[fix] double-free on a non-lowercased transliterator name
[transliteration] regenerate transliteration data
[transliteration] Using revisit strings instead of keeping a backtrack count so we don't have to later map logical characters to the actual string, removing any duplicate keys in the table builder so that if any rules happen to overlap within a step, the first will take precedence
[transliteration] regenerated transliteration_scripts_data.c
[logging] converting most of the steps in building the transliteration table to use debug logging
[fix] initialize next_id by zero in trie_new_from_cstring_array function
[mv] Moving trie_new_from_hash to a module
[unicode] Regenerating unicode_script_data.c
[transliteration] Regenerating transliteration data files
[fix] Windows name