Hello,
Currently PostgreSQL doesn't
support full text search natively for many Asian languages such as Chinese, Japanese and others. These languages are used by a
large portion of the population of the world.
The
two key modules that could be modified to support Asian languages are
the full text search module (including tsvector) and pg_trgm.
I would like to propose that this support be added to PostgreSQL.
For full text search, PostgreSQL could add a new parser (https://www.postgresql.org/docs/9.2/textsearch-parsers.html)
that implements ICU word tokenization. This should be a lot more easier
than before now that PostgreSQL itself already includes ICU
dependencies for other things.
Then allow the
ICU parser to be chosen at run-time (via a run-time config or an option
to to_tsvector). That is all that is needed to support full text search
for many more Asian languages natively in PostgreSQL such as Chinese,
Japanese and Thai.
For example Elastic Search implements this using its ICU Tokenizer plugin:
https://www.elastic.co/guide/en/elasticsearch/guide/current/icu-tokenizer.html
https://www.elastic.co/guide/en/elasticsearch/guide/current/icu-tokenizer.html
Some information about the related APIs in ICU for this are at:
Another
simple improvement that would give another option for searching for
Asian languages is to add a run-time setting for pg_trgm that would tell
it to not drop non-ascii characters, as currently it only indexes ascii
characters and thus all Asian language characters are dropped.
I emphasize 'run-time setting' because when using PostgreSQL via a Database-As-A-Service service provider, most of the time it is not possible to change the config files, recompile sources, or add any new extensions.
PostgreSQL is an awesome project and probably the best RDBMS right now. I hope the maintainers consider this suggestion.
Best Regards,
Chanon
