Unicode normalization

[dstillman]

Normalization is needed for correct search, sorting, and disambiguation.
​http://forums.zotero.org/discussion/12684/special-character-search/

Also for BibTex export:
​http://groups.google.com/group/zotero-dev/browse_thread/thread/6f6d5c2eec1cc9ae?hl=en

The forum discussion covers possible solutions.

[dstillman]

Also: ​http://www.xulplanet.com/references/xpcomref/ifaces/nsIUnicodeNormalizer.html

[simon]

Should we be doing this at the DB or ingester level instead? Otherwise, output will have to be normalized in reports, bibliography, etc. (if we care)

[dstillman]

Good call. Assuming Firefox doesn't do this itself when entering data in a text box (in which case it could be just in the ingester), doing it in the data layer probably makes more sense. Since normalization is supposed to be idempotent, it shouldn't be a problem even if it's run every time a field is changed.

The downside to doing it on-change is that existing data wouldn't be normalized, but 1) the workaround (triggering a re-saving of the field) would be pretty easy and 2) we could even provide a hidden XUL page to run a normalize operation on all data, though it'd take an awfully long time.

[ajlyon]

Reported again at ​http://forums.zotero.org/discussion/12684/special-character-search/ , complete with suggested solutions.

[egh]

FYI - a trick I have used to good effect to strip "accents" from an index is (in python):

def strip_accents(s):

return .join((c for c in unicodedata.normalize('NFD', unicode(s)) if unicodedata.category(c) != 'Mn'))

That is, normalize in NFD, then remove each character in class "Mark, non-spacing".

Of course this is only to add tokens to your index for full text search.

And of course this is a different issue than just normalization to avoid problems with composed v. decomposed characters, but it is what was originally raised in the form post.

[dstillman]

​https://github.com/zotero/zotero/issues/542