<token-definition>
)Definitive list of key terms used to name standard token definitions.
Master location: http://textalign.net/release/TAN-2018/TAN-key/token-definitions.TAN-key.xml
Table 9.11. TAN keywords for types of token definitions
keywords (optional values of @which ) | pattern | Comments |
---|---|---|
|
| General tokenization pattern for any language, words only. Non-letters such as punctuation are ignored. |
|
| General tokenization pattern for any language, treating not only series of letters as word tokens but also individual non-letter characters (e.g., punctuation). |
|
| General tokenization pattern for any language, treating any contiguous run of nonspace marks as a word. |