TAN depends upon a set of relatively stable technologies. Those technologies and the underlying terminology are very briefly defined and explained below, with particular attention to interpretive decisions that have been adopted by TAN validation rules. References to further reading will lead you to better and more thorough introductions.
Unicode is the worldwide standard for the consistent encoding, representation, and exchange of digital texts. Stable but still growing, Unicode is intended to represent all the world's writing systems, living and historical. Maintained by a nonprofit organization, the Unicode standard allows us to share texts in any alphabet and reliably share that data with other people, independent of individual fonts.
With more than 128,000 characters, Unicode is almost as complex as human writing itself. The entire sequence of characters is divided into blocks, each one reserved, more or less, for a particular alphabet or a set of characters that share something in common. Within each block, characters may be grouped further. Each character is assigned a single codepoint.
Because computers work on the binary system, codepoints have been numbered according to the related hexadecimal system (base 16), which uses the digits 0 through 9 and the letters A through F. (The number 10 in decimal is A in hexadecimal; decimal 11 = hex B; decimal 17 = hex 10; decimal 79 = hex 4F.) It is helpful to think of Unicode as a very long ribbon sixteen squares wide, a glyph in each square. This is illustrated nicely in this article. Each position along the width is labeled with a hexadecimal number (0-9, A-F) that always identifies the last digit of a character's code point value.
It is common to refer to Unicode characters by their value or their name. The value customarily starts "U+" and continues with the hexadecimal value, usually at least four digits. The official Unicode name is usually given fully in uppercase. Examples:
Table 3.1. Unicode characters
Character | Unicode value | Unicode name |
---|---|---|
" " (space) | U+0020 | SPACE |
® | U+00AE | REGISTERED SIGN |
ю | U+044E | CYRILLIC SMALL LETTER YU |
TAN validation rules require all data to be normalized according to the Unicode NFC algorithm. Any text in a TAN file that is not NFC normalized will be marked as invalid.
When the characters U+200D ZERO WIDTH JOINER and U+00AD SOFT HYPHEN occur at
the end of a leaf <div>
,
perhaps followed by white space that will be ignored (see below), processors
will assume that the character is to be deleted, and when combined with the
next leaf div, no intervening space should be allowed. Furthermore, because
these characters are difficult to discern from spaces and hyphens, any output
based on the character mapping of the core functions should replace these
characters with their XML entities, ‍
and
­
.
At the core level of conformance, Unicode does not dictate whether combining characters (accents, modifying symbols) should be counted independently or as part of a base character, nor does the family of XML languages. In most circumstances, this point is negligible. But it affects regular expressions and XPath expressions (see below).
Two of the class 2 formats allow the counting of characters. Such counting
is assumed to be made exclusively of non-combining characters, defined as the
regular expression [^\p{M}]
. Any numerical reference made in a TAN
file to an individual character will be found by counting only non-combining
characters. When the nth character is requested, TAN functions will return the
nth base character along with any combining characters that immediately follow.
TAN rules stipulate that combining characters must have a preceding base
character. Any <div>
that
starts with a combining character will be marked as invalid. See also Regular Expressions and Combining Characters.
Because TAN is focused not at all on appearance, the following characters will generate an error if found in a TAN file:
U+00A0 NO-BREAK SPACE
U+2000 EN QUAD
U+2001 EM QUAD
U+2002 EN SPACE
U+2003 EM SPACE
U+2004 THREE-PER-EM SPACE
U+2005 FOUR-PER-EM SPACE
U+2006 SIX-PER-EM SPACE
U+2007 FIGURE SPACE
U+2008 PUNCTUATION SPACE
U+2009 THIN SPACE
U+200A HAIR SPACE
Defined by the W3C, the eXtensible Markup Language (XML) is a machine-actionable markup language that facilitates human readability.
For a basic introduction to XML see Chapter 2, Starting off with the TAN Format.
Validation files are found here: http://textalign.net/release/TAN-2018/schemas/
.
Each TAN file is validated by two types of schema files, one dealing with major rules concerning structure and data type (written in RELAX-NG) the other with very detailed rules (written in Schematron).
The RELAX-NG rules are written primarily in compact syntax
(.rnc
), and then converted to the XML syntax
(.rng
). For TAN-TEI, the special format One Document Does it all
(.odd
) is used to alter the rules for TEI All.
The Schematron files are generally quite short. The primary work is done by a large function library written in XSLT. For more on this process, see the section called “Doing things with TAN files”.
Some validation engines that process a valid TAN-compliant TEI file may
return an error something like conflicting ID-types for attribute "who"
of element "comment" from namespace "tag:textalign.net,2015:ns"
. Such
a message alerts you to the fact that by mixing TEI and TAN namespaces, you
open yourself up to the possibility of conflicting xml:id
values.
It is your responsibility to ensure that you have not assigned duplicate
identifiers. Very often, it is possible for you to configure an XML editor to
ignore this discrepancy. (In oXygen XML editor go to Options > Preferences... >
XML > XML Parser > RELAX NG and uncheck the box ID/IDREF.)
By default in XML, unless otherwise specified, consecutive space characters (space, tab, newline, and carriage return) are considered equivalent to a single space. This gives editors the freedom they need to format XML documents as they like, for either human readability or compactness.
All TAN formats assume space normalization, with an extra caveat, namely,
that some space is assumed to exist between adjacent leaf <div>
s, even if no text node
intervenes. This behavior is overridden if the first leaf <div>
ends in the soft hyphen or
the zero width joiner; see the section called “Unicode characters with special interpretation”).
The TAN format does not stipulate how space-only text nodes should be
interpreted. It is up to processors to analyze the relevant <div-type>
to infer an
appropriate type fo white-space separator.
If retention of multiple spaces is important for your research, then TAN formats may not be appropriate, since TAN is not intended to replicate the appearance of a scriptum. Pure TEI (and not TAN-TEI) might be a practical alternative, since it allows for a literal use of space, and encourages XML files that try to replicate the appearance of a scriptum.
For more on white space see the W3C recommendation.
Many familiar text formats such as TEI, HTML, and Docbook allow what is called mixed content—a mixture of elements and nonspace text as siblings. The TAN formats, aside from TAN-TEI, are committed to a non-mixed content model. Nonspace text nodes and elements are never siblings. The practical effect of this decision is that indentation may be applied to a TAN file as one wishes, and space text nodes may be inserted between any two adjacent elements, without affecting the meaning.
To specify in a class 1 file that two adjacent leaf <div>
s should have no intervening
space, see the section called “Unicode characters with special interpretation”.
XML allow users to develop vocabularies of elements as they wish. One person
may wish to use the element <bank>
to refer to financial
institutions, another to rivers. Perhaps someone wishes to mention both rivers
and financial institutions in the same document. XML was designed to allow
users to mix vocabularies, even when those vocabularies use synonymous element
names. This means that anyone using <bank>
must be allowed to
specify exactly which vocabulary is being used. Disambiguation is accomplished
by associating IRIs (see the section called “Identifiers and Their Use” below) with the
element names. The actual full name of an element is the local name plus the
IRI that qualifies its meaning, e.g.,
bank{http://example1.com/terms/}
and
bank{http://example2.com/terms/}
.
The relationship between the element name and the IRI is analogous to that between a person's given name and their family name. The IRI—the family name—is called the namespace—not an ideal term, but the one that has been adopted. Think of the namespace as the family name for a group of elements.
Namespaces look a lot like attributes (they aren't). They take the form
xmlns="http://example1.com/terms/"
(defining the default
namespace) or xmlns:[PREFIX]="http://example2.com/terms/"
(defining a namespace that has been assigned a particular prefix) placed inside
an opening tag. For example, <bank
xmlns="http://example1.com/terms/">...</bank>
states, in effect,
the namespace for <bank>
and the default namespace for all
descendants (it can be explicitly overridden).
Different types of <bank>
can be mixed through
namespaces:
<bank xmlns="http://example1.com/terms/"> <bank xmlns="http://example2.com/terms/"> ... </bank> </bank> <bank xmlns="http://example1.com/terms/" xmlns:e2="http://example2.com/terms/"> <e2:bank > ... </e2:bank> </bank> <e1:bank xmlns:e1="http://example1.com/terms/" xmlns:e2="http://example2.com/terms/"> <e2:bank > ... </e2:bank> </e1:bank>
The TAN namespace is tag:textalign.net,2015:ns
. The recommended
prefix is tan.
The TAN-TEI format uses as its default the TEI namespace, http://www.tei-c.org/ns/1.0, normally given the prefix tei.
The Text Encoding Initiative (TEI) is a collection of XML rules for the representation of texts in digital form. Developed and maintained by a consortium of scholars and scholarly organizations, TEI includes not only a library of schemas, but guidelines and stylesheetsmore. The TEI Guidelines have been widely used by libraries, museums, publishers, and individual scholars to present texts for online research, teaching, and preservation. In addition to the Guidelines themselves, the Consortium provides a variety of resources and training events for learning TEI, information on projects using the TEI, a bibliography of TEI-related publications, and software.
Note | |
---|---|
Taken from the TEI website http://www.tei-c.org/index.xml, accessed 2017-05-21. |
Any TAN-T module can be easily cast into a TEI file, although much of the computer-actionable semantics will be lost in the process. Likewise, a TEI file can be converted to TAN-T, but there is a greater risk of loss of content, particularly in the header, since the non-TEI TAN formats are restricted to a small subset of TEI tags.
TAN-TEI is TAN's TEI extension, based on an ODD file that is in the same directory as the rest of the schemas. TAN-TEI schemas are generated on the basis of the official TEI All schema that is available at the time of release.
For more about the strictures placed upon the TEI All schema see the section called “Transcriptions Using the Text Encoding Initiative (<TEI>
)”. See also Chapter 4, Patterns and Structures Common to All TAN Encoding Formats and Chapter 5, Class-1 TAN Files, Representations of Textual Objects (Scripta).
Being written purely in XML technologies, TAN adopts its data types, e.g., strings, booleans, and so forth, from the official specifications made by the W3C. The following data types require some special comments.
TAN adopts for language identification Best Common Practices (BCP) 47, which
standardizes identifies for languages and scripts. For most users of TAN, this
will be a simple three-letter abbreviation, sometimes supplemented with a
hyphen and an abbreviation designating a script or regional subtag. For
example, eng
, eng-UK
, and eng-UK-Cyrl
refer, respectively, to English (in general), English from the United Kingdom,
and English from the United Kingdom written in the Cyrillic script. As a
general rule, values of this type should begin with a three-letter language
code, preferably lowercase.
ISO codes for human languages appear in @xml:lang
and <for-lang>
. The former states
what language the enclosed text is in. The second indicates that some statement
or claim is being made about a specific language language. For example,
<for-lang>
in the
context of a TAN-mor file indicates which languages the file was written
for.
For more information, see one of the following:
BCP 47 official specifications
BPC 47 technical details
TAN adopts the standardized ISO form of dates and date-times, as interpreted by XML data types. These begin with years (the largest unit) and ends with days, seconds, or fractions of seconds (the smallest).
The simplest date takes this form: YYYY-MM-DD
. If a time is
included, it is specified by continuing the string, first with a T
(for time) then the form hh:mm:ss.sss(Z|[-+]hh:mm)
. For example,
the following is 2016-09-20T20:38:27.141-04:00
is an ISO date-time
for Tuesday, September 20, 2016 at 8:38 p.m. on the Eastern Time Zone.
More reading:
The acronyms for identifiers, and the meanings of those acronyms, can be mystifying. Here is a synopsis:
IRI: Internationalized Resource Identifier, a generalization of the URI system, allowing the use of Unicode; defined by RFC 3987
URI: Uniform Resource Identifier, a string of characters used to identify a name or a resource; defined by RFC 3986
URL: Uniform Resource Locator, a URI that identifies a Web resource and the communication protocol for retrieving the resource.
URN: Uniform Resource Name, a term that
originally referred to persistent names using the urn:
scheme, but is now applied to a variety of systems that have registered
with the IANA. URNs are generally best thought of as a subset of
URIs.
UUID: Universally Unique Identifier, a
computer-generated 128-bit number used to assign identifiers to any
entity. UUIDs can be built into a URN by prefixing them with
urn:
.
The TAN format generally prefers to refer to IRIs.
See also the section called “Tag URNs”.
Identifiers are used in many contexts for many purposes. One such purpose is called Linked Open Data (LOD) or the Semantic Web, which relies upon a very simple data model called Resource Description Framework (RDF), a family of World Wide Web Consortium (W3C) specifications originally designed as a data model for metadata.
RDF was designed to be a data model to support general assertions. The model rests upon the concept of a statement, made of three parts: subject, predicate, and object. Subjects and predicates take identifiers that act as names of things. The object may take an identifier or just data. The idea behind LOD is that os we begin to use the same URLs for the same concepts, then independently created datasets can be combined and compared. The entire collection of RDF statements on the web allow inferences not possible on the project level.
These URL identifiers look like a web page address (e.g.,
http://...
), but are first and foremost names for things
(the term "Resource"—the R in RDF—is a clumsy way to refer to any person,
place, concept—anything at all). Ideally, those URLs will still name those
things after the domain name expires and the web resource cannot be found.
Much of TAN can be converted to RDF statements. In fact, TAN may be one of the most human-friendly way to read and write RDF. Compare, for example, this snippet (taken from http://linkeddatabook.com/editions/1.0/), written in Turtle syntax, ...
1 @prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> . 2 @prefix foaf: <http://xmlns.com/foaf/0.1/> . 3 4 <http://biglynx.co.uk/people/dave-smith> 5 rdf:type foaf:Person ; 6 foaf:name "Dave Smith" .
...with the TAN equivalent:
<person xml:id="dsmith"> <IRI>http://biglynx.co.uk/people/dave-smith</IRI> <name>Dave Smith</name> </person>
In this case TAN and RDF are converted losslessly. But in many other
cases, TAN statements cannot be reduced to the RDF model. This happens most
often in the context of <claim>
, which is designed to allow scholarly
assertions and claims that are difficult or impossible to express in RDF.
For example, RDF does not allow one to say "Person X is not the author of
text Y."
TAN claims have adapted the core concepts behind RDF to cater to scholarly needs. For more details see the section called “Division-Based Annotations and Alignments (<TAN-A-div>)”.
TAN files make extensive use of tag URNs (see the section called “Identifiers and Their Use”). In fact, TAN's namespace is a tag URN (the section called “Namespaces”). A tag URN has two parts:
Namespace.
tag:
+ an e-mail address or domain name owned by the
person or organization that has authorized the creation of the TAN
file + ,
+ an arbitrary day on which that address or
domain name was owned. The day is expressed in the form
YYYY-MM-DD
, YYYY-MM
, or
YYYY
. A missing MM
or DD
is
implicitly assigned the value of 01
.
Name of the TAN file.
:
+ an arbitrary string (unique to the namespace chosen)
chosen by the namespace owner as a label for the entire file and
related versions. It need not be the same as the filename stored on a
local directory. You should pick a name that is at least somewhat
intelligible to human readers.
Although you may use any tag URN coined by someone else, you may create a tag URN only if you are the owner of that URN's namespace.
Great care must be taken in choosing the IRI name, because you are the sole guarantor of its uniqueness. It is permissible for something to have multiple IRIs, but never acceptable for an IRI to name more than one thing. It is a good practice to keep a master checklist of IRI names you have created. If you find yourself forgetting, or think you run the risk of creating duplicate IRI names, you should start afresh by creating a new namespace for your tag URNs, easily done just by changing the date in the tag URN namespace.
Example 3.1. TAN IRI names
tag:jan@example.com,1999-01-31:TAN-T001 tag:example.com,2001-04:hamlet-tan-t tag:evagriusponticus.net,2014:tan-a-lm:Evagrius_Praktikos_grc_Guillaumonts tag:bbrb@example.org,1995-04-01:pos-grc
The first example comes from someone who owned the email address
jan@example.com
on January 31, 1999 (at the stroke of
midnight, Universal Coordinated Time). The other examples follow a
similar logic. The namespace of the second and third examples are tied to
the owners of specific domain names, not those of email addresses. The
2014
in the fourth example is shorthand for the first
second of January 1, 2014.
The TAN encoding format has chosen tag URNs over URLs for several reasons:
Permanence. Authors of TAN data are creating files that are meant to be relevant for decades and centuries from now, well after specific domain names have changed ownership or fallen into obsolesence, and well after the creators are dead. URLs are not built for such permanence.
Responsibility. The TAN format requires every piece of data to be attributable to someone (a person, organization, or some other agent). A tag URN implies who was responsible for creating the URN.
Accessibility. Tag URNs can be made by anyone who has an email address. No one has to register with any central authority. You can begin naming anything you want, any time you want, without seeking anyone's approval.
Ease. Tag URNs are easier to use than, say, http-form URLs, as recommended by RDF (see the section called “Resource Description Framework (RDF) and Linked Open Data”). Many potential TAN authors never have owned a domain name, and never will. Further, many of those who do own domain names cannot or do not wish to configure and maintain servers to administer the referral mechanisms upon which the semantic web depends.
Disambiguation of name and location. In the semantic web, conflation of name with a location to resolve it is considered a virtue because the single string does double duty, both naming the resource and pointing to a location where more can be learned. But this conflation is unhelpful in the TAN context. TAN files are meant to be distributed widely, and not rely upon a single location. And URLs are in common parlance interpreted as locations for data, not as names for things. Tag URNS don't confuse users by looking like locations. This upholds a principle that is common in scholarly citation, namely, that one should always distinguish the name of a resource from where it might be found.
Further reading:
RFC 4151, the official definition of tag URNs
Regular expressions are patterns for searching text. The term regular here does not mean ordinary. Rather, it derives from Latin regula, and points to a rule-based syntax that provides patterns for finding and replacing text. Regular expressions come in different flavors, and have several layers of complexity. TAN regular expressions adhere closely to the recommendation of XSLT 3.0 (XML Schema Datatypes plus some extensions), and outlined in XPath Fuctions 3.0.
Caution | |
---|---|
XML Schema Datatypes define regular expressions differently than do Perl,
one of the most common forms of regular expression. For example, the pipe
symbol, |, is treated as a word character in XML regular expressions
( Word characters ( Non-word characters ( Some of these choices may seem counterintuitive or wrong. But at this point it does not matter. The distinction is a legacy that will remain in place. It is advisable to familiarize yourself with decisions that, in some respect, are arbitrary. |
A regular expression search pattern is treated just like a conventional search
pattern until the computer reaches a special escape character: . [ ] \ | - ^
$ ? * + { } ( )
. Here is a brief key to how characters behave in regular
expressions, provided they are not in square brackets (on which see the
recommended reading below):
Table 3.2. Special characters in regular expressions
Symbol | Meaning |
---|---|
$ | end of line |
. | any character |
| | or (union) |
^ | start of line |
? | zero or one |
* | zero or more |
+ | one or more |
[ ] | a class of characters |
( ) | a group |
\w | any word character |
\W | any nonword character |
\s | any of the four standard spacing characters: space (U+0020), tab (U+0009), newline (U+000A), carriage return (U+000D) |
\S | anything not a spacing character |
\d | any digit (0-9) |
\D | anything not a digit |
\p{IsGujarati} | any character from the Unicode block named Gujarati |
\\ | backslash (the backslash alone suggests that the next character is a special character) |
\$ | dollar sign |
\( | opening parenthesis |
\[ | opening square bracket |
Some examples:
Table 3.3. Examples of Regular Expressions
Expression | Meaning | What the expression matches when applied to "Wi-fi, good. A_hem* isn't!" |
---|---|---|
^.+$ | one whole line of characters | "Wi-fi, good. A_hem* isn't!" |
[ae] | a or e | "e" |
[a-e] | a, b, c, d, or e | "d", "e" |
[^ae]+ | one or more characters that are anything except a or e | "Wi-fi, good. A_h", "m* isn't!" |
.i | any character followed by i. | "Wi", "fi", " i" |
(.i) | when a character followed by an i is found treat it as a capture group (used only in a search pattern) | "Wi", "fi", " i" |
$1 | first capture group (used only in a replacement pattern, and corresponds to the sequence of capture groups in the search pattern) | In the example above, each match corresponds to $1 |
[aeiou]\w* | any lowercase vowel along with every word character that follows | "i", "i", "ood", "em", "isn" |
[t*]. | any t or * and the following character | "* ", "t!" Note that the asterisk, if inside a character class, acts as itself. |
\s+ | match one or more space characters | " ", " ", " " |
\w+ | match one or more word characters | "Wi", "fi", "good", "A_hem", "isn", "t" |
\W+ | match one or more nonword characters | "-", ", ", ". ", "* ", "'", "!" |
[^q]+ | one or more characters that are not a q | "Wi-fi, good. A_hem* isn't!" |
The examples above provide a taste of how regular expressions are constructed and read.
Regular Expressions and Combining Characters | |
---|---|
Regular expressions come in many different flavors, and each one deals with
some of the more complex issues in Unicode in their own manners. This ambiguity
will most keenly be felt in the use of combining characters. Suppose we have a
string of three characters, áb (i.e., an acute accent over the a,
Unicode has differentiated three levels of support for regular expressions (see official report). Only level one conformance in TAN is guaranteed. Combining characters fall in level two. In TAN, character counts depend exclusively upon base characters, not combining ones (see the section called “Combining characters”). |
TAN includes several functions that usefully extend XML regular expressions.
See tan:regex
, tan:matches
(), tan:replace
(), tan:tokenize
().
Further reading: