% engine=luatex language=uk

% \ctxlua{fonts.otf.trace_features = true}
% \ctxlua{fonts.otf.trace_contexts = true}
% \ctxlua{fonts.otf.trace_ligatures = true}

% \ctxlua{dofile("test.lua")}

%%%% adam: I've tested the code, and with "clig" enabled, when I type in "the synthesis", only the free-standing "the" gets the t_h_e ligature. With "dlig" enabled, both get transformed, and this is by design. The "dlig" feature is supposed to by use only upon user's discretion, usually on single runs, words or even pairs. It makes little sense to enable "dlig" for an entire sentence or paragraph. That's how the OT spec envisions it.

\setupwhitespace[big]
\usemodule[abr-01]

\title{Zapfing fonts}

\subject {features}

In previous chapters we've see support for \OPENTYPE\ features creep into \LUATEX\ and
\CONTEXT\ \MKIV. However, it may not have been clear that so far we were just feeding
the traditional \TEX\ machinery with the right data: ligatures and kerns.

In order to support features in their full glory, more is needed: we need to
manipulate the node list. As a result, we have now a second mechanism built into
\MKIV\ and users can choose what method they like most. Keep in mind that the
first method (called \quote {base}) is less powerfull and less complete than
the one called \quote {node}. Eventually \CONTEXT\ will use the node method by
default.

There are two variants of features: substitutions and positioning. Here we
concentrate on substitutions of which there are several. Positioning is for instance
used for kerning of specialized tasks like typesetting Arab.

One character can be replaced by one or more fixed alternatives or alternatives
chosen from a list of alternatives (substitutions or alternates). Multiple characters
can be replaces by one character (substitutions, alternates or a ligature). The
replacements can depend on preceding and/or following glyphs in which case we say that
the replacement is driven by rules. Rules can deal with single glyphs, combinations of
glyphs, classes (defined in the font) of glyphs and|/|or ranges of  glyphs.

Because the available documentation of \OPENTYPE\ is rather minimalistic and because
most fonts are relatively simple, you can imagine that figuring out how to
implement support for fonts with advanced features (like Zapfino) is not entirely trivial
and involves some trial and error. What also complicate things is that features can
interfere. Yet another complicating factor is that in the order of applying a rule may
obscure a later rule. Such fonts don't ship with manuals and examples of correct output
are not part of the buy.

I like testing \LUATEX's open type support with Palatino Regular and Palatino Sans and
good old \TYPEONE\ support with Optima Nova. So it made sense to test advanced
features with Zapfino Pro. This font has many features, which happen to be
implemented by Adam Twardoch, a well known font expert and familiar with the \TEX\
community. I have the feeling that when \LUATEX\ can support Zapfino Pro, designed by
Hermann Zapf and enhanced by Adam, we have reached a crucial point in the development.

The first thing that one will observe when using this font is that the files are larger
than normal, especially the cached versions in \MKIV. This made me extend some of the
serialization code so that it could handle huge tables better but at the cost of some
speed. Once we could handle the data conveniently and look into the font, it became
clear that implementing for the \quote {calt} and \quote {clig} features would take a
bit of coding.

\subject{rules}

\definefontfeature
   [zapfino]
   [language=nld,script=latn,mode=node,%
    calt=yes,clig=yes,liga=yes,rlig=yes]

\start \font\t=ZapfinoExtraLTPro*zapfino at 24pt \t \input zapf  \par \stop

\start \font\t=ZapfinoExtraLTPro*zapfino at 24pt \t \input tufte \par \stop

\subject{error recovery}

When I was testing node based feature support, the only way to check this was to
selectively support rules. For instance, when a sequence has to be replaced by a
ligature, replacing boils down to:

\startitemize[packed]
\item locate the glyph sequence \type{c/o}
\item find out what the replacement is
\item if it's a ligature, then replace the sequence by this one glyph
\stopitemize

As said, in this case, the replacement glyph is supposed to be a ligature and indeed
there is such a ligature: \type {c_slash_o}. However, when testing this, I was not
looking at this rule but at the (randomly chose) rule that was meant to
intercept the alternative \type {h.2} followed by \type {z.4}. Interesting was that
this resolved to a ligature indeed, but the shape associated with this ligature
was an~\type {h}, which is not right. Actually, a few more of such rules turned
out to be wrong. It took a bit of an effort to reach this conclusion because of the
mentioned interferences of features and rules. At that time, the rule entry (in raw
\LUATEX\ table format) looks as follows:

\starttyping
[44] = {
    ["format"] = "coverage",
    ["rules"] = {
        [1] = {
            ["coverage"] = {
                ["ncovers"] = {
                    [1] = "h.2",
                    [2] = "z.4",
                }
            },
            ["lookups"] = {
                [1] = {
                    ["lookup_tag"] = "L084",
                    ["seq"] = 0,
                }
            }
        }
    }
    ["script_lang_index"] = 1,
    ["tag"] = "calt",
    ["type"] = "chainsub"
}
\stoptyping

Instead of reinventing the wheel, Taco used the \FONTFORGE\ libraries for reading the
\OPENTYPE\ font files. Therefore the \LUATEX\ table is resembling the internal \FONTFORGE\
data structures.

Here \type {ncovers} means that wen the current character has shape \type {h.2} (!!!!)
and the next one is \type {z.4} (a sequence) then we need to apply the lookup
internally tagges \type {L084}. Such a rule can be more extensive, for instance
instead of \type {h.2} one can have a list of characters, and there can be \type
{bcovers} and \type {fcovers} as well, which means that preceding or following character
need to be taken into account as well.

When this rule matches, it resolves to a specification like:

\starttyping
[6] = {
    ["flags"] = 0,
    ["lig"] = {
        ["char"] = "h",
        ["components"] = "h.2 z.4",
    },
    ["script_lang_index"] = 65535,
    ["tag"] = "L084",
    ["type"] = "ligature",
}
\stoptyping

Here \type {tag} and \type {script_lang_index} are kind of special and
are part of an private feature system, i.e.\ they make up the cross reference
between rules and glyphs.

It took a couple of Skype sessions and mails before Taco and I came to the
conclusion that this was probably a glitch in the font. So, what to do when a
font has bugs like this? Should one disable the feature? That would be a pitty
because a font like Zapfino depends on it.

We should keep in mind that Zapfino is an extraordinary case, because it used
the \OPENTYPE\ features extensively. We can also be sure that the problems will
be fixed once they are known, if only because Adam Twardoch (who did the job)
has exceptionaly high standards butit may take a while before the fix reached
the user (who then has to update his or her font). As said, it also takes some
effort to run into the situation described here so the likelyhood of running
into this rule is small.

Can we add some additional checks for problems like this? For a while I
thought that it was possible by assuming that ligatures have names like
\type {h.2_z.4} but alas. Sequences of glyphs are mapped onto ligatures
using mappings like the following:

\starttyping
\starttabulate
\NC three fraction four.2 \NC threequarters \NC \NR
\NC three fraction four   \NC threequarters \NC \NR
\NC d r                   \NC d_r           \NC \NR
\NC e period              \NC e_period      \NC \NR
\NC f i                   \NC fi            \NC \NR
\NC f l                   \NC fl            \NC \NR
\NC f f i                 \NC f_f_i         \NC \NR
\NC f t                   \NC f_t           \NC \NR
\stoptabulate
\stoptyping

Some ligature have no \type {_} in their names and there are also some
inconsistencies, compare the \type {fl} and \type {f_f_i}. Here font
history is painfully reflected in inconsistency and no solution can be
found here.

So, in order to get rid of this problem, I implemented a method to ignore
certain rules but then, this only makes sense if one knows how the rules
are tagged internally. So, in practice this is no solution. However, I can
imagine that at some point \CONTEXT\ ships with a database of fixes that
are applied to known fonts with certain version numbers.

We also found out that the font table that we used was not good enough for our
purpose because the exact order in what rules have to be applies was not
available. Then Taco noticed that in the meantime \FONTFORGE\ had moved on
to version~2 and after consulting the author we quickly came to the conclusion
that it made sense to use the updated representation.

Now the snippet with the previously mentioned rule looks as follows:

\starttyping
["ks_latn_l_66_c_19"]={
 ["format"]="coverage",
 ["rules"]={
  [1]={
   ["coverage"]={
    ["ncovers"]={
     [1]="h.2",
     [2]="z.4",
    }
   },
   ["lookups"]={
    [1]={
     ["lookup"]="ls_l_84",
     ["seq"]=0,
    }
   }
  }
 },
 ["type"]="chainsub",
},
\stoptyping

This rule is mentioned in a feature specification table. Here specific features are
associated with languages and scripts. This is just one of the entries concerning
\type {calt}. You can imagine that it took a while to figure out hwo best to
deal with this, but eventually the \MKIV\ code could do the trick.

\starttyping
 ["gsub"]={
  ...
  [67]={
   ["features"]={
    [1]={
     ["scripts"]={
      [1]={
       ["langs"]={
        [1]="AFK ",
        [2]="DEU ",
        [3]="NLD ",
        [4]="ROM ",
        [5]="TRK ",
        [6]="dflt",
       },
       ["script"]="latn",
      }
     },
     ["tag"]="calt",
    }
   },
   ["name"]="ks_latn_l_66",
   ["subtables"]={
    [1]={
     ["name"]="ks_latn_l_66_c_0",
    },
    ...
    [20]={
     ["name"]="ks_latn_l_66_c_19",
    },
    ...
   },
   ["type"]="gsub_context_chain",
  },
\stoptyping


\startbuffer[a]
\definefontfeature
   [calt]
   [language=nld,script=latn,mode=node,
    calt=yes,clig=yes,liga=yes,rlig=yes]
\definefontfeature
   [none]
   [language=nld,script=latn,mode=node,
    calt=no, clig=yes,liga=yes,rlig=yes]
\definefontfeature
   [fixd]
   [language=nld,script=latn,mode=node,
    calt=yes,clig=yes,liga=yes,rlig=yes]
\stopbuffer

When testing the implementation of features one need to be aware of the fact that
some appearance are also implemented using the regular ligature mechanisms.

\startbuffer[b]
\type{calt:}\font\t=ZapfinoExtraLTPro*calt at 24pt\t\char57780\char57851\quad
\type{none:}\font\t=ZapfinoExtraLTPro*none at 24pt\t\char57780\char57851\quad
\type{fixd:}\font\t=ZapfinoExtraLTPro*fixd at 24pt\t\char57780\char57851\quad
\stopbuffer

\typebuffer[a] \typebuffer[b]

\start \getbuffer[a] \getbuffer[b] \stop

When testing features it helps to use words that look similar. Here are some of
the words that we used.

\definefontfeature
   [zapfing]
   [language=dflt,
    script=latn,
    calt=yes,
    clig=yes,
    rlig=yes,
    mode=node]

\font\Zapfing=ZapfinoExtraLTPro*zapfing at 24pt

% \language0 \noligs=1 \nokerns=1

\startbuffer
troef troef troef troeftroef troef  \par
\ruledhbox{troef troef troef troeftroef troef} \par
\ruledhbox{troef 123} \par
\ruledhbox{troef} \ruledhbox{troef } \ruledhbox{ troef} \ruledhbox { troef } \par
\stopbuffer

\typebuffer \start \Zapfing \getbuffer \stop

\startbuffer
review review review, review \par
itemize, review \par
itemize, review, \par
\stopbuffer

\typebuffer \start \Zapfing \getbuffer \stop

\startbuffer
Welcome to Zapfino
\stopbuffer

\typebuffer \start \Zapfing \getbuffer \stop

\startbuffer
synopsize versus synthesize versus synthase versus sympathy verus synonym
\stopbuffer

\typebuffer \start \Zapfing \getbuffer \stop

\startbuffer
eggen groet ogen hagen \par
dieren druiven onder aard  donder modder \par
fiets effe flater triest troef \par
\stopbuffer

\typebuffer \start \Zapfing \getbuffer \stop

\startbuffer
Mr. Hermann Zapf
\stopbuffer

\typebuffer \start \Zapfing \getbuffer \stop

\stoptext
