% language=uk
%
% copyright=pragma-ade readme=readme.pdf licence=cc-by-nc-sa

\environment mcommon

\setupbackgrounds
  [page]
  [background={color,page},
   backgroundcolor=blue]

\definelayer
  [page]
  [width=\paperwidth,
   height=\paperheight]

\definecolor[yellow][green]

\starttext

\setuplayout[page]

\starttextmakeup

\setlayer
  [page]
  [offset=.5cm]
  {\scale
     [width=\dimexpr\paperwidth-3cm\relax,height=\dimexpr\paperheight-3cm\relax]
     {\vbox \bgroup
        \dostepwiserecurse{1}{26}{1}
          {\definedfont[Mono at 11pt]%
           \let\StepX\recurselevel
           \hbox \bgroup
             \dostepwiserecurse{1}{26}{1}
               {\let\StepY\recurselevel
                \doiflanguageelse{\character\StepX\character\StepY}
                  {\definecolor[supported:f][white]\definecolor[supported:b][red]}%
                  {\definecolor[supported:f][blue]\definecolor[supported:b][yellow]}%
                \framed
                   [frame=off,foregroundcolor=supported:f,
                    background=color,backgroundcolor=supported:b,backgroundoffset=-.33mm]
                   {\Character\StepX\Character\StepY}}%
           \egroup
           \endgraf
           \nointerlineskip}
      \egroup}}

\setlayerframed
  [page]
  [preset=rightbottom,x=2.5cm,y=.5cm]
  [foregroundcolor=white,frame=off,offset=overlay]
  {\tt\scale[height=1.5cm]{HYPHENATION}}

\setlayerframed
  [page]
  [preset=rightbottom,x=.5cm,y=2.5cm]
  [foregroundcolor=white,frame=off,offset=overlay]
  {\tt\rotate{\scale[height=1.5cm]{PATTERNS}}}

\setlayerframed
  [page]
  [preset=righttop,x=.5cm,y=.5cm]
  [foregroundcolor=white,frame=off,offset=overlay]
  {\tt\rotate{\scale[height=.75cm]{\undepthed{Hans Hagen}}}}

\stoptextmakeup

\setupbackgrounds
  [page]
  [background=]

\setuplayout
  [reset]

\section {Pattern files}

\TEX\ has two mysterious commands that the average user will never
or seldom meet:

\starttyping
\hyphenation{as-so-ciates}
\patterns   {.ach4}
\stoptyping

Both commands can take multiple strings, so in fact both commands
should be plural. The first command can be given any time and can be
used to tell \TEX\ that a word should be hyphenated in a certain
way. The second command can only be issued when \TEX\ is in virgin
mode, i.e.\ starting with a clean slate. Normally this only happens
when a format is generated.

The second command is more mysterious than the first one and its
entries are a compact way to tell \TEX\ between what character
sequences it may hyphenate words. The numbers represent weights and
the (often long) lists of such entries are generated with a special
program called \type {patgen}. Since making patterns is work for
specialists, we will not go into the nasty details here.

In the early stage of \CONTEXT\ development it came with its own
pattern files. Their names started with \type{lang-} and their
suffixes were \type {pat} and \type {hyp}.

However, when \CONTEXT\ went public, I was convinced to drop those
files and use the files already available in distributions. This was
achieved by using the \CONTEXT\ filename remapping mechanism. Although
those files are supposed to be generic, this is not always the case,
and it remains a gamble if they work with \CONTEXT. Even worse,
their names are not consistent and the names of some files as well
as locations in the tree keep changing. The price \CONTEXT\ users
pay for this is lack of hyphenation until such changes are noticed
and taken care of. Because constructing the files is an uncoordinated
effort, all pattern files have their own characteristics, most
noticably their encoding.

After the need to adapt the name mapping once again, I decided to
get back to providing \CONTEXT\ specific pattern files. Pattern
cooking is a special craft and \TEX\ users may call themselves
lucky that it's taken care of. So, let's start with thanking all
those \TEX\ experts who dedicate their time and effort to get
their language hyphenated. It's their work we will build (and keep
building) upon.

In the process of specific \CONTEXT\ support, we will take care of:

\startitemize
\item consistent naming, i.e.\ using language codes when possible as
      a prelude to a more sophisticated naming scheme, taking versions
      into account
\item consistent splitting of patterns and hyphenation exceptions in
      files that can be recognized by their suffix
\item making the files encoding independent using named glyphs
\item providing a way to use those patterns in plain \TEX\ as well
\stopitemize

Instead of using a control sequence for the named glyphs, we use a
different notation:

\starttyping
[ssharp] [zcaron] [idiaeresis]
\stoptyping

The advantage of this notation is that we don't have to mess with
spacing so that parsing and cleanup with scripts becomes more robust. The
names conform to the \CONTEXT\ way of naming glyphs and the names
and reverse mappings are taken from the encoding files in the
\CONTEXT\ distribution, so you need to have \CONTEXT\ installed.

The \CONTEXT\ pattern files are generated by a \RUBY\ script. Although
the converting is rather straightforward, some languages need special
treatment, but a script is easily adapted. If you want a whole bunch of
pattern files, just say:

\starttyping
ctxtools --patterns all
\stoptyping

or, if you want one language:

\starttyping
ctxtools --patterns nl
\stoptyping

If for some reason this program does not start, try:

\starttyping
texmfstart ctxtools --patterns nl
\stoptyping

When things run well, this will give you four files:

\starttabulate[|l|p|]
\NC \type{lang-nl.pat} \NC the patterns in an encoding indepent format \NC \NR
\NC \type{lang-nl.hyp} \NC the hyphenation exceptions \NC \NR
\NC \type{lang-nl.log} \NC the conversion log (can be deleted afterwards) \NC \NR
\NC \type{lang-nl.rme} \NC the preambles of the files used (copyright notices and such) \NC \NR
\stoptabulate

If you redistribute the files, it makes sense to bundle the \type
{rme} files as well, unless the originals are already in the
distribution. It makes no sense to keep the log files on your
system. When the file \type {lang-all.xml} is present, the info
from that file will be used and added to the pattern and
hyphenation files. In that case no \type {rme} and \type {log}
file will be generated, unless \type {--log} is provided.

In the Dutch pattern file you will notice entries like the
following:

\starttyping
e[ediaeresis]n3
\stoptyping

So, instead of those funny (encoding specific) \type {^^fc} or
(format specific) \type {\"e} we use names. Although this looks
\CONTEXT\ dependent it is rather easy to map those names back to
characters, especially when one takes into account that most
languages only have a few of those special characters and we only
have to deal with lower case instances.

The \CONTEXT\ support module \type {supp-pat.tex} is quite generic and
contains only a few lines of code. Actually, most of the code is
dedicated to the simple \XML\ handler. Loading a pattern meant
for EC encoded fonts in another system than \CONTEXT\ is done as
follows:

\starttyping
\bgroup

  \input supp-pat

  \lccode"E4="E4  \definepatterntoken adiaeresis ^^e4
  \lccode"F6="F6  \definepatterntoken odiaeresis ^^f6
  \lccode"FC="FC  \definepatterntoken ediaeresis ^^fc
  \lccode"FF="FF  \definepatterntoken ssharp     ^^ff

  \enablepatterntokens
  \enablepatternxml

  \input lang-de.pat
  \input lang-de.hyp

\egroup
\stoptyping

In addition to this one may want to set additional lower and
uppercase codes. In \ETEX\ these are stored with the language.

Just for completeness we provide the magic command to generate
the \XML\ variants:

\starttyping
ctxtools --patterns --xml all
\stoptyping

This will give you files like:

\starttyping
<?xml version='1.0' standalone='yes'?>

<!-- some comment -->

<patterns>
... e&ediaeresis;n3 ...
</patterns>
\stoptyping

This is also accepted as input but for our purpose it's probably
best to stick to the normal method. The pattern language is a \TEX\
specific one anyway.

\section{Installing languages}

Installing a language in \CONTEXT\ should not take too much effort
assuming the language is supported. Language specific labels are
grouped in \type {lang-*} files, like \type {lang-ger.tex} for the
germanic languages.

Patterns will be loaded from the files in the general \TEX\
distribution unless \type {lang-nl.pat} is found, in which case
\CONTEXT\ assumes that you prefer the \CONTEXT\ patterns. In that
case, run

\starttyping
ctxtools --patterns all
\stoptyping

You need to move the files to the \CONTEXT\ base path that you can
locate with:

\starttyping
textools --find context.tex
\stoptyping

You can also use \type {kpsewhich}, but the above method does an
extensive search. Of course you can also generate the files on a
temporary location. Now it's time to generate the formats:

\starttyping
texexec --make --all
\stoptyping

Since \XETEX\ needs patterns in \UTF-8 encoding, we provide a switch
for achieving that:

\starttyping
texexec --make --all --utf8
\stoptyping

Beware: you need to load patterns for each language and encoding
combination you are going to use. You can configure your local
\type {cont-usr} file to take care of this. When an encoding does
not have the characters that are needed, you will get an error.
When using the non \CONTEXT\ versions of teh patterns this may go
unnoticed because the encoding is hard coded in the file. Of
course it will eventually get noticed when the hyphenations come
out wrong.

The \CONTEXT\ distribution has a file lang-all.xml that holds the
copyright and other notes of the patterns. A discription looks like:

\starttyping
<description language='nl'>
  <sourcefile>nehyph96.tex</sourcefile>
  <title>TeX hyphenation patterns for the Dutch language</title>
  <copyright>
    <year>1996</year>
    <owner> Piet Tutelaers (P.T.H.Tutelaers@tue.nl)</owner>
    <comment>8-bit hyphenation patterns for TeX based upon the new
      Dutch spelling, officially since 1 August 1996. These
      patterns follow the new hyphenation rules in the
      `Woordenlijst Nederlandse Taal, SDU Uitgevers, Den Haag
      1995' (the so called `Groene Boekje') described in
      section 5.2 (Het afbreekteken)</comment>
  </copyright>
</description>
\stoptyping

{\em This file is \quote {work in process}: more details will be added
and comments will be enriched.}

\section {Commands}

You can at any moment add additional hyphenation exceptions to the
language specific dictionaries. For instance:

\starttyping
\language[nl] \hyphenation{pa-tiën-ten}
\stoptyping

Switching to another language is done with the \type {\language}
command. The document language is set with \type {\mainlanguage}.

If you want to let \TEX\ know that a word should be hyphenated in a
special way, you use the \type {\-} command, for instance:

\starttyping
Con\-TeXt
\stoptyping

Compound words are not recognized by the hyphenation engine, so there
you need to add directives, like:

\starttyping
the ConTeXt|-|system
\stoptyping

If you are using \XML\ as input format, you need to load the
hyphenation filter module. Here we assume that \UTF\ encoding
is used:

\starttyping
\useXMLfilter[utf,hyp]
\stoptyping

In your \XML\ file you can now add:

\starttyping
<hyphenations language='nl' regime='utf'>
  <hyphenation>pa-tiën-ten</hyphenation>
  <hyphenation>pa-tiën-ten-or-ga-ni-sa-tie</hyphenation>
  <hyphenation>pa-tiën-ten-plat-form</hyphenation>
</hyphenations>
\stoptyping

This filter also defines some auxiliary elements. Explicit
hyphenation points can be inserted as follows:

\starttyping
Zullen we hier af<hyphenate/>bre<hyphenate/>ken of niet?
\stoptyping

The compound token can be anything, but keep in mind that some
tokens are treated special (see other manuals).

\starttyping
Wat is eigenlijk een patiënten<compound token="-"/>platform?
\stoptyping

A language is set with:

\starttyping
nederlands <language code="en">english</language> nederlands
\stoptyping

If you set attribute \type {scope} to \type {global}, labels (as
used for figure captions and such) adapt to the language switch.
This option actually invokes \type {\mainlanguage}.

\section {Languages}

When users in a specific language area use more than one font
encoding, patterns need to be loaded multiple times. In theory
this means that one can end up with more instances than \TEX\ can
host. However, the number of sensible font encodings is limited as
is the number of languages that need hyphenation. Now that memory
is cheap and machines are fast, preloading a lot of pattern files
is no problem. The following table shows the patterns that are
preloaded in the version of \CONTEXT\ that is used to process this
file.

\showpatterns

{\em In the (near) future the somewhat arcane \type {pl0} and \type
{il2} encodings will go away since they are only used for Polish
and Czech/Slovak computer modern fonts, which can be replaced by
Latin Modern alternatives. Also, a new dense encoding may find its
way into this list.}

\section{Hyphenation}

While hypenating, \TEX\ has to deal with ligatures as well. While
Thomas, Taco and I were discussing the best ways to neutralize the
ancient greek patterns, Taco Hoekwater came up with the following
explanation.\footnote {Thomas Schmitz is responsible
for the associated third party module.}

\placefigure
  [nonumber]
  {The most common ligatures.}
  {\startcombination[4*1]
     {\scale[height=3\lineheight]{\color[AltColor]{fi}}}  {}
     {\scale[height=3\lineheight]{\color[AltColor]{fl}}}  {}
     {\scale[height=3\lineheight]{\color[AltColor]{ffi}}} {}
     {\scale[height=3\lineheight]{\color[AltColor]{ffl}}} {}
   \stopcombination}

Any direct use of a ligature (as accessed by \type {\char} or
through active characters) is wrong and will create faulty
hypenation. Normally, when TeX sees \quote {office}, it has the
six tokens \type {office} and it knows from the patterns that it
can hyphenate  between the \type {ff}. It will build an
internal list of four nodes, like this:

\starttyping
[char, o  , ffi ]
[lig , ffi, c   ,[f,f,i]]
[char, c  , e   ]
[char, e  , NULL]
\stoptyping

As you can see from the \type {ffi} line, it has remembered the
original characters. While hyphenating, it temporarily changes
back to that, then re-instates the ligature afterwards.

If you feed it the ligature directly, like so:

\starttyping
[char, o   , ffi ]
[char, ffi , c   ]
[char, c   , e   ]
[char, e   , NULL]
\stoptyping

it cannot do that. It tries to hyphenate as if the \type{ffi}
was a character, and the result is wrong hyphenation.

\stoptext
