% content=tex
%
% copyright=pragma-ade readme=readme.pdf licence=cc-by-nc-sa

\usemodule[mag-01,abr-02]

\setvariables
  [magazine]
  [title={Dealing with XML},
   author=Hans Hagen,
   affiliation=PRAGMA ADE,
   date=November 2004,
   number=8]

\dontcomplain % \showframe

\showXMLlin[title,subtitle,name,street,number,city]
\showXMLlin[fx:td,fx:th,fx:framed,fx:margintext]
\showXMLlin[xsl:value-of,xsl:with-param]

%showXMLtxt[xsl:template]

\definecolor [red]              [darkred]
\definecolor [green]            [darkgreen]
\definecolor [blue]             [darkblue]
\definecolor [yellow]           [darkyellow]

\definecolor [colorprettyone]   [darkred]
\definecolor [colorprettytwo]   [darkgreen]
\definecolor [colorprettythree] [darkblue]
\definecolor [colorprettyfour]  [darkyellow]

\setuptyping[lines=no]

\defineexternalfigure[sample][width=.75\textwidth,frame=on] % A5

\startbuffer[abstract]
In this issue I will drop an idea that has been buzzing in my
mind for a long time: how to comfortably deal with typesetting
documents coded in \XML, while using the manipulative power of
\XSLT\ on the one hand and the typesetting capabilities of
\CONTEXT\ at the other hand, without seeing \TEX\ code.
\stopbuffer

\starttext \setups [titlepage] \setups [title]

\subject{Introduction}

When implementing an \XSLFO\ engine, I found out that it's not
that trivial to implement robust mechanisms, i.e.\ to guarantee
output that conforms to what we normally want to achieve with
\CONTEXT.

So far I encountered roughly two applications where \XSLFO\ are
used:

\startitemize[packed]
\item short (few page) documents, whose appearance resembles
      forms
\item large documents with a relatively simple layout
\stopitemize

The first category demands a simple page builder as well as means
to position data onto the page|(|body). The data is rather static.
The second category is more demanding in terms of typesetting, and
needs ways to deal with mixed page buildup.

The \XSLFO\ model is not that far from cascading stylesheets, the
way to deal with styling text on the web. As with much in the
\XML\ world, it has the promise of being the ultimate solution to
all problems. But getting something on paper is not the same as
getting it on screen, if only because (most noticably in educational
documents) complex page layouts may be used when typesetting books.
A realistic view on its \XSLFO\ applicability depends on the place
someone has in the workflow from \XML\ to (for instance) \PDF. In
this context I often remark that \quotation {The problem hasn't
changed.}. What I mean with this is that, at the level where the
final mapping from one onto the other has to be done, it's the
(quality of) the design that determines the tools to be used. One
time a solution using \XSLFO\ is nice, clean and robust, another
time, mapping onto a full featured typesetting engine like \CONTEXT\
makes more sense. And, when in the process the designs evolve
(often becoming less consistent over time) both solution may end up
in messy source code, and all claims for clean, portable solutions
become invalid. But, given that most documents |<|at the moment
using \XML\ based solutions make most sense when applied to series
of documents|>| are one||time shots, this is not that big an issue.

To come back to the two applications mentioned, it is interesting
to notice that when \XML\ is used in large scale applications where
many pages are to be typeset that all look more or less the same,
direct mappings onto a typesetting language like \TEX\ is still way
more efficient that using all kind of transformations. However, who
wants to learn \TEX, with its not always that intuitive interface
(a mixture between typesetting and programming features)?

The other application I mentioned, simple documents (that can have
hundreds of pages), are often so simple to typeset, that using \TEX\
sounds like overkill. This may be one of the reasons why \TEX\ has
always gained more interest from its original audience
(mathematicians) than from those who need to typeset for instance
novels. Anyhow, setting up a style for a document with just text,
some sectioning, a few tables and some itemized lists, is not that
complex.

So, what then are the problems with using \XSLFO\ for these
documents? It is no secret that an author can mess up a \TEX\
document by using font switches and spacing at will. Scientific
publishers spend much time cleaning up source code, i.e.\ removing
such fine tuning. The same is true for a \TEX\ document style: it's
no problem to make it look like a real mess, and it's often hard to
decide if it is indeed a mess or just a complex solution for a
complex problem. Sometimes users copy code from existing styles,
and as a result, much of the code in styles is not doing anything or
all, or worse, interfering in unexpected ways.

Although an \XSLFO\ source is not supposed to be read by humans,
some of the test files that I used as testbed gave me a kind of
familiar feeling of such a mess. The positive side is that one can
clean up the mess by redesigning the \XSLT\ conversion script. But
even then, in the end there is this huge file with lots of font
switches and spacing directives at each level. When done right, this
is no problem (apart from being time consuming for a formatter).
However, in order to let a typesetting engine do a proper job, it
needs to know what it's dealing with, and since this information is
not present in the file anymore, it ends up gambling. It goes beyond
this MyWay to discuss this in depth, but it's this observation that
brings me to a slightly different approach. As an implementer of a
rather large typesetting system, it gives an uncomfortable feeling
to cook up imperfect solutions, knowing that a little bit more
information already would give better results. Typesetting on grids
is for instance a rather special business, as is typesetting in
columns and tables. There is not one solution. A nice example of a
comparable situation is in \MATHML: in many cases content based
markup gives better results than presentational markup. Alas coding
in structure happens less than coding by pasting together pieces
like in presentational \MATHML.

So, here we are. We have \XML\ document sources. We have a powerfull
language for manipulating such sources called \XSLT. Sometimes the
quality of the \XML\ source may demand some cleanup beforehand,
using for instance the text processing capabilities of modern
scripting languages. The question we need to answer is: will we use
\XSLFO, and one of the available processors, or will we use a
specialized formatting engine, or will we use a combination of both?
(Imagine for instance an \XSLFO\ document instance being used as
placed \XML.)

Here I will discuss a mixture between structured and generated style
markup. When users like it, this may become a thread in the
\CONTEXT\ story. Also, it may provide those who now use \XSLT\ to
map onto \TEX\ code to map onto \XML\ instead. In the end we may end
up with style libraries coded in \XML\ instead of \TEX. But above
all, we can get high quality output, using complex page models and
using features provided by \CONTEXT\ in a different way.

\subject{The process}

{\em Remark: for simplicity we use structural components that are
familiar to \CONTEXT\ users, but in the end we may end up with more
generic constructs. There are many ways leading to Rome and it would
be stupid to think that we can use the same vehicle on each trip.
Consider it an experiment.}

Our first document is a real simple one.

\showXMLfile{dem-0008-1.xml}

This document can be converted into an intermediate format, where
structure is replaced by more rigourous structural components.
However, a paragraph of text remains what it is. Watch the
definition of the layout. It looks familiar for \CONTEXT\ users.

\showXMLfile{dem-0008-1.xsl}

We can feed our document to \XSLTPROC, using this script:

\starttyping
xsltproc --output dem-0008-1.fx dem-0008-1.xsl dem-0008-1.xsl
\stoptyping

This gives:

\showXMLfile{dem-0008-1.fx}

We process this with:

\starttyping
texexec --pdf --use=fx --xml dem-0008-1.fx
\stoptyping

The resulting file is shown in \in {figure} [fig:dem-0008-1]: indeed
just a line of text.

\placefigure
  [here] [fig:dem-0008-1]
  {\type{dem-0008-1.fx}}
  {\externalfigure[dem-0008-1.pdf][sample]}

We will enhance our document with a title. This means that we need
to deal with a typographical construct that  needs special treatment
when encountered in sequence and in relation to the following text.

\showXMLfile{dem-0008-2.xml}

Those who have used \CONTEXT\ for typesetting \XML\ directly, may
think of:

\startTEX
\defineXMLenvironment[document] {\starttext} {\stoptext}
\defineXMLargument   [title]    {\chapter}
\stopTEX

and then use the normal definition and setup commands to tune the
look and feel. This could be a problem for those who don't like
backslashes, so again we will kick in \XSLT.

\showXMLfile{dem-0008-2.xsl}

Now watch this. We don't use the \CONTEXT\ numbering mechanism, but
directly generate the numbers with \XSLT ! \in {Figure}
[fig:dem-0008-2] shows how this looks after processing. Since we're
just demonstrating a way of processing \XML, we don't tune spacing
now.

\showXMLfile{dem-0008-2.fx}

\placefigure
  [here] [fig:dem-0008-2]
  {\type{dem-0008-2.fx}}
  {\externalfigure[dem-0008-2.pdf][sample]}

In \XSLFO\ we've seen such a title being generated using a item
construct, a table, or just a combination of blocks. There one
uses (combinations of) attributes to communicate if we're dealing
with something that flows (a paragraph) or should be considered
kind of fixed.

Here we use the dedicated head construct in combination with
the rather fixed layers and framed constructs instead. It is no
problem to isolate code and reuse it in more than one situation.

\showXMLfile{dem-0008-0.xsl}

The translation now looks like:

\showXMLfile{dem-0008-3.xsl}

This time we left out the colors (\in {figure} [fig:dem-0008-3]).

\placefigure
  [here] [fig:dem-0008-3]
  {\type{dem-0008-3.fx}}
  {\externalfigure[dem-0008-3.pdf][sample]}

We further enhance our document, this time with some tabular
information:

\showXMLfile{dem-0008-4.xml}

In \CONTEXT\ we have several mechanisms for dealing with tables, and
one of them is well suited for tabular information that ends up in
the text flow. Althoug it is technically possible to combine all
specific table features in one mechanism, it is pretty hard to let
it give the desired output without the user providing clues.
Therefore it's more convenient to use dedicated mechanisms. (Also,
using dedicated mechanisms is less error prone due to changes and
updates in the table handler. The way web browsers deal with tables
over time demonstrates that it's far from trivial to suit all
needs.)

\showXMLfile{dem-0008-4.xsl}

The result is shown below and in \in {Figure} [fig:dem-0008-4].

\showXMLfile{dem-0008-4.fx}

\placefigure
  [here] [fig:dem-0008-4]
  {\type{dem-0008-4.fx}}
  {\externalfigure[dem-0008-4.pdf][sample]}

An alternative mapping can be:

\showXMLfile{dem-0008-5.xsl}

This becomes:

\showXMLfile{dem-0008-5.fx}

And shows up as in \in {Figure} [fig:dem-0008-5].

\placefigure
  [here] [fig:dem-0008-5]
  {\type{dem-0008-5.fx}}
  {\externalfigure[dem-0008-5.pdf][sample]}

As said, the tabulate mechanism is well suited for tabular
information that is part of the text flow. This is quite noticable
when one has to deal with typesetting on a grid.

Our last example concerns another sometimes tricky feature, marginal
notes.

\showXMLfile{dem-0008-6.xml}

The mapping is simple:

\showXMLfile{dem-0008-6.xsl}

And the result looks familiar as well:

\showXMLfile{dem-0008-6.fx}

And \in {figure} [fig:dem-0008-6] shows that the text shows up in the
margin indeed.

\placefigure
  [here] [fig:dem-0008-6]
  {\type{dem-0008-6.fx}}
  {\externalfigure[dem-0008-6.pdf][sample]}

If you know a bit of \CONTEXT, you can imagine that processing such
files is rather trivial. Even more interesting is that one has the
full machinery available as well. However, the idea is that we limit
ourselves to only part of \CONTEXT's functionality and let \XSLT\ do
the dirty work. The code shown here is just the result of an simple
experiment and should be considered as such. However, depending on
user demand and input, we may end up with a rather full blown
implementation. Just to give you an idea of how this is done:

\startTEX
\defineXMLenvironment
  [fx:framed] [fox] []
  {\initializefox
   \expanded{\framed[\foxarguments]}\bgroup\ignorespaces}
  {\removeunwantedspaces\egroup}
\stopTEX

with a few definition common to such mappings:

\startTEX
\def\foxarguments
  {\theXMLarguments{fox}}
\def\initializefox
  {\expanded{\getrawparameters[fox][\theXMLarguments{fox}]}}
\stopTEX

We realized that some constructs may demand extensions to \CONTEXT,
for instance, it makes no sense to use the current head mechanisms
for some related \XML\ variant. On the other hand, layers and such
can be used directly, and some |<|yet undocumented but rather
interesting|>| other mechanisms as well. Another topic for discussion
is how to deal with defaults, for instance those used in the standard
layout definition. We may as well start with no defaults at all. The
same is true for handling page numbers, headers and footers. In the
end it may even result in a lightweight version of \CONTEXT. Fonts
are yet another matter. Several methods need to be supported there,
if only because we want the goodies that \PDFTEX\ and \XETEX\ provide.

How well this can be used in combination with \XSLFO\ is to be seen.
In principle combining the technologies is possible, but it may be
needed to disable some of the \XSLFO\ features that spoil the game.

This means that currently we have the following methods (either or not
used in combination) to go from \XML\ to a typeset document:

\startitemize[packed]
\item using the builtin \XML\ parser to map onto \CONTEXT\ code
\item using \XSLT\ to map onto \CONTEXT\ \TEX\ code
\item using \XSLT\ to map onto \CONTEXT\ \XML\ code
\item using \XSLT\ to produce the \XSLFO\ as accepted by \CONTEXT\
\stopitemize

In the third case we do a rather direct mapping, and so we benefit
from the interface documentation already available. If this method,
of which we demonstrated an example here, makes sense to users, we
may even end up with a large repository of transformation scripts
for common \DTD's and interesting layouts. The challenge is to
determine what functionality is needed, so \unknown\ use the
\CONTEXT\ mailing list to let your voice be heard.

% \setups [listing]

\setups [lastpage] \stoptext
