\startcomponent hybrid-grouping

\startbuffer[MyAbstract]
\StartAbstract
    In this article I will discuss a few things that
    are hard to do in traditional \TEX, but reasonable
    well in \LUATEX.
\StopAbstract
\stopbuffer

\doifmodeelse {tugboat} {
    \usemodule[tug-01,abr-01]
    \setvariables
      [tugboat]
      [columns=yes]
    \setvariables
      [tugboat]
      [year=2010,
       volume=99,
       number=9,
       page=99]
    \setvariables
      [tugboat]
      [title=Grouping,
       subtitle=A few things you can do with LUATEX,
       keywords=,
       author=Hans Hagen,
       address=PRAGMA ADE\\Ridderstraat 27\\8061GH Hasselt NL,
       email=pragma@wxs.nl]
    %
    % we use a buffer as abstract themselves are buffers and
    % inside macros we loose line endings and such
    \getbuffer[MyAbstract]
    %
    \StartArticle
} {
    \environment hybrid-environment
    \startchapter[title={Grouping}]
}

\setupbars[rulethickness=.15] % nicer

\startsection [title={Variants}]

After using \TEX\ for a while you get accustomed to one of its
interesting concepts: grouping. Programming languages like
\PASCAL\ and \MODULA\ have keywords \type {begin} and \type {end}.
So, one can say:

\starttyping
if test then begin
    print_bold("test 1")
    print_bold("test 2")
end
\stoptyping

Other languages provide a syntax like:

\starttyping
if test {
    print_bold("test 1")
    print_bold("test 2")
}
\stoptyping

So, in those languages the \type {begin} and \type {end} and|/|or
the curly braces define a \quote {group} of statements. In \TEX\
on the other hand we have:

\starttyping
test \begingroup \bf test \endgroup test
\stoptyping

Here the second \type {test} comes out in a bold font and the
switch to bold (basically a different font is selected) is
reverted after the group is closed. So, in \TEX\ grouping deals
with scope and not with grouping things together.

In other languages it depends on the language of locally defined
variables are visible afterwards but in \TEX\ they're really local
unless a \type {\global} prefix (or one of the shortcuts) is used.

In languages like \LUA\ we have constructs like:

\starttyping
for i=1,100 do
    local j = i + 20
    ...
end
\stoptyping

Here \type {j} is visible after the loop ends unless prefixed by
\type {local}. Yet another example is \METAPOST:

\starttyping
begingroup ;
    save n ; numeric n ; n := 10 ;
    ...
endgroup ;
\stoptyping

Here all variables are global unless they are explicitly saved
inside a group. This makes perfect sense as the resulting graphic
also has a global (accumulated) property. In practice one
rarely needs grouping, contrary to \TEX\ where one really wants to
keep changes local, if only because document content is so
unpredictable that one never knows when some change in state
happens.

In principle it is possible to carry over information across a group
boundary. Consider this somewhat unrealistic example:

\starttyping
\begingroup
    \leftskip 10pt
    \begingroup
        ....
        \advance\leftskip 10pt
        ....
    \endgroup
\endgroup
\stoptyping

How do we carry the advanced leftskip over the group boundary
without using a global assignment which could have more drastic side
effects? Here is the trick:

\starttyping
\begingroup
    \leftskip 10pt
    \begingroup
        ....
        \advance\leftskip 10pt
        ....
        \expandafter
    \endgroup
    \expandafter \leftskip \the\leftskip
\endgroup
\stoptyping

This is typical the kind of code that gives new users the creeps
but normally they never have to do that kind of coding. Also, that
kind of tricks assumes that one knows how many groups are involved.


\stopsection

\startsection [title={Implication}]

What does this all have to do with \LUATEX\ and \MKIV ? The user
interface of \CONTEXT\ provide lots of commands like:

\starttyping
\setupthis[style=bold]
\setupthat[color=green]
\stoptyping

Most of them obey grouping. However, consider a situation where we
use \LUA\ code to deal with some aspect of typesetting, for
instance numbering lines or adding ornamental elements to the
text. In \CONTEXT\ we flag such actions with attributes and often
the real action takes place a bit later, for instance when a
paragraph or page becomes available.

A comparable pure \TEX\ example is the following:

\starttyping
{test test \bf test \leftskip10pt test}
\stoptyping

Here the switch to bold happens as expected but no leftskip of
10pt is applied. This is because the set value is already
forgotten when the paragraph is actually typeset. So in fact we'd
need:

\starttyping
{test test \bf test \leftskip10pt test \par}
\stoptyping

Now, say that we have:

\starttyping
{test test test \setupflag[option=1] \flagnexttext test}
\stoptyping

We flag some text (using an attribute) and expect it to get a
treatment where option~1 is used. However, the real action
might take place when \TEX\ deals with the paragraph or page
and by that time the specific option is already forgotten or
it might have gotten another value. So, the rather natural \TEX\
grouping does not work out that well in a hybrid situation.

As the user interface assumes a consistent behaviour we cannot
simply make these settings global even if this makes much sense in
practice. One solution is to carry the information with the
flagged text i.e.\ associate it somehow in the attribute's value.
Of course, as we never know in advance when this information is
used, this might result in quite some states being stored
persistently.

A side effect of this \quote {problem} is that new commands might
get suboptimal user interfaces (especially inheritance or cloning
of constructs) that are somewhat driven by these \quote
{limitations}. Of course we may wonder if the end user will notice
this.

To summarize this far, we have three sorts of grouping to deal
with:

\startitemize[item]

\startitem \TEX's normal grouping model limits its scope to the
local situation and normally has only direct and local
consequences. We cannot carry information over groups. \stopitem

\startitem Some of \TEX's properties are applied later, for
instance when a paragraph or page is typeset and in order to make
\quote {local} changes effective, the user needs to add explicit
paragraph ending commands (like \type {\par} or \type {\page}).
\stopitem

\startitem Features dealt with asynchronously by \LUA\ are at that
time unaware of grouping and variables set that were active at the
time the feature was triggered so there we need to make sure that
our settings travel with the feature. There is not much that a
user can do about it as this kind of management has to be done by
the feature itself. \stopitem

\stopitemize

\stopsection

It is the third case that we will give an example of in the next
section. We leave it up to the user if it gets noticed on the
user interface.

\startsection [title={An example}]

A group of commands that has been reimplemented using a hybrid
solution is underlining or more generic: bars. Just take a look at
the following examples and try to get an idea on how to deal with
grouping. Keep in mind that:

\startitemize[packed]
\startitem Colors are attributes and are resolved in the backend, so
way after the paragraph has been typeset. \stopitem
\startitem Overstrike is also handled by an attribute and gets applied
in the  backend as well, before colors are applied. \stopitem
\startitem Nested overstrikes might have different settings. \stopitem
\startitem An overstrike rule either inherits from the text or has
its own color setting. \stopitem
\stopitemize

First an example where we inherit color from the text:

\startbuffer
\definecolor[myblue][b=.75]
\definebar[myoverstrike][overstrike][color=]

Test \myoverstrike{%
    Test \myoverstrike{\myblue
        Test \myoverstrike{Test}
        Test}
    Test}
Test
\stopbuffer

\typebuffer \getbuffer

Because color is also implemented using attributes and processed
later on we can access that information when we deal with the bar.

The following example has its own color setting:

\startbuffer
\definecolor[myblue][b=.75]
\definecolor[myred] [r=.75]
\definebar[myoverstrike][overstrike][color=myred]

Test \myoverstrike{%
    Test \myoverstrike{\myblue
        Test \myoverstrike{Test}
        Test}
    Test}
Test
\stopbuffer

\typebuffer \getbuffer

See how we can color the levels differently:

\startbuffer
\definecolor[myblue] [b=.75]
\definecolor[myred]  [r=.75]
\definecolor[mygreen][g=.75]

\definebar[myoverstrike:1][overstrike][color=myblue]
\definebar[myoverstrike:2][overstrike][color=myred]
\definebar[myoverstrike:3][overstrike][color=mygreen]

Test \myoverstrike{%
    Test \myoverstrike{%
        Test \myoverstrike{Test}
        Test}
    Test}
Test
\stopbuffer

\typebuffer \getbuffer

Watch this:

\startbuffer
\definecolor[myblue] [b=.75]
\definecolor[myred]  [r=.75]
\definecolor[mygreen][g=.75]

\definebar[myoverstrike][overstrike][max=1,dy=0,offset=.5]
\definebar[myoverstrike:1][myoverstrike][color=myblue]
\definebar[myoverstrike:2][myoverstrike][color=myred]
\definebar[myoverstrike:3][myoverstrike][color=mygreen]

Test \myoverstrike{%
    Test \myoverstrike{%
        Test \myoverstrike{Test}
        Test}
    Test}
Test
\stopbuffer

\typebuffer \getbuffer

It this the perfect user interface? Probably not, but at least it
keeps the implementation quite simple.

The behaviour of the \MKIV\ implementation is roughly the same as
in \MKII, although now we specify the dimensions and placement in
terms of the ratio of the x-height of the current font.

\startbuffer
Test \overstrike{Test \overstrike{Test \overstrike{Test} Test} Test} Test \blank
Test \underbar  {Test \underbar  {Test \underbar  {Test} Test} Test} Test \blank
Test \overbar   {Test \overbar   {Test \overbar   {Test} Test} Test} Test \blank
Test \underbar  {Test \overbar   {Test \overstrike{Test} Test} Test} Test \blank
\stopbuffer

\typebuffer \getbuffer

As an extra this mechanism can also provide simple backgrounds.
The normal background mechanism uses \METAPOST\ and the
advantage is that we can use arbitrary shapes but it also carries
some limitations. When the development of \LUATEX\ is a bit
further along the road I will add the possibility to use
\METAPOST\ shapes in this mechanism.

Before we come to backgrounds, first take a look at these
examples:

\startbuffer
\startbar[underbar]  \input zapf \stopbar \blank
\startbar[underbars] \input zapf \stopbar \blank
\stopbuffer

\typebuffer \getbuffer

First notice that it is no problem to span multiple lines and that
hyphenation is not influenced at all. Second you can see that
continuous rules are also possible. From such a continuous rule
to a background is a small step:

\startbuffer
\definebar
  [backbar]
  [offset=1.5,rulethickness=2.8,color=blue,
   continue=yes,order=background]

\definebar
  [forebar]
  [offset=1.5,rulethickness=2.8,color=blue,
   continue=yes,order=foreground]
\stopbuffer

\typebuffer \getbuffer

The following example code looks messy but this has to do with
the fact that we want properly spaced sample injection.

\startbuffer
from here
    \startcolor[white]%
        \startbar[backbar]%
            \input zapf
            \removeunwantedspaces
        \stopbar
    \stopcolor
\space till here
\blank
from here
    \startbar[forebar]%
        \input zapf
        \removeunwantedspaces
    \stopbar
\space till here
\stopbuffer

\typebuffer \getbuffer

Watch how we can use the order to hide content. By default rules
are drawn on top of the text.

Nice effects can be accomplished with transparencies:

\startbuffer
\definecolor [tblue] [b=.5,t=.25,a=1]
\setupbars [backbar] [color=tblue]
\setupbars [forebar] [color=tblue]
\stopbuffer

\typebuffer \getbuffer

We use as example:

\startbuffer[sample]
from here {\white \backbar{test test}
    \backbar {nested nested} \backbar{also also}} till here
from here {\white \backbar{test test
    \backbar {nested nested}          also also}} till here
from here {\white \backbar{test test
    \backbar {nested nested}          also also}} till here
\stopbuffer

\typebuffer[sample] \getbuffer[sample]

The darker nested variant is just the result of two transparent
bars on top of each other. We can limit stacking, for instance:

\startbuffer
\setupbars[backbar][max=1]
\setupbars[forebar][max=1]
\stopbuffer

\typebuffer \getbuffer

This gives

\getbuffer[sample]

There are currently some limitations that are mostly due to the
fact that we use only one attribute for this feature and a change
in value triggers another handling. So, we have no real nesting
here.

The default commands are defined as follows:

\starttyping
\definebar[overstrike]  [method=0,dy= 0.4,offset= 0.5]
\definebar[underbar]    [method=1,dy=-0.4,offset=-0.3]
\definebar[overbar]     [method=1,dy= 0.4,offset= 1.8]

\definebar[overstrikes] [overstrike] [continue=yes]
\definebar[underbars]   [underbar]   [continue=yes]
\definebar[overbars]    [overbar]    [continue=yes]
\stoptyping

As the implementation is rather non|-|intrusive you can use bars
almost everywhere. You can underbar a whole document but equally
well you can stick to fooling around with for instance formulas.

\startbuffer
\definecolor [tred]   [r=.5,t=.25,a=1]
\definecolor [tgreen] [g=.5,t=.25,a=1]
\definecolor [tblue]  [b=.5,t=.25,a=1]

\definebar [mathred]   [backbar] [color=tred]
\definebar [mathgreen] [backbar] [color=tgreen]
\definebar [mathblue]  [backbar] [color=tblue]

\startformula
    \mathred{e} = \mathgreen{\white mc} ^ {\mathblue{\white e}}
\stopformula
\stopbuffer

\typebuffer

We get:

\getbuffer

We started this chapter with some words on grouping. In the
examples you see no difference between adding bars and for
instance applying color. However you need to keep in mind that
this is only because behind the screens we keep the current
settings along with the attribute. In practice this is only
noticeable when you do lots of (local) changes to the settings.
Take:

\starttyping
{test test test \setupbars[color=red] \underbar{test} test}
\stoptyping

This results in a local change in settings, which in turn will
associate a new attribute to \type {\underbar}. So, in fact the
following underbar becomes a different one than previous
underbars. When the page is prepared, the unique attribute value
will relate to those settings. Of course there are more mechanisms
where such associations take place.

\stopsection

\startsection [title={More to come}]

Is this all there is? No, as usual the underlying mechanisms can be used
for other purposes as well. Take for instance inline notes:

\startbuffer
According to the wikipedia this is the longest English word:
pneumonoultramicroscopicsilicovolcanoconiosis~\shiftup {other long
words are pseudopseudohypoparathyroidism and
flocci­nauci­nihili­pili­fication}. Of course in languages like Dutch and
German we can make arbitrary long words by pasting words together.
\stopbuffer

\typebuffer

This will produce:

\getbuffer

I wonder when users really start using such features.

\stopsection

\startsection [title={Summary}]

Although under the hood the \MKIV\ bar commands are quite
different from their \MKII\ counterparts users probably won't
notice much difference at first sight. However, the new
implementation does not interfere with the par builder and other
mechanisms. Plus, it is configurable and it offers more
functionality. However, as it is processed rather delayed, side
effects might occur that are not foreseen.

So, if you ever notice such unexpected side effects, you know
where it might result from: what you asked for is processed much
later and by then the circumstances might have changed. If you
suspect that it relates to grouping there is a simple remedy:
define a new bar command in the document preamble instead of
changing properties mid|-|document. After all, you are supposed to
separate rendering and content in the first place.

\stopsection

\doifmodeelse {tugboat} {
    \StopArticle
} {
    \stopchapter
}

\stopcomponent
