% language=uk

\startbuffer[bibtex-commands]
\startsetups bibtex-commands
    \def\MF  {MF}
    \def\MP  {MP}
    \def\TUB {TUGboat}
    \def\Mc  {Mac}
    \def\sltt{\tt}
    \let\acro\firstofoneargument
\stopsetups
\stopbuffer

\getbuffer[bibtex-commands]

\usemodule[abr-01]

\startcomponent hybrid-bibtex

\environment hybrid-environment

\startchapter[title={Bibliographies}]

\startsection [title={Introduction}]

Already early in the history of \CONTEXT\ Taco Hoekwater wrote a
module that dealt with \BIBTEX\ databases in a \CONTEXT\ like way.
Personally I never had to use a bibliography so I'm far from an
expert in this area. However, going from some text database format
to something typeset is generic enough for me to be involved.

The involvement started when \MKIV\ showed up. Because quite some
core mechanisms have been reimplemented, also some that the module
used, a dedicated \MKIV\ variant had to be made. This was not that
hard to do as it mostly meant stripping code and replacing the
specific reference mechanism by one using lists. That way we got
a few bonus features but in general we can say that the module is
downward compatible.

Already a while ago Taco and I discussed supporting bibliographies
that use \XML\ as format and although we have not settled on some
standard it makes sense to explore the possibilities. The
advantage of using \XML\ is that we can use the built in subsystem
for filtering and manipulating entries.

This chapter is dedicated to Thomas Schmitz who not only use
\BIBTEX\ but also has used \MKIV\ \XML\ right from the start and
provides valuable feedback on both subsystems.

Keep in mind that eventually we will provide a high level interface
so that users won't notice much of a difference unless they want to
go beyond what they use now.

\stopsection

\startsection [title=Sessions]

As usual in \CONTEXT, we organize the featureset in such a way that
we can group them and use several such sets in one documents without
interference. It all starts by defining a session:

\startbuffer
\definebibtexsession [somebibtex]
\stopbuffer

\typebuffer \getbuffer

Next we register a couple of databases (from the beebe collection on
\TEX live):

\startbuffer
\registerbibtexfile [somebibtex] [tugboat.bib]
\registerbibtexfile [somebibtex] [komoedie.bib]
\stopbuffer

\typebuffer \getbuffer

The files are loaded immediately and you can check this by looking
at the log where we get a report like:

\startntyping
mkiv lua stats : bibtex load time - 0.125 seconds (1524045 bytes, 2745 definitions, 7 shortcuts)
\stopntyping

In a bibtex database there can be symbols (shortcuts to strings).
Although these are expanded, it has no consequence for memory
usage as \LUA\ hashes its strings.

% used cite tagging: \registerbibtexentry [somebibtex] [Hagen:TB26-2-152]

When we're done registering we need to prepare the session:
\footnote {The convert option will be discussed later.}

\startbuffer
\preparebibtexsession [somebibtex] % [convert]
\stopbuffer

\typebuffer \getbuffer

Say that we want to typeset a table with the publications of where
we only list the author and title. As we use the \XML\ interface
we do so using setups:

\startbuffer
\startxmlsetups bibtex:one
  \starttabulate[|Bl|p|]
  \NC tag   \NC \xmlatt{#1}{tag} \NC\NR
  \NC author\NC \xmlfilter{#1}{/field[@name='author']/context()} \NC\NR
  \NC title \NC \xmlfilter{#1}{/field[@name='title' ]/context()} \NC\NR
  \stoptabulate
\stopxmlsetups
\stopbuffer

\typebuffer \getbuffer

We will now apply this setup to the loaded tree:

\startbuffer
\startxmlsetups bibtex:bibtex
    \xmlfilter{#1}{
        /entry[@category='article']
        /field[@name='author' and (find(text(),'Hans Hagen')
            or find(text(),'Taco Hoekwater'))]
        /../command(bibtex:one)
    }
\stopxmlsetups
\stopbuffer

\typebuffer \getbuffer

We now apply this setup to the session:

\startbuffer
\applytobibtexsession[somebibtex][bibtex]
\stopbuffer

\typebuffer

\bgroup
    \setups[bibtex-commands]
    \getbuffer
\egroup

If this is the first time you see \MKIV's \XML in action you might
be confused by what happens here. When we apply the \type {bibtex}
setup (the second argument), we expand a predefined setup that
looks as follows:

\starttyping
\startxmlsetups bibtex
    \xmlregistereddocumentsetups{#1}{}
    \xmlsetsetup{#1}{bibtex|entry|field}{bibtex:*}
    \xmlmain{#1}
\stopxmlsetups
\stoptyping

Here \type {#1} represents the root node of the current database.
Three elements are mapped to their own name, prefixed by \type
{bibtex:}. In the previous examples we defined the \type
{bibtex:bibtex} one, which will be applied to the root.

Here are a few more predefined setups:

\starttyping
\startxmlsetups bibtex:format
    \par
    \edef\currentbibxmlnode{#1}
    \xmlcommand{#1}{.}{bibtex:\currentbibtexformat:\xmlatt{#1}{category}}
    \par
\stopxmlsetups

\startxmlsetups bibtex:list
    \xmlfilter{#1}{/bibtex/entry/command(bibtex:format)}
\stopxmlsetups

\startxmlsetups bibtex:bibtex
    \xmlfilter{#1}{/entry/command(bibtex:format)}
\stopxmlsetups
\stoptyping

The first one apply a setup to the current node (indicated by the
period).

\starttyping
bibtex:apa:article
\stoptyping

Such setups are defined elsewhere and you can imagine that they look
more complex than what we've seen sofar. But you seldom have to deal
with that.

The second and third setups apply the format to an entry. However, there
is a subtle difference. The second one is called as follows:

\starttyping
\applytobibtexsession[somebibtex][bibtex:list]
\stoptyping

As \type {bibtex:list} is a stand|-|alone setup, it will get the
document root passed, and therefore we need to explicitly add that
root, although the following two calls give the same results (watch
the forward slashes):

\starttyping
\xmlfilter{#1}{/bibtex/entry/command(bibtex:format)}
\xmlfilter{#1}{entry/command(bibtex:format)}
\stoptyping

The \type {bibtex:bibtex} setup however, is using an indirect
approach and only comes into action via the already mentioned
\type {bibtex} setup. In that setup the \type {\xmlmain} command
will expand the root element and when it sees the bibtex element,
it will call the associated \type {bibtex:bibtex} setup. So here
we need to call the \type {bibtex} setup.

\starttyping
\applytobibtexsession[somebibtex][bibtex]
\stoptyping

Let's summarize what is needed to typeset a whole database:

\starttyping
\definebibtexsession  [somebibtex]
\registerbibtexfile   [somebibtex] [tugboat.bib]
\preparebibtexsession [somebibtex] [convert,strip]
\applytobibtexsession [somebibtex] [bibtex:list]
\stoptyping

Here we use the predefined \type {bibtex:list} filter. Of course
you need to define commands that are uses in the database.

\stopsection

\startsection [title={The database}]

The \XML\ database is quite simple and has the form (we omitted some
fields):

\starttyping
<bibtex>
  <entry tag="hagen:tb19-3-311" category="article">
    <field name="number">3</field>
    <field name="bibdate">Fri Jul 13 10:24:20 MDT 2007</field>
    <field name="author">Hans Hagen</field>
    <field name="journal">TUGboat</field>
    <field name="title">{Visual Debugging in \TeX, Part 1: The Story}</field>
    <field name="ISSN">0896-3207</field>
    <field name="year">1998</field>
    <field name="pages">311--317</field>
    <field name="volume">19</field>
  </entry>
</bibtex>
\stoptyping

It is good to keep in mind that we lowercase the name and
category attributes.

By default there are no setups for the one character elements
but if you need them you have to use the bibtex namespace, e.g.:

\starttyping
\startxmlsetups bibtex:field
    \xmlflushcontext{#1}
\stopxmlsetups
\stoptyping

\stopsection

\startsection [title={Sorting}]

% {\em maybe also per session}

We can sort entries. For that we need to define a sort setup.
First we create a sort vector based on some fields. The first
argument (\type {bibtex}) is the sort vector.

\startbuffer
\startxmlsetups bibtex:entry:getkeys
    \xmladdsortentry{bibtex}{#1}
        {\xmlfilter{#1}{/field[@name='author']/text()}}
    \xmladdsortentry{bibtex}{#1}
        {\xmlfilter{#1}{/field[@name='year'  ]/text()}}
    \xmladdsortentry{bibtex}{#1}
        {\xmlatt{#1}{tag}}
\stopxmlsetups
\stopbuffer

\typebuffer \getbuffer

In the next setup we see this sorter being initialized. After that
we filter some entries and add them to the to list of keys. Then
we sort that list and flush it afterwards.

\startxmlsetups xml:bibtex:sorter
    \xmlresetsorter{bibtex}
  % \xmlfilter{#1}{entry/command(bibtex:entry:getkeys)}
    \xmlfilter{#1}{
        bibtex
        /entry[@category='article']
        /field[@name='author' and find(text(),'Knuth')]
        /../command(bibtex:entry:getkeys)}
    \xmlsortentries{bibtex}
    \xmlflushsorter{bibtex}{bibtex:entry:flush}
\stopxmlsetups
\stopbuffer

\typebuffer

\bgroup
    \setups[bibtex-commands]
    \getbuffer
\egroup

The flusher simply shows some fields. You can do anything you
want here with the content.

\startbuffer
\startxmlsetups bibtex:entry:flush
    \xmlfilter{#1}{/field[@name='author']/context()} / %
    \xmlfilter{#1}{/field[@name='year'  ]/context()} / %
    \xmlatt{#1}{tag}\par
\stopxmlsetups
\stopbuffer

\typebuffer \getbuffer

The setup that brings this all together is applied to the
whole tree with the following command.

\startbuffer
\xmlsetup{bibtex:somebibtex}{xml:bibtex:sorter}
\stopbuffer

\typebuffer

The result is:

\startpacked
\getbuffer
\stoppacked

You can call up the list of keys with

\startbuffer
\xmlshowsorter{bibtex}
\stopbuffer

\typebuffer

In our case this gives:

\startpacked
\getbuffer
\stoppacked

\startsection [title=Encodings]

It is a sure bet that many existing databases will use the
traditional \TEX\ accent building commands. As in \MKIV\ we live
in an \UNICODE\ universe, such commands are translated into \UTF\
sequences when the database is loaded. When we pass the \type
{convert} options to the preparation command, the entries will be
cleaned up and accent commands will be replaced by proper \UTF\
sequences. This helps the sorter.

\stopsection

\startsection [title=Messed up entries]

As the \BIBTEX\ fields contain \TEX\ code we need to process the
content as \TEX. This is why in the previous examples we applied
the \type {context()} finalizer. The fact that we have \TEX\ code
means that such databases are rather bound to some macro package.
For our purpose we had to define a few macros:

\typebuffer[bibtex-commands]

You can best do this grouped so that there is no interference with
existing code. You can collect definitions in a setup or buffer and
flush that one inside the group.

However, we also provide another method. A second argument to the
preparation command gives options. Examples of options are \type
{convert}, which converts entries to proper \UTF, and \type {strip}
which converts commands and strips redundant braces.

\starttyping
\preparebibtexsession [somebibtex] [convert,strip]
\stoptyping

All commands are mapped onto \type {\bibtexcommand} which defaults
to using predefined local commands. You predefine such a local
command with:

\starttyping
\defbibtexcommand\MF    {MF}
\defbibtexcommand\MP    {MP}
\defbibtexcommand\TUB   {TUGboat}
\defbibtexcommand\Mc    {Mac}
\defbibtexcommand\sltt  {\tt}
\defbibtexcommand\acro#1{#1}
\stoptyping

If you use a database like \type {tugboat.bib} you will need quite some
more definitions. Unknown commands are reported on the console. If a
command is available in \CONTEXT\ it will be used unless a specific one
is defined.

Here's a setup that shows what goes on inside:

\starttyping
\startxmlsetups bibtex:show
    \xmlshow{#1}
\stopxmlsetups

\applytobibtexsession[somebibtex][bibtex:show]
\stoptyping

\stopsection

\startsection [title=Traditional usage]

In this section we will describe how you can use this approach
as a drop in for the traditional, pure \TEX\ based one.

\stopsection

\stopchapter

\stopcomponent
