documentation and the book module (was reflections on the Drupal 4.3.0 battle)

[Charlie Lowe]

2. Help the documentation team move forward by improving (a) Drupal's
built-in help system and (b) book module. The latter includes (1)
better integration of non-book pages in the book module (Nick Berendsen
has been working on this) and (2) a better review/workflow model to
facilitate the acceptance of updates.

Frankly, I haven't put much effort in the book module because I

believe we

should maintain the Drupal documentation using LaTeX or Docbook but that
is another story.

Just thinking that Drupal should eat it's own dog food here. If the book
module is (or to be) an effective collaborative authoring tool, then
wouldn't it make more sense to expand it's capabilities? Make it more
suitable for producing the documentation and make it easier to integrate
into the help system.

I can understand how LaTeX or Docbook might make a better immediate
solution in terms of

(a) control of formatting
(b) use by the core developers.

But it would seem to do so at the expense of

(c) larger community interaction in developing documentation (for
example, many other users are not comfortable with Docbook or LaTeX or
the process of checking in and out of CVS; keeping it in CVS doesn't
make it very accessible either).
(d) further development of the book module (case in point, Dries's
comment above).

Also, if this line of thinking had been followed in terms of project
development, then would it have made any sense for Drupal developers to
have done a major revision to the project module? Seems as if it have
been easier to have adopted another project solution instead.

Maybe this is just me because I see great potential in the book module,
instead of just as a display mechanism for documentation.

And maybe because I know that it would be easier for me to participate
in documentation writing *if* it was possible to contribute via the
handbook on drupal.org. Right now, because of problems with the module,
workflow difficulties with getting documentation from drupal.org into
the help system, and the fact that much of the documentation used by the
help module is not actually stored in the node table on drupal.org,
contributing is a difficult, discouraging process. For instance, there
are still some updates I wrote for 4.2 a few months ago that are on site
which never made it into 4.3rc for some of these very reasons.

Charlie Lowe
http://cyberdash.com/

[Dries Buytaert]

Post by Charlie Lowe

Frankly, I haven't put much effort in the book module because I
believe we should maintain the Drupal documentation using LaTeX or
Docbook but that is another story.

Just thinking that Drupal should eat it's own dog food here. If the book
module is (or to be) an effective collaborative authoring tool, then
wouldn't it make more sense to expand it's capabilities? Make it more
suitable for producing the documentation and make it easier to integrate
into the help system.

Making the book module suitable involves versioning and workflow that
supports branches: we should be able to retrieve an old version of a book
(not just a page) as well as maintain specific versions.

I have been approached by a publisher that wishes to publish the Drupal
handbook. The current handbook is not suitable because (i) the format is
not suitable for print, (ii) the book lacks consistency (writing style,
mark-up) and is not (iii) well-balanced. Also, we have no means to
generate an index and to do other book-like things.

I believe none of the above is going to happen unless we move the
documentation to CVS and maintain it like we do with the code. I'm
confident that the result will be a better manual.

[Charlie Lowe]

Post by Dries Buytaert

Post by Charlie Lowe

Frankly, I haven't put much effort in the book module because I
believe we should maintain the Drupal documentation using LaTeX or
Docbook but that is another story.

Just thinking that Drupal should eat it's own dog food here. If the book
module is (or to be) an effective collaborative authoring tool, then
wouldn't it make more sense to expand it's capabilities? Make it more
suitable for producing the documentation and make it easier to integrate
into the help system.

Making the book module suitable involves versioning and workflow that
supports branches: we should be able to retrieve an old version of a book
(not just a page) as well as maintain specific versions.
I have been approached by a publisher that wishes to publish the Drupal
handbook. The current handbook is not suitable because (i) the format is
not suitable for print, (ii) the book lacks consistency (writing style,
mark-up) and is not (iii) well-balanced. Also, we have no means to
generate an index and to do other book-like things.

First, I never thought it was trivial to modify the book module to be
more effective :)

Second, what concerns me is that you are suggesting more of a
programmer-friendly method of documentation writing and versioning
rather than a writer/community-friendly one. Formatting for markup and
creating indices are, in my mind, trivial when compared to content
creation and editing for consistency in style. Give me the handbook for
a weekend as it stands right now, and I think I can put it in a
publisher ready markup format. Then we could develop the index from
there. And that only needs to be done once every so often for the
publisher (maybe once every few years). But editing the handbook for
consistency, expanding on content (some areas of the administration
guide are in need of more user-friently, detailed information) is a lot
more work then that and an ongoing project that will continue throughout
versions of Drupal.

So why not, then, devote serious attention to figuring out how to make
Drupal a more powerful publishing engine, rather than simmply abandoning
Drupal as the platform for producing it's own documentation? Isn't that
worth consideration first?

Post by Dries Buytaert
I believe none of the above is going to happen unless we move the
documentation to CVS and maintain it like we do with the code. I'm
confident that the result will be a better manual.

Certainly, if it gets the developers motivated to work on the manual.
But it will discourage myself and others who are not comfortable with
CVS and don't find it a particularly well-suited collaborative writing
environment. What happens if developers are no more motivated as they
are now after you make the change?

[Adrian Rossouw]

Post by Dries Buytaert
I believe none of the above is going to happen unless we move the
documentation to CVS and maintain it like we do with the code. I'm
confident that the result will be a better manual.

+1

I agree with dries.

[Sohodojo Jim]

I am setting up a site with 4.3RC code that will not have public users (no
sign up solicitation). I have the logon block enabled for the admin path
with the idea that if you are staff and want to add content, you just go to
http://mysite.tld/admin (with clean urls configured and working okay). This
is working fine on my intranet development platform but fails on the
deployment platform. However, if I do http://mysite.tld/admin/system (or any
other 'deeper' level admin url request), it works as expected on both
platforms.

When http://mysite.tld/admin fails on the deployment platform, it returns
the site index.php page content with a pre-head message:

warning: main(): stream does not support seeking in
/path/to/mysite/www/error.php on line 19.

while my browser address widget shows http://mysite.tld/admin/

Both are running PHP 4.3.2, however my intranet runs Apache/2.0.43 (Win32)
while my deployment server is Apache/1.3.28.

Any obvious (or not so obvious) reasons why this is happening? And is there
a config fix? Or do I just work around by using an always resolvable admin
logon url link?

--Sohodojo Jim--

[Peter VanDijck]

It seems to me we have 2 audiences:
- (some) developers want to write docs using CVS and some hardcore doc
format.
- (some) less techie contributors want to contribute to the
documentation.

Both have good arguments.

My pov is that editing and writing is indeed a lot more work than
generating an index and such, so we should optimize for that.

Cheers,
Peter

[Dries Buytaert]

Post by Peter VanDijck
- (some) developers want to write docs using CVS and some hardcore doc
format.
- (some) less techie contributors want to contribute to the
documentation.
Both have good arguments.

And what if we'd make a small module that loads a LaTeX or Docbook source
file, presents it in a textarea and lets you update it? You would not be
able to preview your changes but once you click submit, a patch is
generated (against the source file) and send out to the drupal-user
mailing list for people to pick up, test, and - eventually - commit to
CVS.

Post by Peter VanDijck
My pov is that editing and writing is indeed a lot more work than
generating an index and such, so we should optimize for that.

That is exactly why I suggest to use LaTeX or Docbook; they enforce
consistency and take over nearly all visual aspects. Using these tools,
the writer can focus on writing text. They are specifically designed for
documentation/book writing.

Anyway, I give this discussion some more thought. In the mean time, I
rest my case; no significant documentation improvements are going to
happen unless someone puts a lot of time and energy in improving the book
module.

Implementing such improvements is not on my radar of current priorities
because I know that the result will never match that of the manuals on my
book shelves -- or that of a carefully crafted PDF file.

Hopefully someone will stand up and proof me wrong; either with code,
significant documentation updates or a better idea.

[Charlie Lowe]

Post by Dries Buytaert
I know that the result will never match that of the manuals on my
book shelves -- or that of a carefully crafted PDF file.

Hope I'm not beating a dead horse. Here's a demonstration of a clean,
pdf made from the a Drupal collaborative book text.

1) Modified book.module to eliminate the eliminate the class, name and
id tags in <h1>, <h2> and <h3> for the print-friendly version.

2) Saved the print-friendly version available from the link on this page
as html source: http://cyberdash.net/celfa03/node/view/2.

3) Open the text in OpenOffice.

4) Copied the page from the webwriter part of open office and pasted it
into the regular word processing window.

5) Spent 20 minutes modifying the word processing style sheets and
making a few layout adjustments (Note that I'm a little slow with this
since I don't usually do anything but straight text in OpenOffice).

6) Exported to pdf: http://cyberdash.com/temp/calendar.pdf

BTW: Note that the links don't work, but that's because I'm still
unfamiliar with the pdf function in openoffice.

Charlie
http://cyberdash.com/

[Gerhard Killesreiter]

Post by Dries Buytaert

Post by Peter VanDijck
- (some) developers want to write docs using CVS and some hardcore doc
format.
- (some) less techie contributors want to contribute to the
documentation.
Both have good arguments.

And what if we'd make a small module that loads a LaTeX or Docbook
source file, presents it in a textarea and lets you update it? You

I don't know about Docbook, but LaTeX is usually quite scary for people
not used to it. BTW: IIRC there is a Docbook format that lets you
generate LaTeX output.

Post by Dries Buytaert
would not be able to preview your changes but once you click submit,
a patch is generated (against the source file) and send out to the
drupal-user mailing list for people to pick up, test, and -
eventually - commit to CVS.

I would suggest a slightly different approach:

- Maintain the Docs in some format in CVS.
- If update through the web is desired, generate HTML from the Source
(ususally easy).
- Generate a diff against that HTML as you proposed.
- Let somebody else put the changes into CVS.

The problem would be that the person who puts the patches into CVS would
need to convert them to LaTeX or whatever. But this is a much smaller
problem than presenting non-LaTeX users with the task to edit LaTeX
sources.

Post by Dries Buytaert

Post by Peter VanDijck
My pov is that editing and writing is indeed a lot more work than
generating an index and such, so we should optimize for that.

That is exactly why I suggest to use LaTeX or Docbook; they enforce
consistency and take over nearly all visual aspects. Using these
tools, the writer can focus on writing text. They are specifically
designed for documentation/book writing.

That is correct, although you can spend a lot of time with making your
results perfect. (Eg: in German texts you have to remove a lot of
ligatures by hand (or by a programm that I wrote :)).

Post by Dries Buytaert
Anyway, I give this discussion some more thought. In the mean time,
I rest my case; no significant documentation improvements are going
to happen unless someone puts a lot of time and energy in improving
the book module.

I think that using the limited workflow possibilities that we
have (queue.module) would help. It should not be available to the
public, but to those who have the "admin node" permission now. From
several comments I infer that it is not available on Drupal.org.

By using queue.module, the people who update/commit book nodes would at
least get some feedback on what is happening to their node.

Post by Dries Buytaert
Implementing such improvements is not on my radar of current
priorities because I know that the result will never match that of
the manuals on my book shelves -- or that of a carefully crafted PDF
file.

You always need to put in some work by hand. This is what I learned from
a addressbook project. I thought "this is going to be easy, it is all in
the database". It wasn't.

Post by Dries Buytaert
Hopefully someone will stand up and proof me wrong; either with
code, significant documentation updates or a better idea.

I've never been good with writing docs, but I might have some ideas (and
even some code: workflow.module) for better workflow.

Cheers,
Gerhard

[Kjartan Mannes]

I've been following the various discussions on documentation and have
had some discussions on the side with various people as well, but
honestly I don't know much about DOCBOOK or LaTeX to comment on how they
can be used.

What I have noticed is that the needs haven't really been discussed,
people seem to have jumped to solutions.

Currently Drupal is documented in part in the source code and in part in
the book module on drupal.org. This presents a few challenges as to
update the documentation you have to figure out where it is stored. If
it is in the book you can update it, but then it has to be approved by
someone who has access to do so. If it is in the source code someone
needs to submit a patch and commit it. This presents a maintenance and
administration problem -- hard to update and hard to see what changes
have been made and approve them, and occasionally something gets lost or
forgotten.

Another problem is that Drupal is a moving target. The documentation for
Drupal 4.3.0 won't be the same as the documentation for 5.0.0. This
makes it hard for the end users to find the proper documentation for
the release they are currently using, and causes problems between
releases as the book becomes updated some sections are irrelevant for
the current release, but apply to a future release.

We should also have a set of formatting and style standard so all pages
look the same.

What I think we need is:
* A way to bring all the documentation into one place so it can be
edited independently of source code or anything else. This of course
means that Drupal somehow has to be able to access the documentation
for the help.module, or we have to give up on combining them and
maintain them separate.

* Support for several versions of the documentation for each of the
Drupal releases.

* Transparent updating that can be reviewed by the community, either
before it is included or afterwards (requires revision control of
some kind).

* A consistent form of structure and styling.

* Editable in some form by people who are not necessarily technical in
nature.

Some of this introduces new challenges. If we take the documentation out
of the Drupal source then we also need to do that for contributed
modules, so we need a way that they can include documentation or support
both external and internal help.

No matter what solution we end up with it results in a need for some
development. If we use the book module then it needs to be extended to
support branches and enforce structure, we also need some way to bring
the documentation back into Drupal for the internal help.

Using DOCBOOK or LaTaX with CVS greatly increases the bar for users to
create documentation as there doesn't seem to be any easy to use tools
available, but I could be wrong about that.

We can combine the two in some form where the documentation is stored in
a DOCBOOK format in CVS, but we have a module on drupal.org that lets
you edit the page online (and send a diff to the -user list for someone
technical to apply it?).

Or maybe there is some other alternative that hasn't come up yet.

[Dries Buytaert]

Post by Kjartan Mannes
* A way to bring all the documentation into one place so it can be
edited independently of source code or anything else. This of course
means that Drupal somehow has to be able to access the documentation
for the help.module, or we have to give up on combining them and
maintain them separate.
* Support for several versions of the documentation for each of the
Drupal releases.
* Transparent updating that can be reviewed by the community, either
before it is included or afterwards (requires revision control of
some kind).
* A consistent form of structure and styling.
* Editable in some form by people who are not necessarily technical in
nature.

Additional requirement:

* The documentation should be translatable.

[Charlie Lowe]

Post by Kjartan Mannes
* A way to bring all the documentation into one place so it can be
edited independently of source code or anything else. This of course
means that Drupal somehow has to be able to access the documentation
for the help.module, or we have to give up on combining them and
maintain them separate.
* Support for several versions of the documentation for each of the
Drupal releases.
* Transparent updating that can be reviewed by the community, either
before it is included or afterwards (requires revision control of
some kind).

I seem to remeber that someone suggested that the next upgrade to the
help system might include storing help as HTML files. Assuming this were
the case, consider

1) Documentation development for CVS version is done on drupal.org in
the book.module. Older versions would occur by updating the HTML files
in the CVS repository. This allows the larger community to be involved
in the main documentation creation and updating.

2) Each documentation page needs a unique identifier. Could just use the
identifiers generated by the node. Drupal.org would need a table of
those identifiers and their corresponding file in CVS. Each night,
before CVS nightly tarballs are created, Drupal would need a script to
diff the HTML files stored in CVS with book module pages.

3) When a new module is added to contrib and documentation work is begun
on site in the book module, the table of identifiers on drupal.org would
have to be updated manually. A drawback, but this probably should not
happen too often. And a whole lot simpler than the current system of
building a page on the book module, then updating the code in CVS within
the module file.

4) Translations into other languages of documentation can be done in CVS
with the HTML files (note also that HTML versions of documentation also
makes it very easy for anyone to customize their documentation for their
individual drupal site).

5) I've seen a Docbook HTML conversion tool. Seems as if these HTML
files could be converted to Docbook, then a style sheet applied, for
doing fancy pdf work or producing the print publication Dries mentioned.

Post by Kjartan Mannes
* A consistent form of structure and styling.

This is the hard part no matter which way you go. In my experience,
publishers and other projects with many writers require the use of an
editorial style sheet. However, even with qualified, dedicated writers,
an editor(s) still has to go behind the writers and create more
consistency to make someting of publishable quality.

Post by Kjartan Mannes
* Editable in some form by people who are not necessarily technical in
nature.

Thus the use of the book module, for the greater work on the
documentation, and HTML, for permanent storage and versioning. HTML is
about the most standard mark up language you are going to find that is
familiar to all of the Drupal community.

Then let those doing specialized projects with the documentation convert
HTML to the markup language of their choice, just as I did with the
sample pdf I posted yesterday. After all, the differences between the
pdf I put up and the original book module output are just another style
sheet used by the particular tool that I like. With HTML as the base
mark up for documentation, anyone else can do the same, whether it be
with Docbook, an XML editor, Dreamweaver, Microsoft Office, whatever.

Charlie Lowe