ePub demystified – Tomorrow’s e-book reader the web browser?

Thank you. I was starting to wonder if I had somehow missed reading the “real” specification, since what I saw seemed pretty straightforward.

Leveraging XHTML makes so much sense, not just in bootstrapping readers (as you say), but on the back-end too. When I wanted to export epub for my platform I just grabbed existing open tools that generated XHTML and wrapped those in some custom code to generate the OPF package document and zip things up. It only took a few hours to throw together something that validated, and that time included reading the epub spec.

Jon, while I agree with much that you say here, you’re trying slip a whopper past your audience (IMO):

It is actually not correct to call them incompatible or that they contribute to the Tower of eBabel since the same reading application for one can be adapted with relative ease for another, and inter-conversion is possible.

It’s exactly the proliferation of these very-similar-but-not-mine formats that causes the Tower of eBabel that David is so fond of. And clearly they are incompatible, because there is no app developed for either which also works with the other.

Close-isn’t-bad is exactly the wrong way to think about ebook formats. Different-but-not-really-different is a killer for the industry.

It would be nice if there were some sort of directory of ePub-supporting software. I’m interested in the format, but when I went looking for a converter the other day, the only one I could find any information about was the BookGlutton one which does not currently support CSS, rendering it somewhat less-than-useful for me. Also, what apps can read it? Sure, there’s Adobe Digital Editions for the PC, but (for example) are there any J2ME readers that’ll run on my new mobile phone?

Good question, Dan.

First, the YahooGroup ePub-Community is a place where information about ePub-related tools may be posted.

Second, the tools to produce ePub from a given XHTML document (or set of documents) and associated CSS and JPEG/PNG images, will be quite easy to construct.

This is the amazing thing that everyone is overlooking, and which I tried to address in this article. ePub is, for all intents and purposes, a packaged XHTML 1.1 “web site”. Yes, one has to add a Package file (which is easy to compose — a tool to auto-generate the Package which queries for the metadata and such should be almost trivial to write) and then ZIP it properly (again, a tool to do so should be easy to build — already existing open-source code has probably been written that can be adapted.)

I’d be happy to consult with anyone who is contemplating building the simple ePub creation tools when the XHTML and CSS are already created.

I also note that for books the XHTML documents are very easy to create in a text editor since one doesn’t have to worry about adding all the stuff today’s web pages typically have, such as various menus, multiple panes, and such.

Liza’s comment above illustrates how easy it is to create ePub from text editors and such. With a couple tools to speed-up the process, it can be made easy even for novices to produce ePub using humble NotePad for authoring the XHTML and CSS (which can start off as “fill-in-the-blank” templates.)

To summarize, let’s not view ePub as this complicated format that requires some commercial tool (such as InDesign) to create. This is the primary purpose of my “demystified” article, to empower small publishers and others who wish to produce ePub to go ahead and learn the few things they need to learn to do this, as well as to encourage developers to write the simple tools as just noted. Let’s get moving!

OK, it’s just a packaged “web site”. But it uses the “Package” instead of what a web site uses, which is typically an “index.html” or “default.html” file. Fair enough.

On the eBabel front, let’s examine the other web site packaging formats. Googling for “web page archive format” gives one an interesting list. Microsoft’s MHTML, introduced in IE 5, documented in RFC 2557, and apparently also supported by the Opera browser. And Apple’s WebArchive, used by Safari. The Library of Congress’ WARC, which they’re using to preserve Web site captures, and is a draft ISO standard. WARC is based on ARC_IA, the format used by the Internet Archive. There’s also MAF, the Mozilla Archive Format for Firefox 3, Christopher Ottley’s project.

Can you say something about how similar and/or different ePub is to these other pre-existing and widely used formats?

Here’s information on the epub converter that I developed, which expects TEI as the source XML format. It leverages entirely free tools and is free itself. Just give it a TEI document (it expects a chaptered work) and it spits out an epub book with each chapter as an individual XHTML page.

If there’s interest I’d be happy to adapt it to other markup formats, or package it as a binary executable (right now it requires some installation of tools in the Python language).

By the way, I think the proliferation of web site archive formats makes it clear that there’s little agreement about how to do this. One of the clear disagreements is about how to handle dynamic content in an archive, like Javascript.

Thanks Bill for pointing out the other “web site packaging” formats. Other than MHTML, I’ve not heard of the others. So give me a few days to study them and compare them with IDPF’s OCF (which of course is based upon what the OpenOffice folk use, closely related to JAR.)

It definitely would behoove IDPF to look again at packaging technologies.

Wow, Liza. I invite you to post the info about your ePub converter to the epub-community group!

…And Liza, I, too, will welcome such info for the TeleBlog. Nice going! David

Note that there are already link types that can be embedded in each HTML document to define next/prev links.

Most browsers have never made these visible, but it seems you can do it yourself with JavaScript.

http://www.alistapart.com/articles/dynanav

“Second, the tools to produce ePub from a given XHTML document (or set of documents) and associated CSS and JPEG/PNG images, will be quite easy to construct.”

I’m not denying that the concepts involved in ePub seem pretty easy to grasp, and that it probably wouldn’t be terribly difficult for those with the right skills to write a program to do the conversion.

My point, however, is that I shouldn’t have to. Unlike other standards (such as the JPEG and PNG image formats you mention) there does not seem to be any “reference implementation” for ePub. If it’s not hard to write code to handle, then it would be advantageous if someone were to step up and create an open source ePub-handling library which developers can simply plug in to their application. Sure, it will have some problems to begin with, but every project does, and by making it open source, anyone with the right skills could join in and help instead of it becoming the sort of proprietary spaghetti we all dread.

As far as I myself am concerned, at the moment my ebook library consists mainly of HTML documents (usually exploded from Microsoft Reader files, although I also have some Baen books which are natively HTML) which are of varying code standards. Some have CSS and images and some do not. That means that as well as an ePub packaging utility I’d also need something which can take any old HTML file and process it into whichever XHTML version ePub is expecting, rather than simply saying “hey, this isn’t XHTML, rejected!”. In addition, it also needs to be able to follow links, as for example Baen books are often split into chapter files, whereas MS Reader books are a mixture depending on the application that was used to author them.

@Dan: The BookGlutton tool currently supports inline CSS as well as external link references to CSS files. This is mandated by the OPS spec:

“The link element allows for the specification of various relationships with other documents. Reading Systems must recognize external style sheet references specified via the href attribute and the associated rel attribute (for the values rel=”stylesheet” and rel=”alternate stylesheet”.)”

This means single file XHTML conversions can still use CSS, either in external files (via URL) or inline (via style elements).

As for the flavor of XHTML needed for epub, XHTML 1.1 is bleeding-edge enough that not only will you not find many web sites out there using it, you’ll have trouble conforming to it. It’s extremely intolerant of much current Web content. The BookGlutton tool will convert HTML to XHTML Strict, which for all practical purposes (such as being viewable in Digital Editions) is enough.

Soon we’re releasing an upgrade to the converter which will support multi-file zip archives with JPG, SWF, GIF, XML, CSS and PNG files.

Aaron

To comment on Aaron’s comment, the more precise way to describe XHTML 1.1 is that it does not support all the elements and attributes that one may use in legacy (“tag soup”) HTML. XHTML 1.1 is very close to XHTML 1.0 Strict, close enough that many XHTML 1.0 Strict documents can be DOCTYPE “rebranded” (for those who want to validate to a DTD) and they will work “as is”. Any differences can readily be fixed using a simple substitution script or even a quick edit using a text editor.

In fact, here are the actual differences between XHTML 1.0 Strict and XHTML 1.1:

1. ‘lang’ attribute becomes ‘xml:lang’ for XHTML 1.1.

2. On the <a> and <map> elements, the ‘name’ attribute is not supported in XHTML 1.1. Use ‘id’ instead.

For those with really legacy HTML, “HTML Tidy” may be used to get you much of the way to XHTML 1.0 Strict.

Btw, here’s an interesting article describing the benefits of XHTML 1.0 Strict over “tag soup” HTML. I probably don’t need to comment why IDPF, from the beginning in 1999, specified XHTML.

To make another point on Aaron’s comment, ePub (and the underlying OPS spec) is an e-book format, not a web page. Typical HTML authoring for web pages is a whole lot different than that used for books. In fact, the markup needed for book use is almost trivial compared to that needed for today’s complex “space shuttle cockpit layouts” used in most web pages, which almost always require a tool to author. So one has to be careful not to compare apples with oranges – both are fruit, but are quite different fruit from each other.

Comically, in my work with epub that id/name change is the only problem I hit.

The publicly-available TEI/XHTML stylesheet will automatically convert xml:id (from TEI) to HTML id attributes. It will also helpfully create anchor links to those paragraphs.

If you ask it to generate HTML 4.0, you get:

<p id=”id123″><span name=”id123″/>

which as Jon points out is not valid XHTML 1.1 because ‘name’ has been removed from the spec. (In fact it wasn’t valid anyway because ‘name’ was never allowed on a span.)

…but if you ask it to generate XHTML you get:

<p id=”id123″><span id=”id123″/>

…which is also invalid XHTML 1.1 because of the duplicate IDs.

I couldn’t find the right combination of arguments to the base TEI stylesheets to get the behavior I wanted so I had to modify them directly. I’ll be passing a bug report on to the TEI folks.

To put a finer point on Jon’s comments, HTML Tidy will not reliably produce epub files which validate in epubcheck. It will produce XHTML Strict, but not strictly, if you get my drift. In other words, it might leave 574 name attributes and disallowed br tags (between divs, for example). Removing these by hand to get a compliant epub is not worth the effort. Even a script would be wasting cycles trying to conform to XHTML after converting from “tag soup.”

This is a conflict between declared and actual doctype, as seen in the validation errors for the OPS spec itself:

http://validator.w3.org/check?uri=http%3A%2F%2Fwww.idpf.org%2F2007%2Fops%2FOPS_2.0_final_spec.html&charset=%28detect+automatically%29&doctype=XHTML+1.1&group=0

I disagree that it’s not worth the effort. For one thing, removing all the name attributes is trivial in XSLT. I do agree that cleaning up arbitrary content might be intractable, but most publishers will have a workflow that starts with at least moderately-consistent source documents, and any inconsistencies can be tweaked at the end of the pipeline.

But the larger point is, if validation is not part of your workflow because it always produces error messages (even if they’re non-critical), then people learn to ignore the validation output altogether, or even disable it. The whole process of validation becomes meaningless, and ultimately systems will produce seriously non-conforming documents.

This is bad for everyone: the consumer, because some ostensibly-portable documents now look bad on some devices; the device-maker, because they get blamed for not handling invalid markup; and the publisher, because readers are turned off by the whole ebook experience.

Software engineers are the first line of defense here, and we should whenever possible build tools that include validation, and push hard for documents that can be accurately encoded.

Of course validation is part of our workflow. It should be for everyone, I agree. And publishers should be expected to conform, sure. But I disagree that it’s worth the effort to strip name attributes. In many cases, it has no effect on the appearance of documents. In others, it’s actually necessary for legacy systems, or new systems based on legacy engines.

Being strict for strictness sake stifles innovation and development. The e-book format crowd needs to learn lessons from the Web and loosen up.

When we have HTML with <br> at the block level, that HTML is badly broken.

Publishers will discover as they move to ePub that the XHTML they produce will likely be better formed and much more repurposeable.

It’s about time that the e-book industry grows up and does things right – the winners will be the publishers, small and large.

Jon Noring said “Future web browsers may even natively incorporate the ability to seamlessly present ePub and similar DOT-based framework publications.”

That would be great! The acceptance of e-books would accelerate if Firefox, Safari, Opera, IE and other browsers could read a dominant e-book format. This approach would leverage the pre-existing wide deployment of web browsers on large and small devices.

The publisher Tor has been releasing e-books in a simple format that is readable by browsers. It consists of a zipped file containing an HTML file together with a folder containing image files. The images are in JPG and GIF formats. However, reading this format raises a difficulty. How can a reader using a browser create a “bookmark” or “stopping point” within a very long HTML file when there is no convenient anchor point?

The “bookmark” would point to a phrase within a text or an offset within an HTML file. Do any browsers implement this function? This should be doable in the current generation of web browsers I think for simple HTML files. It probably would be tricky in complicated HTML files.

I have not used an ePub reader. How do you create a “bookmark” within a text while reading? Does it automatically paginate? Do page numbers change when fonts are resized?

Much thanks to the posters on this thread. I appreciate the expertise that is displayed and I am grateful for your attempts to move e-books forward.

The Openberg Lector addon for Firefox (https://addons.mozilla.org/en-US/firefox/addon/5275) used to allow one to read epub documents (among other formats) in the browser.

Lector was reviewed on Teleread a year ago (http://newteleread.com/wordpress/blog/2007/12/04/an-e-reader-that-accepts-any-xml/).

Unfortunately it has not been updated for the latest version of Firefox, and the link to its homepage redirects to an empty blog. It’s a great shame, because this was a promising project. Does anyone know any more about it?