ePub standard is not really a standard – just a guideline

Paul,

I think you are being quite unfair to the IDPF. ePub is most certainly a standard, just as much as XHTML, CSS, and the other technologies it is based on are standards. The problem here is not the IDPF, the problem is the device and software manufacturers who are not following the standard properly, and who are trying to hack the standard.

Adobe and Apple are both breaking the standard, and that is affecting our ability to have ePub files that look good on all devices. Just like the browser wars of the ’90s, the current state of the ePub world is dependent on how the people in charge of displaying the content actually display it.

For example, Adobe:
– supports its own XPGT template system instead of pointing designers to CSS for display formatting.
– supports its own page-map standard, not the NCX PageList standard for linking page numbers with the print book
– Does not support some basic formatting like font-variant: smallcaps
– does not ship ADE or the SDK with a default font that actually displays a decent amount of Unicode characters. The software does allow embedded fonts, which can help that issue, but that is a silly way to work around an issue that should not be present.

Also for example, Apple (in iBooks):
– does not support the use of CSS3 for page layout, even though its official docs say to use it
– does not support the NCX PageList standard even though its official docs say to use it.
– Overrides font formatting specified by the designer, even when that formatting is done properly according to CSS
– Does not support font embedding
– Overrides default paragraph alignment by allowing the user to set justified or flush-left alignment

These and many other issues are what break ePub books, not the IDPF. Until the developers of these systems get over their petty differences and work together, we will continue to have issues like this.

If they don’t support the standard, why do they get to call the files ePub?
Because the “standard” has no teeth.
Because there is no penalty to violating the spec.
Because non-compliance gets swept under the rug.
A standard with no teeth behind it is no standard, but a guideline.

Mr Biba’s right on this one; just because you call a spec a standard doesn’t make it one, no matter how many committees rubber stamp it.

Are you willing to pay what it would cost the IDPF to try to enforce a “standard” as you define it? What about the W3C? Should they have to enforce the HTML standard? The cost of doing that would be exorbitant, and well beyond the means of a non-profit trying to help the industry get itself pulled together.

According to Webster, a standard is:
3: something established by authority, custom, or general consent as a model or example

Wikipedia (fwiw) says that a technical standard “is an established norm or requirement. It is usually a formal document that establishes uniform engineering or technical criteria, methods, processes and practices.”

Note that both definitions do not require a standard to be enforced to be able to be called a standard.

A standard like html or any of the sgml derivatives, can be defined as such by a body as the IDPF. Entirely different is how different engineers code applications that parse and read that standard.

In this instance, IDPF might have exercised a good deal more control, had they been at all savvy about what they were getting into. For instance, they might have come up with an official, trade-marked logo. Then they might have insisted anybody marketing an application claiming to be an epub reader or generator pass that application through the IDPF — IF they want to get to display the official EPUB logo.

Any drinks company can make and market ‘cola’ but only an officially-licensed one can market Coca-Cola — and to do so they must use the official formula.

The IDPF has shown itself to be pretty darned clueless in all of this. But it’s a committee, and not all the committee members have the same goals in mind. The lack of coherent or sensible results from IDPF might be due to these internal conflicts, but that’s just my guess. Another guess might be they wanted this sort of confusion so as to encourage innovation among ebook reader software and device manufacturers.

But the inconsistencies mentioned here must be making matters worse, not better, for publishers — and I thought the IDPF was doing all this to help publishers?

This is how a real universal standard (audio CD) is enforced:
http://en.wikipedia.org/wiki/Red_Book_(audio_Compact_Disc_standard)

Also, note how the enforcers took note of violations and promptly called out the violators:

“Some major recording publishers have begun to sell CDs that violate the Red Book standard. Some do so for the purpose of copy prevention, using systems like Copy Control.
“Some do so for extra features such as DualDisc, which includes both a CD layer and a DVD layer whereby the CD layer is much thinner, 0.9 mm, than required by the Red Book, which stipulates a nominal 1.2 mm, but at least 1.1 mm. Philips and many other companies have warned them that including the Compact Disc Digital Audio logo on such non-conforming discs may constitute trademark infringement.”

Yes, it costs money to enforce a standard. That’s why you *charge* for use of the Logo and the trademark. If the standard is at all valuable, the moneygrubbers will pay.

It’s interesting that you cite a trademarked proprietary standard as a model for how ePUB should be enforced; some people would consider a trademarked proprietary standard to be an unconscionable violation…

And some people consider trademarks (and patents and copyright) unconscionable, too. You can find just about any opinion if you look far enough.

On the other hand, the trademarked propietary solutions (CD, DVD, HD-DVD, cassette, iOS, Windows, etc) result in universally interoperable solutions. When you buy content in *those* formats it plays the same everywhere it is supposed to play. You can’t say that about ePub.

Ideological purity might satisfy theoreticians and pundits but what consumers want is working solutions.

Deeds, not words.

ePub is at risk of failing to deliver true interoperability if the standard isn’t tightened up and soon.

For the most part, I think the original author (Martyn Daniels) is placing the blame on the wrong people or misunderstanding the standard.

1. The ePUB for the iPad needs a new-standard CSS for iPad-ePUB which will generically apply colour for Part/Chapter/Sections/Noteboxes/CodeListing and others. Fairly straightforward but not required until colour was delivered.

Yea, so the iPad uses its own default CSS. How is that the fault of ePub? This is standard operating procedure for web browsers, and anyone familiar with CSS would know that. Others would say that the ability to have different default CSS settings is an advantage for the users, such as allowing user-customized styling.

2. Image/Graphic format: iPad-ePUB supports JPG and PNG formats, but PNG (transparent) is recommended. Transparency of images helps’ if a coloured Notebox has an inline equation then image will not render white-patch of inline equation image.

Umm… duh? PNG is of course better if you want transparency, which is impossibly in JPG. But if your image is a photo, JPG offers better compression. This “problem” is a basic difference between PNG and JPG, and is not related to ePub, iPad, or ADE at all.

3.The graphic dimension requirement is different for as images which occupy 25% or 40% of the page need more scaling as compared to Sony/ADE. This applies to cover dimension too.

One point of ePub is that it is supposed to allow us to create ebooks that look (at least) ok regardless of screen dimensions and size. Of course, that’s harder to do when you start dealing with images that are essentially fixed size. But ultimately, that’s the nature of the beast. Sometimes things might not scale to fit perfectly. But you can still create a ePub ebook that will fit the whole cover to the smallest dimension of the screen. As an ebook designer, you should design the book to work on any screen dimension, even if screens oriented horizontally instead of vertically. You’re doing it wrong if you’re designing for a particular screen size in mind. This is the same for web design.

3. Special Character support is extensive for iPad-ePUB as compared to ADE and SonyReader. This means that the iPad can render more characters as text as opposed to image.

Lack of character support is NOT a problem of ePub, but a problem of software and devices that do not have fonts to display the full range of Unicode characters (ie Sony Reader). ADE on windows should display just about any unicode character just fine. Even Sony Reader can if the font is embedded (I wish sony would remedy this, however).

5. iPad-ePUB does not work on XSL-stylesheet as ADE and SonyReader does. iPad-ePUB entirely works on CSS.

XSL-stylesheets are not part of the ePub standard. They are an optional hack that Adobe added support for in ADE. Simply don’t use XSL stylesheets.

“Although iPad-ePUB can be viewed on ADE but will not display colour enhancements in SonyReader (because of its grey screen). We can’t assure 100% cross-compatibility of ePUB between iPad and ADE/Sony, as the owner of respective specifications also don’t claim this cross-compatibility.”

Big surprise–color devices and black-and-white devices are different! I have a revolutionary idea for you, however…. Test your ePub ebook on a few common devices before releasing it. It’s easy enough to load it up on a few devices and make a quick glance for QA. This is, again, no different from web development. All web developers test their designs on the most common browsers. Furthermore, if you don’t add a lot of unnecessary style to your ebook, there is less to break on the various reading systems.

For the most part, ePub design and creation is the same as website design and creation, though more text centric. This shouldn’t come as a surprise, since ePub is just XHTML and CSS–the same as a website. Most of the problems are the same too. The ePub standard is far from perfect, but the idea of iPad-ePub and ADE-ePub is a false construct in the minds of designers who don’t “get it”.

However, as Paul mentions, DRM is the biggest incompatibility between various “types” of ePub.

I completely agree with Ben. Regarding encryption and DRM, IDPF could introduce a standard scheme. They provide an example of how a PKI-scheme could work, but instead they should make it a formal part of the standard. Bookstores can then generate a pair of public/private keys when a user creates an account and let an e-reader download the private key. A license system for the lending business could also perfectly be standardised. All this is technically so trivial that one can only conclude that the major players just want to make money out of the situation for as long as possible. They know very well people won’t buy all of their books in just one shop, nor will people buy several e-readers to work around that. The irony is that those major players would make much more money in an open environment, because the e-book market as a whole would grow more quickly. Until that happens paper books, which are delivered in a very standard way, will prevail.

Felix,

“Ideological purity might satisfy theoreticians and pundits but what consumers want is working solutions.

Deeds, not words”

That really says it. And I’m tired of seeing countless gadget-news articles that say of any vendor’s DRM’d ePub files, “And their files are ‘standard ePub’!” and then crowds of comment-area voices chanting the two-word mantra that few seem to understand. Faith based.

Still where does that leave us. I remember a discussion I had here a couple of weeks ago about ADE.

To make a long story short, it is perfectly possible to make an ePUB that Adobes own epubcheck flags as VALID, and have that ePUB display wonderfully on anything… except ADE.

I am actually stuck in a position where the ePUB’s I generate are valid (according to Adobe), and are beautiful on anything I have to test on… except Adobe (haven’t tested it on Apple).

So who is to blame here. Am I doing something wrong (no according to epubcheck, yes according to ADE). Or is Adobe trying to skew the standard in its own favor (by ignoring key bits of the standard without telling anybody).

My solution is of course known. Ignore Adobe, stick to the standard. But that makes me look like an idiot in the eyes of people who view the (ePUB)world through ADE.

Having tested this quite intensely I can safely say… ADE truly sucks. Take any book off FeedBooks and open it in Calibre’s ebook viewer, ADE and FBReader side by side. Calibre is close to 100% here, FBReader is around 80-90% but ADE just happily ignores large chunks.

Given the weight of Adobe you have no other choice but to test your book designs also on ADE and perhaps use a tool that handles those quirks as much as possible. In the end, if you can create a style sheet that works well on all readers once, then you can benefit from it many times. It is like website design as Ben mentioned.

Except… When the epubcheck says “VALID” and ADE still chooses to render it “wrong”, you are a little out of luck.

Sure I could ask Adobe for help (and they would probably love to sell me some eyeballs), but thats not really the point. If I did that, then Adobe “won” (and are smiling all the way to the bank).

If everybody did that, we would have a new browser-war (hey, its here already). But I guess Im a bit more idealistic than that. If Adobe cant be bothered to follow the standard (and it IS a standard), I cant be bothered to cater for their customers. (and no, I don’t check my webpages in Explorer, I follow the standard, let the browsers choke on it if they cant understand it).

In my book the only way forward is to follow the standard and point fingers when people ask why it looks like crap on their reader.

Jotunbane: epubcheck is not from Adobe. It’s an open source program written by Peter Sorotokin, Garth Conboy,
Markus Gylling, Piotr Kula, and Paul Norton.

Look at the last line on this page : https://code.google.com/p/epubcheck/

“EpubCheck development was largely done at Adobe Systems.”

Im not pointing any fingers here (ok I am).

Lets not split hairs on that one, epubcheck is a great tool and I use it all the time. I am simply trying to point out that a perfectly valid ePUB file (according to the standard) can be looking like a million dollars on one reader and look like something the cat dragged in on others.

But that is not the fault of the standard.

Epubcheck is NOT the end-all-be-all of epub quality assurance. In fact, it’s largely useless and stupid. In some ways, it’s worse than useless because it leads people into a false sense of validity, thinking that passing epubcheck is the same as making a good epub. It’s ok to test for some very basic errors, but it is by no means a guarantee of anything. There is no automagic way to check your epubs. The only way to check is to actually load them up on actual devices.

We should also remember that Adobe, Apple and other stakeholders are members of this committee and that the IDPF has already embarked on defining the next iteration of the standard whose objectives include greater interactivity and richer media. Concurrently, HTML is transitioning from version 4 to 5 and that has many implications for ePub. Finally, there are new device coming along and older ones dying out. It’s a dynamic (mad, mad) world out there.