Generalized Container Format — multiple e-book formats in one package

This is a rather disappointing idea for me, from a practical standpoint. In organizing and opening my files, I like to be able to tell from the extension what software and/or reader will work with the file. Unless it has each file in all possible formats, which would be a huge waste of space and bandwidth, I cannot tell if a file will work with a specific reader without trying it. Yes, when I buy the file, I am sure I will be told of the contents, but I don’t want a huge library of bloated files, each with an external text file acting as a packing slip.

For me, it is a great idea, and we are already working on something like that; we will certainly considering any attempt to find a standard.

Multiple formats are a sad reality, and typical document workflows require several formats for each file (like .doc, .indd, .pdf, .lrf), so I already have to store them all on my harddisk. It would be great if converters (especially XSLT and regular expressions) can be part of the package too. This way, if the reader application does not know any of the native formats, it can attempt to construct its own.

Dear Jon Noring,

as usual, I don’t understand what and why you are doing.
Is this container a glorified .zip file?
And why do we love XHTML so much? e.g. all the rendering in dotReader is done by an embedded web browser. If XHTML is the way to go then we don’t need all this development, standardization etc.

I know I’m being a bit provocative but I seriously think that most people do not understand your efforts and that is the reason why OpenReader “standard” has never seen any implementation.
Please provide some examples, tutorials and human digestible explanations.

Regards

Tamas Simon

Jack, thanks for your comment.

Two points in reply:

• The GCF is designed for anyone to use and subset for their own purposes. For example, someone may want to use it to distribute their particular type of content (which could be software!)

• Identifying a file as conforming to GCF is easy to program.

  1. Look for a ZIP file (inspect the first few bytes of the file).

  2. If apparently ZIP, then look to see if it contains the file META-INF/container.xml.

  3. If META-INF/container.xml is present, then unpack that file and inspect the default namespace of that document. If the default namespace value is that defined in the IDPF/OCF spec (which GCF has generalized), urn:oasis:names:tc:opendocument:xmlns:container, then one has high confidence the container is a GCF Container.

  4. If a GCF Container, then further dig into the META-INF/container.xml document looking at the individual <rootfile> elements and associated attribute values to see if any reference items of interest to the application.

We have not yet thought of an acceptable mechanism by which an application can know, without having to locate and dig into META-INF/container.xml that a file is a GCF-conforming container. Unfortunately, our fundamental requirement that all conforming IDFP/OCF 1.0 containers also conform to GCF makes this difficult. We are open to suggestions, but are pessimistic there’s a workable solution given the fundamental requirement.

We do want to work with IDPF and the Open Document people at OASIS when they start work on a next-generation container, so for GCF 2.0 (or whatever it will be called), this problem can be resolved in a truly generalized way.

Joscha, I’d be happy to talk with you more about your requirements for a container. I believe it possible you can create a container standard for your specific need which conforms with GCF and meets your needs. I say this because we intentionally made GCF quite general.

Tamas, I suppose one could view GCF, as well as the IDPF Container, and JAR files, etc., as glorified ZIP files. We prefer to think of it as using ZIP along with other standards and various constraints to meet a set of requirements for a generalized container to formally distribute multiple content resources.

A generalized container for the purpose of distributing content adds a set of requirements that simply “zipping everything up” does not solve. Thus the requirement, for example, for including an XML document called META-INF/container.xml which contains certain information for organizing the content and for optionally including identifiers (important for formal content!) The ZIP format in and of itself provides none of this important framework.

Now, about how GCF fits in some grander e-book scheme. Wow, that’s a tall order, because the answer is fairly complex. Nevertheless, GCF has applicability beyond just e-books. As mentioned in the article, there’s a group looking at GCF for its needs, and they are not distributing textual content (and it is not web-content, either.)

Regarding examples, I think Section 3.4 of the draft GCF spec is an excellent example of using GCF for web-content (Lee, maybe we should put together a real example.)

Simply put, express an e-book as web-content, like one would put it online for viewing by web browsers, with one of the (X)HTML documents being the “home page”. Then reference that home page document in META-INF/container.xml.

Finally, if you reread my article above, you notice a paragraph where I state that OpenReader and OEBPS will, in my opinion, provide a much better e-book reading experience than just representing an e-book by web-content. So in a sense I did address this. Nevertheless, there are applications now, in this interim period as we move towards an OEBPS/OpenReader future, where representing digital publications as web-content (preferably XHTML) has a role.

If nothing else, there’s already a huge amount of web-content out there, and the GCF provides a convenient and powerful means, better than others already developed, to portably contain and distribute web-content. David Rothman’s TeleRead blog, for example, can be output as a complete set of web-content and put into a GCF container for offline reading.

So is all of this just another specialized zip file? I realize this is not specifically meant for ebooks, as you say you can send just about anything using it, but if that is what I want, why reinvent the wheel? Why not stick to zip files? It wouldn’t hurt to give a special extension to zip files loaded with ebboks, like CBZ files, really, which are used for comic book images. CBZ files actually help organize things, however. Especially as they only one copy of the book — or perhaps, one regular copy and a bunch of thumbnails.

I still say it is a huge waste of bandwidth to send three to six copies of the same text each time someone wants a single copy. If you expect the user to retain the entire container, only speaking for myself, I will simply extract the copy I need and discard the container, so as not to continue wasting disk space. All the while assuming you allow such extraction, which is probably a poor assumption these days. If not, I will not bother with the container in the first place.

Jack, appreciate your feedback.

Yes, a GCF is a conforming ZIP file. But so is JAR, so is the IDPF/OCF container, so is the Open Document Format container, etc.

But GCF, following the pioneering lead of IDPF/OCF, provides an intelligent framework stitting on top of ZIP (as one way to think of it) that just “zipping up the files in ZIP” (without any framework) does not provide.

Regarding filename suffix, GCF has intentionally not recommended anything since we leave that up to those who want to use GCF for specific purposes to decide. We believe that filename suffixes are brittle anyway, while META-INF/container.xml provides the much better mechanism to define what exactly is inside.

Regarding what an end-user does with the contents of the Container, that is up to them. But remember the GCF provides a standardized framework which benefits both the distributors and the end-users of content, especially if the identifier and/or digital signatures features are enabled (not provided in “classic” ZIP), which is expected for distribution of more “formal” content.

Why GCF is not the way to do it

Alice sells ebooks on her website.
Bob buys an ebook from Alice and receives it in a GCF container.
Bob’s ebook reader extracts the best format it supports from the GCF file.
Bob can read the ebook, he is happy. He even decides to save the entire GCF with all the different formats in it… just in case.
A few month later Bob buys a new ebook device. This device is better than last year’s and supports a new file format. (or maybe it just needs a PDF optimized for its larger display size)
Bob wants to read the ebook HE ALREADY PURCHASED but he cannot find the new format inside the GCF.
Alice upgrades her webshop to support the new format, but this doesn’t help Bob.
Bob is angry at Alice.

ps. I think you guessed the solution 🙂

Let me note that the inclusion of identifiers, both container-level, and digital rendition level, is one of the two major innovations in GCF. This allows content to actually be identified for pointing purposes (such as for linking and referencing/citation.) It also permits not breaking existing pointers to digital renditions when content is shuffled around into different containers.

For example, a particular digital rendition, say a PDF document, is contained inside a GCF container, and is assigned a digital rendition identifier. At a later time that same PDF can be put into a different GCF, mixed with a whole different set of other content, and redistributed. Pointers which were created to point to the PDF in the first container will still find that PDF in the second container.

<laugh/>

Tamas, I agree with you that GCF won’t solve all the problems of mankind, nor of the e-book industry. And GCF is NOT the solution to the Tower of eBabel. A container is simply a container, and not an e-book format in and of itself.

Some points:

First, GCF is the formalism that we now can use to build the OpenReader Container Format, which is needed. (The spec for that will probably be a few lines long since GCF, which itself builds upon IDPF/OCF, has covered nearly all the gory details.) If GCF won’t be used for anything else, at least we have that.

Second, GCF allows one to package a web-site with identifiers and alternate digital renditions of that web site. This is a particular use that goes beyond e-bookdom and one which we think is a sleeper (yes, there are already “container formats” for web sites, but we believe GCF is the best of them all, a “next generation” for a variety of reasons.)

Third, GCF can be used for all kinds of content distribution purposes outside of e-bookdom (a generalization of point #2).

Fourth, a publisher may use GCF to package multiple renditions as they exist at the time for a publication. Certainly as new formats come out, a person who received the first container will be missing the book rendered in those new formats. This is not a fatal problem with GCF, since if the person bought the original e-book in a single format, the problem still exists. (We are not touting GCF as a final solution to this anyway. We think it has a role to play in the digital publication ecology.)

(As an aside, imagine a person buying an e-book and given the choice of formats they want, so GCF provides the container to send the book in the formats they check off.)

Fifth, there is an interesting subtlety not yet discussed, and this deals with DRM. The GCF now provides a mechanism by which a single DRM system (actually TPM as Lee would correctly say) can be used for a bunch of formats inside of it. Imagine it like an umbrella. Now imagine the formats that could be inside, all of which are NOT using their own built-in DRM system, but instead using the “umbrella” DRM: LIT, PDF, Mobipocket, etc. Now we can imagine a distributor/retailer using a more consumer-friendly DRM/TPM, hopefully one which is “open standard,” for applying to all these formats in umbrella fashion.

This subtlety is now not so subtle. <smile/> Obviously, we’ve not talked much about this because we tend to have our doubts about the long-term viability of using TPM for ordinary trade books (but realize many publishers want it and so GCF permits it), and because the ramifications are immense in the e-book industry. We prefer to stay a little bit outside of this controversy, although now that I’ve mentioned it, I am reminded of the famous quote from Monty Python’s Flying Circus: “hint, hint, wink, wink, nod, nod, say no more, say no more…”

Update of 22-Feb-2006: An example GCF container has been created. It contains three digital renditions (plain text, HTML and Plucker) of Mark Twain’s The Tragedy of Pudd’nhead Wilson from Project Gutenberg. The files are extractable using any ZIP application. The file META-INF/container.xml expresses the core GCF information about the container contents.

Thanks for the update Jon.
I plan on writing another article about social DRM and the broader topic of what publishers and ebookstores can do instead of using DRM.
I think the GCF container and the identifiers would provide a good foundation for having a digital proof of purchase.Technically speaking, this would be a digital certificate that would testify that the customer has purchased the digital product. Unlike DRM, digital certificates do work, are not breakable and digital signatures are in most countries legally binding.

Tamas, thanks for your feedback.

Digital signatures/certificates are definitely of interest, and the IDPF/OCF specification, which GCF “generalizes”, provides the basic mechanism for enabling these.

As I noted today in a message to The eBook Community, smaller and independent ebook publishers who distribute their ebooks without DRM/TPM, now have a mechanism to distribute multiple renditions in one package, and to take advantage of digital signatures. Lee is more the expert on this, and I’ll let him know about this discussion.

Hi

I read the related section of the IDPF/OCF spec.
I can see that sometimes a publisher may want to digitally sign a document to prevent it from being altered.
e.g. a market research company which sells a study online wouldn’t want the data to be altered and associate their name with messed up “facts”

What I meant it different.
I was thinking of a proof of purchase document (maybe XML, maybe human readable text or both) that would be added to the container.

To stick with your example this document would says something like
Tamas Simon purchased Mark Twain: The Tragedy of Pudd’nhead Wilson for 1$ on 23-Feb-2007. This document would then be digitally signed by the publisher/bookstore.
So if someone searches my harddrive and starts asking questions how I got a copy of Mark Twain’s book, I could show the proof of purchase (since it’s signed I could not have created it).

The GCF would be a great tool to contain this kind of information.

Cool! I see some real possibilities here to benefit the smaller and independent e-book publisher.