XDM Serialization Use Case 8

Creating and reading expanding files, such as logging data.

Some process require storing data incrementally over time. For example, a web server stores access logs for each HTTP request. Long running processes may run indefinitely so at any time (between writing each item) the file should be syntactically valid so that it can be read without waiting for the process to terminate. For simplicity and efficiency, these files should be able to have new items written without rewriting any data. This implies there should be no "end markers" which need to be removed in order to append new data.

Files generated by long lived processes may become indefinitely large, so typically a file rotation scheme is used. The side effect is that the producer is not in control of when a file is moved and needs to be able to append to empty files. A file or stream should be able to be appended with no knowledge of it state; appending to an existing file or stream should be the same as writing the an empty file or stream.

Parsing these files requires that the format is valid and consistent regardless of the number of items present and without knowledge of which item in the original sequence is the first item in the file. The first item (or file) may not be available at all .

These uses imply the following constraints on a serialization


An XDM Producer
A process (producer) which produces XDM data incrementally. The process may be long lived and the data output at any time.

An XDM Consumer
A process (consumer) which reads the data output by the producer. The consumer may wish to read the data at any time and should be able to read every complete item in the sequence. The consumer may not know the state of the data, that is if it is the original start of the data or not.

A file manager
Data stored in files (or in memory) from a long lived process may grow indefinately without management.
A file manager may be used to provide file (or memory) manipulations to manage the size and number of files.
These operations may include

Nice to Have

The consumer may wish to skip over items without inuring the penalty of fully parsing them.
A file manager needs to be able to identify items so that files may be split and concatenated. Ideally this should be possible without having to parse each item. For example, sequences of large document nodes should be able to be split into individual files without having to do a full XML parsing of each document.

Expected Results
Assume that write operations are atomic at the level of items (by either file locking, buffering, or other mechanisms).

In the presence of any supported file operations (via the actor "A file manager" and the actor "XDM Producer")


There are no comments on this page. [Add comment]

Valid XHTML 1.0 Transitional :: Valid CSS :: Powered by WikkaWiki