DESCRIPTION

The built-in functions fmt_read(), fmt_write() and fmt_close() provide an interface to C code containing routines that implement specific file formats for reading and/or writing spec data arrays (and sometimes headers) to various file formats, that can include binary data. (The standard spec file format uses only ASCII data.) Overhead code in spec takes care of opening and closing the files. The spec distribution includes the source code for several formats. Sites can add local code to implement custom formats.

File formats currently included in the spec distribution are named cbf, esrf, immcat, pgm, raw and tiff. See FORMATS below for details.

FUNCTIONS

fmt_write(file, fmt, arr [, header [, flags]])

Writes the values from the data array arr to the given file using the specified format, fmt. header can be an associative array or a string. Possible values for flags are explained below.

fmt_read(file, fmt, arr [, header [, flags]])

Reads values from the given file using the specified format, fmt, and assigns them to the data array arr. header can be an associative array or a string. The possible value for flags is explained below.

fmt_close(file, fmt)

Allows for file clean up activity.

Some formats may only implement saving or reading. None of the current included format specifications have a fmt_close() functionality.

FORMATS

Recognition of the following format names is included in the standard spec distribution. The string comparison in the code is case insensitive.

raw

The file fmt_raw.c has code that simply reads or writes the data array passed as argument using native binary format. There is no header.

pgm

The file fmt_pgm.c implements "Portable PixMap" pgm format for grey-scale images and works with 8-bit or 16-bit integer arrays. The source code has the details.

cbf

The spec file fmt_cbf.c supports a subset of the crystallographic binary file (CBF) format standard. The code for basic reading and writing of the first image in such files is included. No library is needed. Only raw and byte-offset compression is supported. Only 16-bit and 32-bit integer arrays are supported. The code can read header comments in a format used by Dectris detectors. The code can also write a header using something similar to the Dectris format. See the source code for details.

esrf

The file fmt_esrf.c contains the implementation of the ESRF data format. Both read and write are supported along with header support. See the code for details.

immcat

The file fmt_immcat.c implements a format used at the former IMM-CAT beamline at the Advanced Photon Source and includes headers. See the source code for details.

tiff or tif

See fmt_tiff.c for implementation of reading and saving files using the TIFF image format. Code for saving TIFF files is included in the source. To read TIFF image files, spec must be linked with the TIFF library. See the source code for details on achieving that. A header can be saved to or read from the file using the TIFF IMAGEDESCRIPTION tag. Two formats, as seen with Dectris detectors, are recognized.

test

The file fmt_test.c demonstrates some aspects of creating a format implementation.

HEADERS

The esrf and immcat headers are well defined with the format viewable in the fmt_esrf.c and fmt_immcat.c files. The headers used with the CBF and TIFF formats are less well defined.

When writing CBF headers, given an associative array containing:

head["item1"] = 1234
head["item2"] = "some text"
...

spec will add the following text to a CBF file:

_array_data.header_convention "spec_1.0"
_array_data.header_contents
;
# item1: 12345
# item2: some text
# ...
;

That is, the associative array index (preferably, one word) will be written, followed by a colon, followed by the array value. Each line starts with a pound sign followed by a space. Standard CBF lines precede the values. The set of values begins and ends with a line containing just a semicolon.

When reading CBF headers, the code looks for lines that begin with a #, and tries to extract names and values to assign to the spec associative array given as an argument. In general the first word after the # is the array index and the remaining text is the array value. A colon (:) or equals sign (=) after the first word is discarded.

For TIFF headers, the header argument in fmt_write() can be a string or an associative array. If a string, its value is saved to the file with the IMAGEDESCRIPTION tag. If an associative array, the contents are converted to a string using two possible formats, and then that string is saved as the IMAGEDESCRIPTION.

Format 1 writes the array as a json-like string, as in:

{"item1": 12345, "item2": some text}

Note that values are not enclosed in quotes. For a compliant json string, use:

fmt_write("some_file.tif", "tif", arr, encode("json", head))

See the spec encode help file.

Format 2 creates a string along the same lines as the CBF header, using just the lines that begin with a #, as in:

# item1: 12345
# item2: some text

The default is format 1. If the header array contains an item with index "format", its value, either 1 or 2, selects the format. The "format" array item will not be written to the string.

With fmt_read() for tiff files, the code tries to detect whether the IMAGEDESCRIPTION string is format 1 or format 2, and parses the string accordingly, assigning values to the associative array header argument.

FLAGS

When saving data with fmt_write() possible values for flags are:

"append"
"overwrite"
"#number"

Note, more than one of these options can be passed in a space- or comma-separated list. The default when opening files to write is to append, but that default can be changed in the source file implementation. The cbf, esrf, pgm, raw and tiff are all set to overwrite. The default selection can be overridden with the flags value.

For fmt_read(), only the "#number*" flag is relevant. The #number option passes the given number as a separate number to the C code implementation function. The number can be a file position number, for example. The implementation can use that value for saving or retrieving a particular data set from the file. See the esrf implementation for an example.

ADDING A NEW FORMAT

To include a new_fmt.c file in the spec executable, the distribution file u_hook.c needs to be modified to have the new code linked into spec. The current distribution contains these lines near the end of u_hook.c:

/* Select which data file formats are available */
{
    void    esrf_fmt_init(void);
    void    raw_fmt_init(void);
    void    pgm_fmt_init(void);
    void    imm_fmt_init(void);
    void    test_fmt_init(void);
    void    tiff_fmt_init(void);
    void    cbf_fmt_init(void);

    esrf_fmt_init();
    raw_fmt_init();
    pgm_fmt_init();
    imm_fmt_init();
    test_fmt_init();
    tiff_fmt_init();
    cbf_fmt_init();
}

To add a new format, include a call of the associated initialization function. If the format implementation is submitted to CSS, it can be included in the standard source. Otherwise, u_hook.c will need to be updated locally with each updated spec distribution.

A minimal format implementation could contain the following items:

static void
read_file(void *fd, void *data, int rows, int cols,
                    int dtype, int dsize, int *err, int num) {
    /* implementation */
}
static void
save_file(void *fd, void *data, int rows, int cols,
                    int dtype, int dsize, int *err, int num) {
    /* implementation */
}
static
struct fmt_file_ops my_ops = {
        FF_USE_OPEN,
        "new",
        read_file,
        save_file,
};
void
new_fmt_init(void) {
        file_fmt_install(&my_ops);
}

Here the format is called "new" and a call of the function new_fmt_init() would need to be added to u_hook.c. If there is no implementation for reading or saving, leave out the associated function and initialize the fmt_file_ops structure element with NULL.

The choice of FF_USE_OPEN versus FF_USE_FOPEN determines which system or library functions need to be used for reading or writing the binary file. The first would use the system read() and write() functions, for example, while the second would use the standard C library fread(), fwrite() and related functions. The argument fd is a pointer to an integer for FF_USE_OPEN and a FILE pointer if using FF_USE_FOPEN. As examples from the formats included in the spec distribution, fmt_cbf.c uses FF_USE_OPEN and fmt_tiff.c uses FF_USE_FOPEN.

The data pointer points to an array of type given by the dtype argument with rows and columns given by the rows and cols arguments. The dsize argument is the number of bytes in the array. Possible values for dtype are the defined constants:

Constant	Key word	Description
ARR_STRING	"string"	String array
ARR_UCHAR	"ubyte"	Unsigned character (1 byte)
ARR_CHAR	"byte"	Character (1 byte)
ARR_USHORT	"ushort"	Unsigned short (16-bit integer)
ARR_SHORT	"short"	Signed short (16-bit integer)
ARR_ULONG	"ulong"	Unsigned long (32-bit integer)
ARR_LONG	"long"	Signed long (32-bit integer)
ARR_FLOAT	"float"	Single precision floating point (32 bits)
ARR_DOUBLE	"double"	Double precision floating point (64 bits)
ARR_ULONG64	"ulong64"	Unsigned long long (64-bit integer)
ARR_LONG64	"long64"	Signed long long (64-bit integer)

See the arrays help file for details on data types.

The err argument is a pointer to an integer that should be assigned one of these possible error codes when there is an error reading or writing the file:

FF_READ_ERROR	File read error
FF_WRITE_ERROR	File write error
FF_UNSUPPORTED_TYPE	Unsupported data type for format
FF_NO_DATA	No data in array
FF_BAD_HEADER	Bad file header
FF_DATA_MISMATCH	Array type mismatch
FF_SIZE_MISMATCH	Array size mismatch
FF_SHORT_READ	Read returned less data than requested
FF_SHORT_WRITE	Couldn't write all the data
FF_UNSUPPORTED_FUNCTION	Implementation doesn't support request

Any other value for err will generate an "unspecified error" error message. FF_READ_ERROR and FF_WRITE_ERROR should only be set if there is an error on the associated read() or write() system call, as the overhead code will print the error string associated with the current value of the errno C library global variable when those codes are returned.

The num argument is set to a value of 1 for each format when spec starts. spec keeps track of the number separately for fmt_read() and fmt_write() calls and increments the associated value after each such call. The value can also be set using the #num in the optional flags argument for each function, but the value will still be automatically incremented after each fmt_read() or fmt_write(). The number could be used as a file position number in the implementation.

C FUNCTIONS

int file_fmt_install(struct fmt_file_ops *p)

Must be called from the ###_fmt_init() function with a pointer to the structure that has the flag values and function pointers for the implementation.

const char *ff_get_specversion(void)

Returns a string containing the spec version number.

const char *ff_get_filename(void)

Returns a string containing the name of the data file passed to the fmt_read(), fmt_write() or fmt_close() function.

const char *ff_typedstring(int id)

Returns a one-word string describing the data type of the array as coded in the dtype argument of fmt_read() or fmt_write(), as in "short", "ushort", "long", etc.

int ff_get_byte_order(void)

Returns FF_BIG_ENDIAN or FF_LITTLE_ENDIAN as defined in fmtfile.h as appropriate for the native byte order on the host computer.

int ff_adjust_byte_order(int order, int dtype, void *data, int cnt)

Modifies, if necessary, the array in data, which is of type dtype and which has cnt elements, to make it either big or little endian, as indicated by the order argument, which must be FF_BIG_ENDIAN or FF_LITTLE_ENDIAN. If the native byte order already matches order, nothing is changed.

char *ff_get_named_head_item(char *ind)

Used in the saving function to retrieve an item from the header argument in the fmt_write() call from the spec user level. If header is an associative array, and if there is an item with the index ind, its value is returned as a string. If header has a string value, returns that string no matter the value of ind. Returns NULL if there is no matching item or if there was no header argument.

char *ff_get_next_head_item(char *cmd, char **id)

Similar to the previous function, but can be used to retrieve each item from the header array where the array index values may be unknown. If header is an associative array, if cmd is the string next, the next value in the array is returned. A pointer to a string containing the array index is assigned to id. If cmd is the string reset, the cycle is restarted. Returns NULL if there are no more items to return or if there was no header array. If cmd is a string that matches the index of an element of the header array, that value is returned and id is set to NULL. Note, an implicit "reset" occurs on each call of fmt_write().

If there is a header argument to fmt_write(), but it is not an associative array, returns the string value of the argument, for any value of cmd, including NULL, and id is set to NULL.

int ff_put_head_item(char *item, char *value)

Used in the reading function to place an item in the fmt_read() header argument, if it exists. If header is a string value and item is NULL, assigns value to header. If header is an associative array, assigns value to the element with index item. Returns zero for success and -1 if there is no header, value is NULL, header is an associative array and item is NULL, or item is NULL but header cannot be assigned a string value.