pkg-utils/doc/fileformat.md

Package File Format

Record Structure

A package file consists of a series of records with possibly compressed payload. Each record has a header, indicating the type of record, raw and compressed size of the payload data and the compression algorithm used.

All multi byte integers are stored in little endian byte order.

Knowledge of the payload size lets a decoder skip unknown record types inserted by an encoder that supports a newer version of the format, allowing for some degree of backwards compatibility.

The diagram below illustrates what a record header looks like. The horizontally arranged numbers indicate a byte offset inside a 32 bit word and the vertical numbers count 32 bit words from the start of the header.

      0       1       2       3
  +-------+-------+-------+-------+
0 |             magic             |
  +-------+-------+-------+-------+
1 | comp  |reserved for future use|
  +-------+-------+-------+-------+
2 |                               |
  |        compressed size        |
3 |                               |
  +-------+-------+-------+-------+
4 |                               |
  |       uncompressed size       |
5 |                               |
  +-------+-------+-------+-------+
6 |                               |
. |            payload            |
. |                               |
  |/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/

   /\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/|
  |                               |
  |_______________________________|

The field magic holds a 32 bit magic number indicating the chunk type.

Currently, the following chunk types are supported:

• PKG_MAGIC_HEADER with the value 0x21676B70 (ASCII "pkg!"). The overall package header with information about the package.
• PKG_MAGIC_TOC with the value 0x21636F74 (ASCII "toc!"). The table of contents record.
• PKG_MAGIC_DATA with the value 0x21746164 (ASCII "dat!"). The package data record.

The byte labeled comp holds a compression algorithm identifier. Currently, the following compression algorithms are supported:

• PKG_COMPRESSION_NONE with the value 0. The payload is uncompressed. The compressed size of the record payload must equal the uncompressed size.
• PKG_COMPRESSION_ZLIB with the value 1. The record payload area contains a raw zlib stream.
• PKG_COMPRESSION_LZMA with the value 2. The record payload area contains lzma compressed data.

The compressor ID is padded with 3 bytes that must be set to zero by an encoder and are currently reserved for future use.

Package Header Record

The header record must be present in ever package and must always be the first record in a package file. If a decoder encounters a file which does not start with the magic value of the header record, it must reject the file.

Future versions of the format that break backwards compatibility can simply introduce a new magic value for the (possibly altered) header record. Older decoders are expected to reject a file with the newer format, while newer decoders can implement different behavior depending on what magic value they find at the start.

The payload of header record is currently only used to store package dependencies. It is byte aligned and contains a 16 bit integer, indicating the number of dependencies, followed by a sequence of dependent packages.

Each dependent package is encoded as follows:

     0         1         2         length - 1
+---------+---------+-------     -+---------+
|  type   | length  | name   ...  |         |
+---------+---------+-------     -+---------+

The type field must be set to 0, indicating that a dependency is required by a package.

The length byte indicates the length of the package name that follows, allowing for up to 255 bytes of package name afterwards.

If all dependencies have been processed, but there is still payload data left in the header record, a decoder must ignore the extra data and skip to the end of the record.

Table of Contents Record

For each file, directory, symlink, et cetera contained in the package, the table of contents contains an entry with the following common structure:

      0       1       2       3
  +-------+-------+-------+-------+
0 |              mode             |
  +-------+-------+-------+-------+
1 |            user id            |
  +-------+-------+-------+-------+
2 |           group id            |
  +-------+-------+-------+-------+
3 |  path length  |
  +---------------+

The mode field contains standard UNIX permissions. The user ID and group ID fields contain the numeric IDs of the user and group respectively that own the file. The path length field indicates the length of the byte aligned, absolute file name that follows.

The file path is expected to neither start nor end with a slash, contain no sequences of more than one slash and not contain the components . or ...

On the bit level, the mode field is structured as follows:

1 1 1 1 1 1
5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0
  type |U|G|S|r|w|x|r|w|x|r|w|x|
    |   | | | | | | | | | | | |
    |   | | | | | | | | | | | +--- others execute permission flag
    |   | | | | | | | | | | +----- others write permission flag
    |   | | | | | | | | | +------- others read permission flag
    |   | | | | | | | | +--------- group execute permission flag
    |   | | | | | | | +----------- group write permission flag
    |   | | | | | | +------------- group read permission flag
    |   | | | | | +--------------- owner execute permission flag
    |   | | | | +----------------- owner write permission flag
    |   | | | +------------------- owner read permission flag
    |   | | +--------------------- sticky bit
    |   | +----------------------- set GID bit
    |   +------------------------- set UID bit
    +----------------------------- file type

The upper 16 bit of the mode filed must be set to zero.

Currently, the following file types are supported:

• S_IFCHR with a value of 2. The entry defines a character device.
• S_IFDIR with a value of 4. The entry defines a directory.
• S_IFBLK with a value of 6. The entry defines a block device.
• S_IFREG with a value of 8. The entry defines a regular file.
• S_IFLNK with a value of 10. The entry defines a symbolic link.

For character and block devices, the file path is followed by a byte aligned, 64 bit device number.

For regular files, the path is followed by a byte aligned, 64 bit integer indicating the total size of the file in bytes, followed by a byte aligned, 32 bit integer containing a unique file ID.

For symlinks, the path is followed by a byte aligned, 16 bit integer holding the length of the target path, followed by the byte aligned symlink target.

Data Record

A package may contain multiple data records holding raw file data. Each data record contains a sequence of a byte aligned, 32 bit file ID, followed by raw data until the file is filled (size indicated by table of contents is reached).

A file may not span across multiple data records. A file ID must not occur more than once in a data record and must only occur in a single data record.