/
githubmirror
/
squashfs-tools
Обзор
Документация
Войти
/
githubmirror
/
squashfs-tools
Код
Запросы
0
Пакеты
0
Релизы
0
Аналитика
Безопасность
4.4
USAGE
1 216 строк
48 KB
Phillip Lougher
Squashfs-tools 4.4 release
29 авг 2019, 04:58
29 авг 2019, 04:58
52eb4c2
Код
Авторство
О чём код?
SQUASHFS 4.4 - A squashed read-only filesystem for Linux Copyright 2002-2019 Phillip Lougher <phillip@squashfs.org.uk> Released under the GPL licence (version 2 or later). Welcome to Squashfs version 4.4. Please read the README-4.3 and CHANGES files for details of changes. Squashfs is a highly compressed read-only filesystem for Linux. It uses either gzip/xz/lzo/lz4/zstd compression to compress both files, inodes and directories. Inodes in the system are very small and all blocks are packed to minimise data overhead. Block sizes greater than 4K are supported up to a maximum of 1Mbytes (default block size 128K). Squashfs is intended for general read-only filesystem use, for archival use (i.e. in cases where a .tar.gz file may be used), and in constrained block device/memory systems (e.g. embedded systems) where low overhead is needed. 1. SQUASHFS OVERVIEW -------------------- 1. Data, inodes and directories are compressed. 2. Squashfs stores full uid/gids (32 bits), and file creation time. 3. In theory files up to 2^64 bytes are supported. In theory filesystems can be up to 2^64 bytes. 4. Inode and directory data are highly compacted, and packed on byte boundaries. Each compressed inode is on average 8 bytes in length (the exact length varies on file type, i.e. regular file, directory, symbolic link, and block/char device inodes have different sizes). 5. Squashfs can use block sizes up to 1Mbyte (the default size is 128K). Using 128K blocks achieves greater compression ratios than the normal 4K block size. 6. File duplicates are detected and removed. 7. Filesystems can be compressed with gzip, xz (lzma2), lzo, lz4 or zstd compression algorithms. 1.1 Introducing reproducible builds ----------------------------------- Ever since Mksquashfs was parallelised back in 2006, there has been a certain randomness in how fragments and multi-block files are ordered in the output filesystem even if the input remains the same. This is because the multiple parallel threads can be scheduled differently between Mksquashfs runs. For example, the thread given fragment 10 to compress may finish before the thread given fragment 9 to compress on one run (writing fragment 10 to the output filesystem before fragment 9), but, on the next run it could be vice-versa. There are many different scheduling scenarios here, all of which can have a knock on effect causing different scheduling and ordering later in the filesystem too. Mkquashfs doesn't care about the ordering of fragments and multi-block files within the filesystem, as this does not affect the correctness of the filesystem. In fact not caring about the ordering, as it doesn't matter, allows Mksquashfs to run as fast as possible, maximising CPU and I/O performance. But, in the last couple of years, Squashfs has become used in scenarios (cloud etc) where this randomness is causing problems. Specifically this appears to be where downloaders, installers etc. try to work out the differences between Squashfs filesystem updates to minimise the amount of data that needs to transferred to update an image. Additionally, in the last couple of years has arisen the notion of reproducible builds, that is the same source and build environment etc should be able to (re-)generate identical output. This is usually for verification and security, allowing binaries/distributions to be checked for malicious activity. See https://reproducible-builds.org/ for more information. Mksquashfs from release 4.4 now generates reproducible images by default. Images generated by Mksquashfs will be ordered identically to previous runs if the same input has been supplied, and the same options used. 1.1.1 Dealing with timestamps Timestamps embedded in the filesystem will stiil cause differences. Each new run of Mksquashfs will produce a different mkfs (make filesystem) timestamp in the super-block. Moreover if any file timestamps have changed (even if the content hasn't), this will produce a difference. To prevent timestamps from producing differences, the following new Mksquashfs options have been added. 1.1.2 -mkfs-time <time> This option takes a positive time value (which is the number of seconds since the epoch of 1970-01-01 00:00:00 UTC), and sets the file system timestamp to that. Squashfs uses an unsigned 32-bit integer to store time, and the time given should be in that range. Obviously you can use the date command to convert dates into this value, i.e. % mksquashfs source source.sqsh -mkfs-time $(date +%s -d "Jan 1 2019 19:00") 1.1.3 -all-time <time> This option takes a positive time value (which is the number of seconds since the epoch of 1970-01-01 00:00:00 UTC), and sets the timestamp on all files to that (but not the mkfs time). 1.1.4 environment variable SOURCE_DATE_EPOCH As an alternative to the above command line options, you can set the environment variable SOURCE_DATE_EPOCH to a time value. This value will be used to set the mkfs time. Also any file timestamps which are after SOURCE_DATE_EPOCH will be clamped to SOURCE_DATE_EPOCH. See https://reproducible-builds.org/docs/source-date-epoch/ for more information. Note: both SOURCE_DATE_EPOCH and the command line options cannot be used at the same time. They are different ways to do the same thing, and both have FORCE sematics which mean they can't be over-ridden elsewhere (otherwise it would defeat the purpose). 1.1.5 -not-reproducible This option tells Mksquashfs that the files do not have to be strictly ordered. This will make Mksquashfs behave like version 4.3. 1.2 Extended attributes (xattrs) -------------------------------- Squashfs filesystems now have extended attribute support. The extended attribute implementation has the following features: 1. Layout can store up to 2^48 bytes of compressed xattr data. 2. Number of xattrs per inode unlimited. 3. Total size of xattr data per inode 2^48 bytes of compressed data. 4. Up to 4 Gbytes of data per xattr value. 5. Inline and out-of-line xattr values supported for higher performance in xattr scanning (listxattr & getxattr), and to allow xattr value de-duplication. 6. Both whole inode xattr duplicate detection and individual xattr value duplicate detection supported. These can obviously nest, file C's xattrs can be a complete duplicate of file B, and file B's xattrs can be a partial duplicate of file A. 7. Xattr name prefix types stored, allowing the redundant "user.", "trusted." etc. characters to be eliminated and more concisely stored. 8. Support for files, directories, symbolic links, device nodes, fifos and sockets. Extended attribute support is in 2.6.35 and later kernels. Filesystems with extended attributes can be mounted on 2.6.29 and later kernels, the extended attributes will be ignored with a warning. 2. USING SQUASHFS ----------------- Squashfs filesystems should be mounted with 'mount' with the filesystem type 'squashfs'. If the filesystem is on a block device, the filesystem can be mounted directly, e.g. %mount -t squashfs /dev/sda1 /mnt Will mount the squashfs filesystem on "/dev/sda1" under the directory "/mnt". If the squashfs filesystem has been written to a file, the loopback device can be used to mount it (loopback support must be in the kernel), e.g. %mount -t squashfs image /mnt -o loop Will mount the squashfs filesystem in the file "image" under the directory "/mnt". 3. MKSQUASHFS ------------- 3.1 Mksquashfs options and overview ----------------------------------- As squashfs is a read-only filesystem, the mksquashfs program must be used to create populated squashfs filesystems. SYNTAX:mksquashfs source1 source2 ... dest [options] [-e list of exclude dirs/files] Filesystem build options: -comp <comp> select <comp> compression Compressors available: gzip (default) lzo lz4 xz zstd -b <block_size> set data block to <block_size>. Default 128 Kbytes Optionally a suffix of K or M can be given to specify Kbytes or Mbytes respectively -reproducible build images that are reproducible (default) -not-reproducible build images that are not reproducible -mkfs-time <time> set mkfs time to <time> which is an unsigned int -fstime <time> synonym for mkfs-time -all-time <time> set all inode times to <time> which is an unsigned int -no-exports don't make the filesystem exportable via NFS -no-sparse don't detect sparse files -no-xattrs don't store extended attributes -xattrs store extended attributes (default) -noI do not compress inode table -noId do not compress the uid/gid table (implied by -noI) -noD do not compress data blocks -noF do not compress fragment blocks -noX do not compress extended attributes -no-fragments do not use fragments -always-use-fragments use fragment blocks for files larger than block size -no-duplicates do not perform duplicate checking -all-root make all files owned by root -root-mode <mode> set root directory permissions to octal <mode> -force-uid <uid> set all file uids to <uid> -force-gid <gid> set all file gids to <gid> -nopad do not pad filesystem to a multiple of 4K -keep-as-directory if one source directory is specified, create a root directory containing that directory, rather than the contents of the directory Filesystem filter options: -p <pseudo-definition> Add pseudo file definition -pf <pseudo-file> Add list of pseudo file definitions Pseudo definitions should be of the format filename d mode uid gid filename m mode uid gid filename b mode uid gid major minor filename c mode uid gid major minor filename f mode uid gid command filename s mode uid gid symlink -sort <sort_file> sort files according to priorities in <sort_file>. One file or dir with priority per line. Priority -32768 to 32767, default priority 0 -ef <exclude_file> list of exclude dirs/files. One per line -wildcards Allow extended shell wildcards (globbing) to be used in exclude dirs/files -regex Allow POSIX regular expressions to be used in exclude dirs/files Filesystem append options: -noappend do not append to existing filesystem -root-becomes <name> when appending source files/directories, make the original root become a subdirectory in the new root called <name>, rather than adding the new source items to the original root Mksquashfs runtime options: -version print version, licence and copyright message -exit-on-error treat normally ignored errors as fatal -recover <name> recover filesystem data using recovery file <name> -no-recovery don't generate a recovery file -quiet no verbose output -info print files written to filesystem -no-progress don't display the progress bar -progress display progress bar when using the -info option -processors <number> Use <number> processors. By default will use number of processors available -mem <size> Use <size> physical memory. Currently set to 4096M Optionally a suffix of K, M or G can be given to specify Kbytes, Mbytes or Gbytes respectively Miscellaneous options: -root-owned alternative name for -all-root -offset <offset> Skip <offset> bytes at the beginning of <dest>. Optionally a suffix of K, M or G can be given to specify Kbytes, Mbytes or Gbytes respectively. Default 0 bytes. -o <offset> synonym for -offset -noInodeCompression alternative name for -noI -noIdTableCompression alternative name for -noId -noDataCompression alternative name for -noD -noFragmentCompression alternative name for -noF -noXattrCompression alternative name for -noX -Xhelp print compressor options for selected compressor Compressors available and compressor specific options: gzip (default) -Xcompression-level <compression-level> <compression-level> should be 1 .. 9 (default 9) -Xwindow-size <window-size> <window-size> should be 8 .. 15 (default 15) -Xstrategy strategy1,strategy2,...,strategyN Compress using strategy1,strategy2,...,strategyN in turn and choose the best compression. Available strategies: default, filtered, huffman_only, run_length_encoded and fixed lzo -Xalgorithm <algorithm> Where <algorithm> is one of: lzo1x_1 lzo1x_1_11 lzo1x_1_12 lzo1x_1_15 lzo1x_999 (default) -Xcompression-level <compression-level> <compression-level> should be 1 .. 9 (default 8) Only applies to lzo1x_999 algorithm lz4 -Xhc Compress using LZ4 High Compression xz -Xbcj filter1,filter2,...,filterN Compress using filter1,filter2,...,filterN in turn (in addition to no filter), and choose the best compression. Available filters: x86, arm, armthumb, powerpc, sparc, ia64 -Xdict-size <dict-size> Use <dict-size> as the XZ dictionary size. The dictionary size can be specified as a percentage of the block size, or as an absolute value. The dictionary size must be less than or equal to the block size and 8192 bytes or larger. It must also be storable in the xz header as either 2^n or as 2^n+2^(n+1). Example dict-sizes are 75%, 50%, 37.5%, 25%, or 32K, 16K, 8K etc. zstd -Xcompression-level <compression-level> <compression-level> should be 1 .. 22 (default 15) Source1 source2 ... are the source directories/files containing the files/directories that will form the squashfs filesystem. If a single directory is specified (i.e. mksquashfs source output_fs) the squashfs filesystem will consist of that directory, with the top-level root directory corresponding to the source directory. If multiple source directories or files are specified, mksquashfs will merge the specified sources into a single filesystem, with the root directory containing each of the source files/directories. The name of each directory entry will be the basename of the source path. If more than one source entry maps to the same name, the conflicts are named xxx_1, xxx_2, etc. where xxx is the original name. To make this clear, take two example directories. Source directory "/home/phillip/test" contains "file1", "file2" and "dir1". Source directory "goodies" contains "goodies1", "goodies2" and "goodies3". usage example 1: %mksquashfs /home/phillip/test output_fs This will generate a squashfs filesystem with root entries "file1", "file2" and "dir1". example 2: %mksquashfs /home/phillip/test goodies output_fs This will create a squashfs filesystem with the root containing entries "test" and "goodies" corresponding to the source directories "/home/phillip/test" and "goodies". example 3: %mksquashfs /home/phillip/test goodies test output_fs This is the same as the previous example, except a third source directory "test" has been specified. This conflicts with the first directory named "test" and will be renamed "test_1". Multiple sources allow filesystems to be generated without needing to copy all source files into a common directory. This simplifies creating filesystems. The -keep-as-directory option can be used when only one source directory is specified, and you wish the root to contain that directory, rather than the contents of the directory. For example: example 4: %mksquashfs /home/phillip/test output_fs -keep-as-directory This is the same as example 1, except for -keep-as-directory. This will generate a root directory containing directory "test", rather than the "test" directory contents "file1", "file2" and "dir1". The Dest argument is the destination where the squashfs filesystem will be written. This can either be a conventional file or a block device. If the file doesn't exist it will be created, if it does exist and a squashfs filesystem exists on it, mksquashfs will append. The -noappend option will write a new filesystem irrespective of whether an existing filesystem is present. 3.2 Changing compression algorithm and compression specific options ------------------------------------------------------------------- By default Mksquashfs will compress using the gzip compression algorithm. This algorithm offers a good trade-off between compression ratio, and memory and time taken to decompress. Squashfs also supports LZ4, LZO and XZ (LZMA2) compression. LZO offers worse compression ratio than gzip, but is faster to decompress. XZ offers better compression ratio than gzip, but at the expense of greater memory and time to decompress (and significantly more time to compress). LZ4 is similar to LZO, but, support for it is not yet in the mainline kernel, and so its usefulness is currently limited to using Squashfs with Mksquashfs/Unsquashfs as an archival system like tar. If you're not building the squashfs-tools and kernel from source, then the tools and kernel may or may not have been built with support for LZ4, LZO or XZ compression. The compression algorithms supported by the build of Mksquashfs can be found by typing mksquashfs without any arguments. The compressors available are displayed at the end of the help message, e.g. Compressors available and compressor specific options: gzip (default) -Xcompression-level <compression-level> <compression-level> should be 1 .. 9 (default 9) -Xwindow-size <window-size> <window-size> should be 8 .. 15 (default 15) -Xstrategy strategy1,strategy2,...,strategyN Compress using strategy1,strategy2,...,strategyN in turn and choose the best compression. Available strategies: default, filtered, huffman_only, run_length_encoded and fixed lzo -Xalgorithm <algorithm> Where <algorithm> is one of: lzo1x_1 lzo1x_1_11 lzo1x_1_12 lzo1x_1_15 lzo1x_999 (default) -Xcompression-level <compression-level> <compression-level> should be 1 .. 9 (default 8) Only applies to lzo1x_999 algorithm lz4 -Xhc Compress using LZ4 High Compression xz -Xbcj filter1,filter2,...,filterN Compress using filter1,filter2,...,filterN in turn (in addition to no filter), and choose the best compression. Available filters: x86, arm, armthumb, powerpc, sparc, ia64 -Xdict-size <dict-size> Use <dict-size> as the XZ dictionary size. The dictionary size can be specified as a percentage of the block size, or as an absolute value. The dictionary size must be less than or equal to the block size and 8192 bytes or larger. It must also be storable in the xz header as either 2^n or as 2^n+2^(n+1). Example dict-sizes are 75%, 50%, 37.5%, 25%, or 32K, 16K, 8K etc. zstd -Xcompression-level <compression-level> <compression-level> should be 1 .. 22 (default 15) If the compressor offers compression specific options (all the compressors now have compression specific options except the deprecated lzma1 compressor) then these options are also displayed (.i.e. in the above XZ is shown with two compression specific options). The compression specific options are, obviously, specific to the compressor in question, and the compressor documentation and web sites should be consulted to understand their behaviour. In general the Mksquashfs compression defaults for each compressor are optimised to give the best performance for each compressor, where what constitutes best depends on the compressor. For gzip/xz best means highest compression, for LZO/LZ4 best means a tradeoff between compression and (de)-compression overhead (LZO/LZ4 by definition are intended for weaker processors). 3.3 Changing global compression defaults used in mksquashfs ----------------------------------------------------------- There are a large number of options that can be used to control the compression in mksquashfs. By and large the defaults are the most optimum settings and should only be changed in exceptional circumstances! Note, this does not apply to the block size, increasing the block size from the default of 128Kbytes will increase compression (especially for the xz compressor) and should increase I/O performance too. However, a block size of greater than 128Kbytes may increase latency in certain cases (where the filesystem contains lots of fragments, and no locality of reference is observed). For this reason the block size default is configured to the less optimal 128Kbytes. Users should experiment with 256Kbyte sizes or above. The -noI, -noD and -noF options (also -noInodeCompression, -noDataCompression and -noFragmentCompression) can be used to force mksquashfs to not compress inodes/directories, data and fragments respectively. Giving all options generates an uncompressed filesystem. The -no-fragments tells mksquashfs to not generate fragment blocks, and rather generate a filesystem similar to a Squashfs 1.x filesystem. It will of course still be a Squashfs 4.0 filesystem but without fragments, and so it won't be mountable on a Squashfs 1.x system. The -always-use-fragments option tells mksquashfs to always generate fragments for files irrespective of the file length. By default only small files less than the block size are packed into fragment blocks. The ends of files which do not fit fully into a block, are NOT by default packed into fragments. To illustrate this, a 100K file has an initial 64K block and a 36K remainder. This 36K remainder is not packed into a fragment by default. This is because to do so leads to a 10 - 20% drop in sequential I/O performance, as a disk head seek is needed to seek to the initial file data and another disk seek is need to seek to the fragment block. Specify this option if you want file remainders to be packed into fragment blocks. Doing so may increase the compression obtained BUT at the expense of I/O speed. The -no-duplicates option tells mksquashfs to not check the files being added to the filesystem for duplicates. This can result in quicker filesystem generation and appending although obviously compression will suffer badly if there is a lot of duplicate files. The -b option allows the block size to be selected, both "K" and "M" postfixes are supported, this can be either 4K, 8K, 16K, 32K, 64K, 128K, 256K, 512K or 1M bytes. 3.4 Specifying the UIDs/GIDs used in the filesystem --------------------------------------------------- By default files in the generated filesystem inherit the UID and GID ownership of the original file. However, mksquashfs provides a number of options which can be used to override the ownership. The options -all-root and -root-owned (both do exactly the same thing) force all file uids/gids in the generated Squashfs filesystem to be root. This allows root owned filesystems to be built without root access on the host machine. The "-force-uid uid" option forces all files in the generated Squashfs filesystem to be owned by the specified uid. The uid can be specified either by name (i.e. "root") or by number. The "-force-gid gid" option forces all files in the generated Squashfs filesystem to be group owned by the specified gid. The gid can be specified either by name (i.e. "root") or by number. 3.5 Excluding files from the filesystem --------------------------------------- The -e and -ef options allow files/directories to be specified which are excluded from the output filesystem. The -e option takes the exclude files/directories from the command line, the -ef option takes the exlude files/directories from the specified exclude file, one file/directory per line. Two styles of exclude file matching are supported: basic exclude matching, and extended wildcard matching. Basic exclude matching is a legacy feature retained for backwards compatibility with earlier versions of Mksquashfs. Extended wildcard matching should be used in preference. 3.5.1 Basic exclude matching ---------------------------- Each exclude file is treated as an exact match of a file/directory in the source directories. If an exclude file/directory is absolute (i.e. prefixed with /, ../, or ./) the entry is treated as absolute, however, if an exclude file/directory is relative, it is treated as being relative to each of the sources in turn, i.e. %mksquashfs /tmp/source1 source2 output_fs -e ex1 /tmp/source1/ex2 out/ex3 Will generate exclude files /tmp/source1/ex2, /tmp/source1/ex1, source2/ex1, /tmp/source1/out/ex3 and source2/out/ex3. 3.5.2 Extended exclude file handling ------------------------------------ Extended exclude file matching treats each exclude file as a wildcard or regex expression. To enable wildcard matching specify the -wildcards option, and to enable regex matching specify the -regex option. In most cases the -wildcards option should be used rather than -regex because wildcard matching behaviour is significantly easier to understand! In addition to wildcards/regex expressions, exclude files can be "anchored" or "non-anchored". An anchored exclude is one which matches from the root of the directory and nowhere else, a non-anchored exclude matches anywhere. For example given the directory hierarchy "a/b/c/a/b", the anchored exclude "a/b" will match "a/b" at the root of the directory hierarchy, but it will not match the "/a/b" sub-directory within directory "c", whereas a non-anchored exclude would. A couple of examples should make this clearer. Anchored excludes 1. mksquashfs example image.sqsh -wildcards -e 'test/*.gz' Exclude all files matching "*.gz" in the top level directory "test". 2. mksquashfs example image.sqsh -wildcards -e '*/[Tt]est/example*' Exclude all files beginning with "example" inside directories called "Test" or "test", that occur inside any top level directory. Using extended wildcards, negative matching is also possible. 3. mksquashfs example image.sqsh -wildcards -e 'test/!(*data*).gz' Exclude all files matching "*.gz" in top level directory "test", except those with "data" in the name. Non-anchored excludes By default excludes match from the top level directory, but it is often useful to exclude a file matching anywhere in the source directories. For this non-anchored excludes can be used, specified by pre-fixing the exclude with "...". Examples: 1. mksquashfs example image.sqsh -wildcards -e '... *.gz' Exclude files matching "*.gz" anywhere in the source directories. For example this will match "example.gz", "test/example.gz", and "test/test/example.gz". 2. mksquashfs example image.sqsh -wildcards -e '... [Tt]est/*.gz' Exclude files matching "*.gz" inside directories called "Test" or "test" that occur anywhere in the source directories. Again, using extended wildcards, negative matching is also possible. 3. mksquashfs example image.sqsh -wildcards -e '... !(*data*).gz' Exclude all files matching "*.gz" anywhere in the source directories, except those with "data" in the name. 3.5.3 Exclude files summary --------------------------- The -e and -ef exclude options are usefully used in archiving the entire filesystem, where it is wished to avoid archiving /proc, and the filesystem being generated, i.e. %mksquashfs / /tmp/root.sqsh -e proc /tmp/root.sqsh Multiple -ef options can be specified on the command line, and the -ef option can be used in conjuction with the -e option. 3.6 Appending to squashfs filesystems ------------------------------------- Running squashfs with the destination directory containing an existing filesystem will add the source items to the existing filesystem. By default, the source items are added to the existing root directory. To make this clear... An existing filesystem "image" contains root entries "old1", and "old2". Source directory "/home/phillip/test" contains "file1", "file2" and "dir1". example 1: %mksquashfs /home/phillip/test image Will create a new "image" with root entries "old1", "old2", "file1", "file2" and "dir1" example 2: %mksquashfs /home/phillip/test image -keep-as-directory Will create a new "image" with root entries "old1", "old2", and "test". As shown in the previous section, for single source directories '-keep-as-directory' adds the source directory rather than the contents of the directory. example 3: %mksquashfs /home/phillip/test image -keep-as-directory -root-becomes original-root Will create a new "image" with root entries "original-root", and "test". The '-root-becomes' option specifies that the original root becomes a subdirectory in the new root, with the specified name. The append option with file duplicate detection, means squashfs can be used as a simple versioning archiving filesystem. A squashfs filesystem can be created with for example the linux-2.4.19 source. Appending the linux-2.4.20 source will create a filesystem with the two source trees, but only the changed files will take extra room, the unchanged files will be detected as duplicates. 3.7 Appending recovery file feature ----------------------------------- Recovery files are created when appending to existing Squashfs filesystems. This allows the original filesystem to be recovered if Mksquashfs aborts unexpectedly (i.e. power failure). The recovery files are called squashfs_recovery_xxx_yyy, where "xxx" is the name of the filesystem being appended to, and "yyy" is a number to guarantee filename uniqueness (the PID of the parent Mksquashfs process). Normally if Mksquashfs exits correctly the recovery file is deleted to avoid cluttering the filesystem. If Mksquashfs aborts, the "-recover" option can be used to recover the filesystem, giving the previously created recovery file as a parameter, i.e. mksquashfs dummy image.sqsh -recover squashfs_recovery_image.sqsh_1234 The writing of the recovery file can be disabled by specifying the "-no-recovery" option. 3.8 Pseudo file support ----------------------- Mksquashfs supports pseudo files, these allow fake files, directories, character and block devices to be specified and added to the Squashfs filesystem being built, rather than requiring them to be present in the source directories. This, for example, allows device nodes to be added to the filesystem without requiring root access. Mksquashfs 4.1 added support for "dynamic pseudo files" and a modify operation. Dynamic pseudo files allow files to be dynamically created when Mksquashfs is run, their contents being the result of running a command or piece of shell script. The modifiy operation allows the mode/uid/gid of an existing file in the source filesystem to be modified. Mksquashfs 4.4 adds support for Symbolic links. Two Mksquashfs options are supported, -p allows one pseudo file to be specified on the command line, and -pf allows a pseudo file to be specified containing a list of pseduo definitions, one per line. 3.8.1. Creating a dynamic file ------------------------------ Pseudo definition Filename f mode uid gid command mode is the octal mode specifier, similar to that expected by chmod. uid and gid can be either specified as a decimal number, or by name. command can be an executable or a piece of shell script, and it is executed by running "/bin/sh -c command". The stdout becomes the contents of "Filename". Examples: Running a basic command ----------------------- /somedir/dmesg f 444 root root dmesg creates a file "/somedir/dmesg" containing the output from dmesg. Executing shell script ---------------------- RELEASE f 444 root root \ if [ ! -e /tmp/ver ]; then \ echo 0 > /tmp/ver; \ fi; \ ver=`cat /tmp/ver`; \ ver=$((ver +1)); \ echo $ver > /tmp/ver; \ echo -n `cat /tmp/release`; \ echo "-dev #"$ver `date` "Build host" `hostname` Creates a file RELEASE containing the release name, date, build host, and an incrementing version number. The incrementing version is a side-effect of executing the shell script, and ensures every time Mksquashfs is run a new version number is used without requiring any other shell scripting. The above example also shows that commands can be split across multiple lines using "\". Obviously as the script will be presented to the shell as a single line, a semicolon is need to separate individual shell commands within the shell script. Reading from a device (or fifo/named socket) -------------------------------------------- input f 444 root root dd if=/dev/sda1 bs=1024 count=10 Copies 10K from the device /dev/sda1 into the file input. Ordinarily Mksquashfs given a device, fifo, or named socket will place that special file within the Squashfs filesystem, the above allows input from these special files to be captured and placed in the Squashfs filesystem. 3.8.2. Creating a block or character device ------------------------------------------- Pseudo definition Filename type mode uid gid major minor Where type is either b - for block devices, and c - for character devices mode is the octal mode specifier, similar to that expected by chmod. uid and gid can be either specified as a decimal number, or by name. For example: /dev/chr_dev c 666 root root 100 1 /dev/blk_dev b 666 0 0 200 200 creates a character device "/dev/chr_dev" with major:minor 100:1 and a block device "/dev/blk_dev" with major:minor 200:200, both with root uid/gid and a mode of rw-rw-rw. 3.8.3. Creating a directory --------------------------- Pseudo definition Filename d mode uid gid mode is the octal mode specifier, similar to that expected by chmod. uid and gid can be either specified as a decimal number, or by name. For example: /pseudo_dir d 666 root root creates a directory "/pseudo_dir" with root uid/gid and mode of rw-rw-rw. 3.8.4. Creating a symbolic link ------------------------------- Pseudo definition Filename s mode uid gid symlink uid and gid can be either specified as a decimal number, or by name. Note mode is ignored, as symlinks always have "rwxrwxrwx" permissions. For example: symlink s 0 root root example creates a symlink "symlink" to file "example" with root uid/gid. 3.8.5. Modifying attributes of an existing file ----------------------------------------------- Pseudo definition Filename m mode uid gid mode is the octal mode specifier, similar to that expected by chmod. uid and gid can be either specified as a decimal number, or by name. For example: dmesg m 666 root root Changes the attributes of the file "dmesg" in the filesystem to have root uid/gid and a mode of rw-rw-rw, overriding the attributes obtained from the source filesystem. 3.9 Miscellaneous options ------------------------- The -info option displays the files/directories as they are compressed and added to the filesystem. The original uncompressed size of each file is printed, along with DUPLICATE if the file is a duplicate of a file in the filesystem. The -nopad option informs mksquashfs to not pad the filesystem to a 4K multiple. This is performed by default to enable the output filesystem file to be mounted by loopback, which requires files to be a 4K multiple. If the filesystem is being written to a block device, or is to be stored in a bootimage, the extra pad bytes are not needed. 4. UNSQUASHFS ------------- Unsquashfs allows you to decompress and extract a Squashfs filesystem without mounting it. It can extract the entire filesystem, or a specific file or directory. Unsquashfs can decompress all official Squashfs filesystem versions. The Unsquashfs usage info is: SYNTAX: unsquashfs [options] filesystem [directories or files to extract] -v[ersion] print version, licence and copyright information -d[est] <pathname> unsquash to <pathname>, default "squashfs-root" -q[uiet] no verbose output -n[o-progress] don't display the progress bar -no[-xattrs] don't extract xattrs in file system -x[attrs] extract xattrs in file system (default) -u[ser-xattrs] only extract user xattrs in file system. Enables extracting xattrs -p[rocessors] <number> use <number> processors. By default will use number of processors available -i[nfo] print files as they are unsquashed -li[nfo] print files as they are unsquashed with file attributes (like ls -l output) -l[s] list filesystem, but don't unsquash -ll[s] list filesystem with file attributes (like ls -l output), but don't unsquash -lln[umeric] -lls but with numeric uids and gids -lc list filesystem concisely, displaying only files and empty directories. Don't unsquash -llc list filesystem concisely with file attributes, displaying only files and empty directories. Don't unsquash -o[ffset] <bytes> skip <bytes> at start of <dest> Optionally a suffix of K, M or G can be given to specify Kbytes, Mbytes or Gbytes respectively. Default 0 bytes. -f[orce] if file already exists then overwrite -ig[nore-errors] Treat errors writing files to output as non-fatal -st[rict-errors] Treat all errors as fatal -s[tat] display filesystem superblock information -UTC Use UTC rather than local time zone when displaying time -mkfs-time display filesystem superblock time -fstime synonym for -mkfs-time -e[f] <extract file> list of directories or files to extract. One per line -da[ta-queue] <size> Set data queue to <size> Mbytes. Default 256 Mbytes -fr[ag-queue] <size> Set fragment queue to <size> Mbytes. Default 256 Mbytes -r[egex] treat extract names as POSIX regular expressions rather than use the default shell wildcard expansion (globbing) Decompressors available: gzip lzo lz4 xz zstd To extract a subset of the filesystem, the filenames or directory trees that are to be extracted can be specified on the command line. The files/directories should be specified using the full path to the files/directories as they appear within the Squashfs filesystem. The files/directories will also be extracted to those positions within the specified destination directory. The extract files can also be given in a file using the "-e[f]" option. Similarly to Mksquashfs, wildcard matching is performed on the extract files. Wildcard matching is enabled by default. Examples: 1. unsquashfs image.sqsh 'test/*.gz' Extract all files matching "*.gz" in the top level directory "test". 2. unsquashfs image.sqsh '[Tt]est/example*' Extract all files beginning with "example" inside top level directories called "Test" or "test". Using extended wildcards, negative matching is also possible. 3. unsquashfs image.sqsh 'test/!(*data*).gz' Extract all files matching "*.gz" in top level directory "test", except those with "data" in the name. 4.1 Unsquashfs options ---------------------- The "-ls" option can be used to list the contents of a filesystem without decompressing the filesystem data itself. The "-lls" option is similar but it also displays file attributes (ls -l style output). The "-lln" option is the same but displays uids and gids numerically. The "-lc" option is similar to the -ls option except it only displays files and empty directories. The -llc option displays file attributes. The "-info" option forces Unsquashfs to print each file as it is decompressed. The -"linfo" is similar but it also displays file attributes. The "-dest" option specifies the directory that is used to decompress the filesystem data. If this option is not given then the filesystem is decompressed to the directory "squashfs-root" in the current working directory. The "-force" option forces Unsquashfs to output to the destination directory even if files or directories already exist. This allows you to update an existing directory tree, or to Unsquashfs to a partially filled directory. Without the "-force" option, Unsquashfs will refuse to overwrite any existing files, or to create any directories if they already exist. This is done to protect data in case of mistakes, and so the "-force" option should be used with caution. The "-stat" option displays filesystem superblock information. This is useful to discover the filesystem version, byte ordering, whether it has a NFS export table, and what options were used to compress the filesystem, etc. The -mkfs-time option displays the make filesystem time contained in the super-block. This is displayed as the number of seconds since the epoch of 1970-01-01 00:00:00 UTC. The -UTC option makes Unsquashfs display all times in the UTC time zone rather than using the default local time zone. 4.2. Dealing with errors ------------------------ Unsquashfs splits errors into two categories: fatal errors and non-fatal errors. Fatal errors are those which cause Unsquashfs to abort instantly. These are generally due to failure to read the filesystem (corruption), and/or failure to write files to the output filesystem, due to I/O error or out of space. Generally anything which is unexpected is a fatal error. Non-fatal errors are generally where support is lacking in the output filesystem, and it can be considered to be an expected failure. This includes the inability to write extended attributes (xattrs) to a filesystem that doesn't support them, the inability to create files on filesystem that doesn't support them (i.e. symbolic links on VFAT), and the inability to execute privileged operations as a user-process. The user may well know the filesystem cannot support certain operations and would prefer Unsquashfs to ignore then without aborting. In the past Unsquashfs was much more tolerant of errors, in this release a significant number of errors that were non-fatal have been hardened to fatal. 4.2.1. -ignore-errors This makes Unsquashfs behave like previous versions, and treats more errors as non-fatal. 4.2.2 -strict-errors This makes Unsquashfs treat every error as fatal, and it will abort instantly. 5. FILESYSTEM LAYOUT -------------------- A squashfs filesystem consists of a maximum of nine parts, packed together on a byte alignment: --------------- | superblock | |---------------| | compression | | options | |---------------| | datablocks | | & fragments | |---------------| | inode table | |---------------| | directory | | table | |---------------| | fragment | | table | |---------------| | export | | table | |---------------| | uid/gid | | lookup table | |---------------| | xattr | | table | --------------- Compressed data blocks are written to the filesystem as files are read from the source directory, and checked for duplicates. Once all file data has been written the completed super-block, compression options, inode, directory, fragment, export, uid/gid lookup and xattr tables are written. 5.1 Compression options ----------------------- Compressors can optionally support compression specific options (e.g. dictionary size). If non-default compression options have been used, then these are stored here. 5.2 Inodes ---------- Metadata (inodes and directories) are compressed in 8Kbyte blocks. Each compressed block is prefixed by a two byte length, the top bit is set if the block is uncompressed. A block will be uncompressed if the -noI option is set, or if the compressed block was larger than the uncompressed block. Inodes are packed into the metadata blocks, and are not aligned to block boundaries, therefore inodes overlap compressed blocks. Inodes are identified by a 48-bit number which encodes the location of the compressed metadata block containing the inode, and the byte offset into that block where the inode is placed (<block, offset>). To maximise compression there are different inodes for each file type (regular file, directory, device, etc.), the inode contents and length varying with the type. To further maximise compression, two types of regular file inode and directory inode are defined: inodes optimised for frequently occurring regular files and directories, and extended types where extra information has to be stored. 5.3 Directories --------------- Like inodes, directories are packed into compressed metadata blocks, stored in a directory table. Directories are accessed using the start address of the metablock containing the directory and the offset into the decompressed block (<block, offset>). Directories are organised in a slightly complex way, and are not simply a list of file names. The organisation takes advantage of the fact that (in most cases) the inodes of the files will be in the same compressed metadata block, and therefore, can share the start block. Directories are therefore organised in a two level list, a directory header containing the shared start block value, and a sequence of directory entries, each of which share the shared start block. A new directory header is written once/if the inode start block changes. The directory header/directory entry list is repeated as many times as necessary. Directories are sorted, and can contain a directory index to speed up file lookup. Directory indexes store one entry per metablock, each entry storing the index/filename mapping to the first directory header in each metadata block. Directories are sorted in alphabetical order, and at lookup the index is scanned linearly looking for the first filename alphabetically larger than the filename being looked up. At this point the location of the metadata block the filename is in has been found. The general idea of the index is ensure only one metadata block needs to be decompressed to do a lookup irrespective of the length of the directory. This scheme has the advantage that it doesn't require extra memory overhead and doesn't require much extra storage on disk. 5.4 File data ------------- Regular files consist of a sequence of contiguous compressed blocks, and/or a compressed fragment block (tail-end packed block). The compressed size of each datablock is stored in a block list contained within the file inode. To speed up access to datablocks when reading 'large' files (256 Mbytes or larger), the code implements an index cache that caches the mapping from block index to datablock location on disk. The index cache allows Squashfs to handle large files (up to 1.75 TiB) while retaining a simple and space-efficient block list on disk. The cache is split into slots, caching up to eight 224 GiB files (128 KiB blocks). Larger files use multiple slots, with 1.75 TiB files using all 8 slots. The index cache is designed to be memory efficient, and by default uses 16 KiB. 5.5 Fragment lookup table ------------------------- Regular files can contain a fragment index which is mapped to a fragment location on disk and compressed size using a fragment lookup table. This fragment lookup table is itself stored compressed into metadata blocks. A second index table is used to locate these. This second index table for speed of access (and because it is small) is read at mount time and cached in memory. 5.6 Uid/gid lookup table ------------------------ For space efficiency regular files store uid and gid indexes, which are converted to 32-bit uids/gids using an id look up table. This table is stored compressed into metadata blocks. A second index table is used to locate these. This second index table for speed of access (and because it is small) is read at mount time and cached in memory. 5.7 Export table ---------------- To enable Squashfs filesystems to be exportable (via NFS etc.) filesystems can optionally (disabled with the -no-exports Mksquashfs option) contain an inode number to inode disk location lookup table. This is required to enable Squashfs to map inode numbers passed in filehandles to the inode location on disk, which is necessary when the export code reinstantiates expired/flushed inodes. This table is stored compressed into metadata blocks. A second index table is used to locate these. This second index table for speed of access (and because it is small) is read at mount time and cached in memory. 5.8 Xattr table --------------- The xattr table contains extended attributes for each inode. The xattrs for each inode are stored in a list, each list entry containing a type, name and value field. The type field encodes the xattr prefix ("user.", "trusted." etc) and it also encodes how the name/value fields should be interpreted. Currently the type indicates whether the value is stored inline (in which case the value field contains the xattr value), or if it is stored out of line (in which case the value field stores a reference to where the actual value is stored). This allows large values to be stored out of line improving scanning and lookup performance and it also allows values to be de-duplicated, the value being stored once, and all other occurences holding an out of line reference to that value. The xattr lists are packed into compressed 8K metadata blocks. To reduce overhead in inodes, rather than storing the on-disk location of the xattr list inside each inode, a 32-bit xattr id is stored. This xattr id is mapped into the location of the xattr list using a second xattr id lookup table. 6. AUTHOR INFO -------------- Squashfs was written by Phillip Lougher, email phillip@squashfs.org.uk, in Chepstow, Wales, UK. If you like the program, or have any problems, then please email me, as it's nice to get feedback!