squashfs4.3/0000755000175000017500000000000012334252013013017 5ustar phillipphillipsquashfs4.3/ACKNOWLEDGEMENTS0000644000175000017500000001402212334227665015312 0ustar phillipphillip ACKNOWLEDGEMENTS Thanks to everyone who have downloaded Squashfs. I appreciate people using it, and any feedback you have. The following have provided useful feedback, which has guided some of the extra features in squashfs. This is a randomly ordered (roughly in chronological order) list, which is updated when I remember... Acknowledgements for Squashfs 4.3 --------------------------------- Thanks to Bruno Wolff III and Andy Lutomirski for useful feedback during the long development process of Squashfs 4.3. Acknowledgements for Squashfs 4.2 --------------------------------- Thanks to Lasse Collin (http://tukaani.org/xz/) for mainlining XZ decompression support. Acknowledgements for Squashfs 4.1 --------------------------------- Thanks to Chan Jeong and LG for the patches to support LZO compression. Acknowledgements for Squashfs 4.0 --------------------------------- Thanks to Tim Bird and CELF (Consumer Electronics Linux Forum) for helping fund mainstreaming of Squashfs into the 2.6.29 kernel and the changes to the Squashfs tools to support the new 4.0 file system layout. Acknowledgements for Squashfs-3.3 ------------------------------------ Peter Korsgaard and others sent patches updating Squashfs to changes in the VFS interface for 2.6.22/2.6.23/2.6.24-rc1. Peter also sent some small patches for the Squashfs kernel code. Vito Di Leo sent a patch extending Mksquashfs to support regex filters. While his patched worked, it unfortunately made it easy to make Mksquashfs perform unpredictably with poorly choosen regex expressions. It, however, encouraged myself to add support for wildcard pattern matching and regex filters in a different way. Acknowledgements for Squashfs-3.2-r2 ------------------------------------ Junjiro Okajima discovered a couple of SMP issues, thanks. Junjiro Okajima and Tomas Matejicek have produced some good LZMA patches for Squashfs. Acknowledgements for Squashfs-3.2 --------------------------------- Peter Korsgaard sent a patch updating Squashfs to changes in the VFS interface in Linux 2.6.20. Acknowledgements for Squashfs-3.1 --------------------------------- Kenneth Duda and Ed Swierk of Arastra Inc. identified numerous bugs with Squashfs, and provided patches which were the basis for some of the fixes. In particular they identified the fragment rounding bug, the NFS bug, the initrd bug, and helped identify the 4K stack overflow bug. Scott James Remnant (Ubuntu) also identified the fragment rounding bug, and he also provided a patch. Ming Zhang identified the Lseek bug in Mksquashfs. His tests on the performance of Mksquashfs on SMP systems encouraged the rewrite of Mksquashfs. Peter Korsgaard, Daniel Olivera and Zilvinas Valinskas noticed Squashfs 3.0 didn't compile on Linux-2.6.18-rc[1-4] due to changes in the Linux VFS interfaces, and provided patches. Tomas Matejicek (SLAX) suggested the -force option on Unsquashfs, and noticed Unsquashfs didn't return the correct exit status. Yann Le Doare reported a kernel oops and provided a Qemu image that led to the identification of the simultaneously accessing multiply mounted Squashfs filesystems bug. Older acknowledgements ---------------------- Mark Robson - pointed out early on that initrds didn't work Adam Warner - pointed out that greater than 2GB filesystems didn't work. John Sutton - raised the problem when archiving the entire filesystem (/) there was no way to prevent /proc being archived. This prompted exclude files. Martin Mueller (LinuxTV) - noticed that the filesystem length in the superblock doesn't match the output filesystem length. This is due to padding to a 4K boundary. This prompted the addition of the -nopad option. He also reported a problem where 32K block filesystems hung when used as initrds. Arkadiusz Patyk (Polish Linux Distribution - PLD) reported a problem where 32K block filesystems hung when used as a root filesystem mounted as a loopback device. Joe Blow emailed me that I'd forgotten to put anything in the README about mounting the squashfs filesystem. David Fox (Lindows) noticed that the exit codes returned by Mksquashfs were wrong. He also noticed that a lot of time was spent in the duplicate scan routine. Cameron Rich complained that Squashfs did not support FIFOs or sockets. Steve Chadsey and Thomas Weissmuller noticed that files larger than the available memory could not be compressed by Mksquashfs. "Ptwahyu" and "Hoan" (I have no full names and I don't like giving people's email addresses), noticed that Mksquashfs 1.3 SEGV'd occasionally. Even though I had already noticed this bug, it is useful to be informed by other people. Don Elwell, Murray Jensen and Cameron Rich, have all sent in patches. Thanks, I have not had time to do anything about them yet... Drew Scott Daniels has been a good advocate for Squashfs. Erik Andersen has made some nice suggestions, unfortunately, I have not had time to implement anything. Artemiy I. Pavlov has written a useful LDP mini-howto for Squashfs (http://linuxdoc.artemio.net/squashfs). Yves Combe reported the Apple G5 bug, when using Squashfs for his PPC Knoppix-mib livecd project. Jaco Greeff (mklivecd project, and maintainer of the Mandrake squashfs-tools package) suggested the new mksquashfs -ef option, and the standalone build for mksquashfs. Mike Schaudies made a donation. Arkadiusz Patyk from the Polish Linux Distribution reported that Squashfs didn't work on amd64 machines. He gave me an account on a PLD amd64 machine which allowed myself to track down these bugs. Miles Roper, Peter Kjellerstedt and Willy Tarreau reported that release 2.1 did not compile with gcc < 3.x. Marcel J.E. Mol reported lack of kernel memory issues when using Squashfs on small memory embedded systems. This prompted the addition of the embedded system kernel configuration options. Era Scarecrow noticed that Mksquashfs had not been updated to reflect that smaller than 4K blocks are no longer supported. Kenichi Shima reported the Kconfig file had not been updated to 2.2. Aaron Ten Clay made a donation! Tomas Matejicek (SLAX) made a donation! squashfs4.3/README0000644000175000017500000012001412334226167013710 0ustar phillipphillip SQUASHFS 4.3 - A squashed read-only filesystem for Linux Copyright 2002-2014 Phillip Lougher Released under the GPL licence (version 2 or later). Welcome to Squashfs version 4.3. Please read the README-4.3 and CHANGES files for details of changes. Squashfs is a highly compressed read-only filesystem for Linux. It uses either gzip/xz/lzo/lz4 compression to compress both files, inodes and directories. Inodes in the system are very small and all blocks are packed to minimise data overhead. Block sizes greater than 4K are supported up to a maximum of 1Mbytes (default block size 128K). Squashfs is intended for general read-only filesystem use, for archival use (i.e. in cases where a .tar.gz file may be used), and in constrained block device/memory systems (e.g. embedded systems) where low overhead is needed. 1. SQUASHFS OVERVIEW -------------------- 1. Data, inodes and directories are compressed. 2. Squashfs stores full uid/gids (32 bits), and file creation time. 3. In theory files up to 2^64 bytes are supported. In theory filesystems can be up to 2^64 bytes. 4. Inode and directory data are highly compacted, and packed on byte boundaries. Each compressed inode is on average 8 bytes in length (the exact length varies on file type, i.e. regular file, directory, symbolic link, and block/char device inodes have different sizes). 5. Squashfs can use block sizes up to 1Mbyte (the default size is 128K). Using 128K blocks achieves greater compression ratios than the normal 4K block size. 6. File duplicates are detected and removed. 7. Filesystems can be compressed with gzip, xz (lzma2), lzo or lz4 compression algorithms. 1.1 Extended attributes (xattrs) -------------------------------- Squashfs filesystems now have extended attribute support. The extended attribute implementation has the following features: 1. Layout can store up to 2^48 bytes of compressed xattr data. 2. Number of xattrs per inode unlimited. 3. Total size of xattr data per inode 2^48 bytes of compressed data. 4. Up to 4 Gbytes of data per xattr value. 5. Inline and out-of-line xattr values supported for higher performance in xattr scanning (listxattr & getxattr), and to allow xattr value de-duplication. 6. Both whole inode xattr duplicate detection and individual xattr value duplicate detection supported. These can obviously nest, file C's xattrs can be a complete duplicate of file B, and file B's xattrs can be a partial duplicate of file A. 7. Xattr name prefix types stored, allowing the redundant "user.", "trusted." etc. characters to be eliminated and more concisely stored. 8. Support for files, directories, symbolic links, device nodes, fifos and sockets. Extended attribute support is in 2.6.35 and later kernels. Filesystems with extended attributes can be mounted on 2.6.29 and later kernels, the extended attributes will be ignored with a warning. 2. USING SQUASHFS ----------------- Squashfs filesystems should be mounted with 'mount' with the filesystem type 'squashfs'. If the filesystem is on a block device, the filesystem can be mounted directly, e.g. %mount -t squashfs /dev/sda1 /mnt Will mount the squashfs filesystem on "/dev/sda1" under the directory "/mnt". If the squashfs filesystem has been written to a file, the loopback device can be used to mount it (loopback support must be in the kernel), e.g. %mount -t squashfs image /mnt -o loop Will mount the squashfs filesystem in the file "image" under the directory "/mnt". 3. MKSQUASHFS ------------- 3.1 Mksquashfs options and overview ----------------------------------- As squashfs is a read-only filesystem, the mksquashfs program must be used to create populated squashfs filesystems. SYNTAX:./mksquashfs source1 source2 ... dest [options] [-e list of exclude dirs/files] Filesystem build options: -comp select compression Compressors available: gzip (default) lzo lz4 xz -b set data block to . Default 128 Kbytes Optionally a suffix of K or M can be given to specify Kbytes or Mbytes respectively -no-exports don't make the filesystem exportable via NFS -no-sparse don't detect sparse files -no-xattrs don't store extended attributes -xattrs store extended attributes (default) -noI do not compress inode table -noD do not compress data blocks -noF do not compress fragment blocks -noX do not compress extended attributes -no-fragments do not use fragments -always-use-fragments use fragment blocks for files larger than block size -no-duplicates do not perform duplicate checking -all-root make all files owned by root -force-uid uid set all file uids to uid -force-gid gid set all file gids to gid -nopad do not pad filesystem to a multiple of 4K -keep-as-directory if one source directory is specified, create a root directory containing that directory, rather than the contents of the directory Filesystem filter options: -p Add pseudo file definition -pf Add list of pseudo file definitions -sort sort files according to priorities in . One file or dir with priority per line. Priority -32768 to 32767, default priority 0 -ef list of exclude dirs/files. One per line -wildcards Allow extended shell wildcards (globbing) to be used in exclude dirs/files -regex Allow POSIX regular expressions to be used in exclude dirs/files Filesystem append options: -noappend do not append to existing filesystem -root-becomes when appending source files/directories, make the original root become a subdirectory in the new root called , rather than adding the new source items to the original root Mksquashfs runtime options: -version print version, licence and copyright message -exit-on-error treat normally ignored errors as fatal -recover recover filesystem data using recovery file -no-recovery don't generate a recovery file -info print files written to filesystem -no-progress don't display the progress bar -progress display progress bar when using the -info option -processors Use processors. By default will use number of processors available -mem Use physical memory. Currently set to 1922M Optionally a suffix of K, M or G can be given to specify Kbytes, Mbytes or Gbytes respectively Miscellaneous options: -root-owned alternative name for -all-root -noInodeCompression alternative name for -noI -noDataCompression alternative name for -noD -noFragmentCompression alternative name for -noF -noXattrCompression alternative name for -noX -Xhelp print compressor options for selected compressor Compressors available and compressor specific options: gzip (default) -Xcompression-level should be 1 .. 9 (default 9) -Xwindow-size should be 8 .. 15 (default 15) -Xstrategy strategy1,strategy2,...,strategyN Compress using strategy1,strategy2,...,strategyN in turn and choose the best compression. Available strategies: default, filtered, huffman_only, run_length_encoded and fixed lzo -Xalgorithm Where is one of: lzo1x_1 lzo1x_1_11 lzo1x_1_12 lzo1x_1_15 lzo1x_999 (default) -Xcompression-level should be 1 .. 9 (default 8) Only applies to lzo1x_999 algorithm lz4 -Xhc Compress using LZ4 High Compression xz -Xbcj filter1,filter2,...,filterN Compress using filter1,filter2,...,filterN in turn (in addition to no filter), and choose the best compression. Available filters: x86, arm, armthumb, powerpc, sparc, ia64 -Xdict-size Use as the XZ dictionary size. The dictionary size can be specified as a percentage of the block size, or as an absolute value. The dictionary size must be less than or equal to the block size and 8192 bytes or larger. It must also be storable in the xz header as either 2^n or as 2^n+2^(n+1). Example dict-sizes are 75%, 50%, 37.5%, 25%, or 32K, 16K, 8K etc. Source1 source2 ... are the source directories/files containing the files/directories that will form the squashfs filesystem. If a single directory is specified (i.e. mksquashfs source output_fs) the squashfs filesystem will consist of that directory, with the top-level root directory corresponding to the source directory. If multiple source directories or files are specified, mksquashfs will merge the specified sources into a single filesystem, with the root directory containing each of the source files/directories. The name of each directory entry will be the basename of the source path. If more than one source entry maps to the same name, the conflicts are named xxx_1, xxx_2, etc. where xxx is the original name. To make this clear, take two example directories. Source directory "/home/phillip/test" contains "file1", "file2" and "dir1". Source directory "goodies" contains "goodies1", "goodies2" and "goodies3". usage example 1: %mksquashfs /home/phillip/test output_fs This will generate a squashfs filesystem with root entries "file1", "file2" and "dir1". example 2: %mksquashfs /home/phillip/test goodies output_fs This will create a squashfs filesystem with the root containing entries "test" and "goodies" corresponding to the source directories "/home/phillip/test" and "goodies". example 3: %mksquashfs /home/phillip/test goodies test output_fs This is the same as the previous example, except a third source directory "test" has been specified. This conflicts with the first directory named "test" and will be renamed "test_1". Multiple sources allow filesystems to be generated without needing to copy all source files into a common directory. This simplifies creating filesystems. The -keep-as-directory option can be used when only one source directory is specified, and you wish the root to contain that directory, rather than the contents of the directory. For example: example 4: %mksquashfs /home/phillip/test output_fs -keep-as-directory This is the same as example 1, except for -keep-as-directory. This will generate a root directory containing directory "test", rather than the "test" directory contents "file1", "file2" and "dir1". The Dest argument is the destination where the squashfs filesystem will be written. This can either be a conventional file or a block device. If the file doesn't exist it will be created, if it does exist and a squashfs filesystem exists on it, mksquashfs will append. The -noappend option will write a new filesystem irrespective of whether an existing filesystem is present. 3.2 Changing compression algorithm and compression specific options ------------------------------------------------------------------- By default Mksquashfs will compress using the gzip compression algorithm. This algorithm offers a good trade-off between compression ratio, and memory and time taken to decompress. Squashfs also supports LZ4, LZO and XZ (LZMA2) compression. LZO offers worse compression ratio than gzip, but is faster to decompress. XZ offers better compression ratio than gzip, but at the expense of greater memory and time to decompress (and significantly more time to compress). LZ4 is similar to LZO, but, support for it is not yet in the mainline kernel, and so its usefulness is currently limited to using Squashfs with Mksquashfs/Unsquashfs as an archival system like tar. If you're not building the squashfs-tools and kernel from source, then the tools and kernel may or may not have been built with support for LZ4, LZO or XZ compression. The compression algorithms supported by the build of Mksquashfs can be found by typing mksquashfs without any arguments. The compressors available are displayed at the end of the help message, e.g. Compressors available and compressor specific options: gzip (default) -Xcompression-level should be 1 .. 9 (default 9) -Xwindow-size should be 8 .. 15 (default 15) -Xstrategy strategy1,strategy2,...,strategyN Compress using strategy1,strategy2,...,strategyN in turn and choose the best compression. Available strategies: default, filtered, huffman_only, run_length_encoded and fixed lzo -Xalgorithm Where is one of: lzo1x_1 lzo1x_1_11 lzo1x_1_12 lzo1x_1_15 lzo1x_999 (default) -Xcompression-level should be 1 .. 9 (default 8) Only applies to lzo1x_999 algorithm lz4 -Xhc Compress using LZ4 High Compression xz -Xbcj filter1,filter2,...,filterN Compress using filter1,filter2,...,filterN in turn (in addition to no filter), and choose the best compression. Available filters: x86, arm, armthumb, powerpc, sparc, ia64 -Xdict-size Use as the XZ dictionary size. The dictionary size can be specified as a percentage of the block size, or as an absolute value. The dictionary size must be less than or equal to the block size and 8192 bytes or larger. It must also be storable in the xz header as either 2^n or as 2^n+2^(n+1). Example dict-sizes are 75%, 50%, 37.5%, 25%, or 32K, 16K, 8K etc. If the compressor offers compression specific options (all the compressors now have compression specific options except the deprecated lzma1 compressor) then these options are also displayed (.i.e. in the above XZ is shown with two compression specific options). The compression specific options are, obviously, specific to the compressor in question, and the compressor documentation and web sites should be consulted to understand their behaviour. In general the Mksquashfs compression defaults for each compressor are optimised to give the best performance for each compressor, where what constitutes best depends on the compressor. For gzip/xz best means highest compression, for LZO/LZ4 best means a tradeoff between compression and (de)-compression overhead (LZO/LZ4 by definition are intended for weaker processors). 3.3 Changing global compression defaults used in mksquashfs ----------------------------------------------------------- There are a large number of options that can be used to control the compression in mksquashfs. By and large the defaults are the most optimum settings and should only be changed in exceptional circumstances! Note, this does not apply to the block size, increasing the block size from the default of 128Kbytes will increase compression (especially for the xz compressor) and should increase I/O performance too. However, a block size of greater than 128Kbytes may increase latency in certain cases (where the filesystem contains lots of fragments, and no locality of reference is observed). For this reason the block size default is configured to the less optimal 128Kbytes. Users should experiment with 256Kbyte sizes or above. The -noI, -noD and -noF options (also -noInodeCompression, -noDataCompression and -noFragmentCompression) can be used to force mksquashfs to not compress inodes/directories, data and fragments respectively. Giving all options generates an uncompressed filesystem. The -no-fragments tells mksquashfs to not generate fragment blocks, and rather generate a filesystem similar to a Squashfs 1.x filesystem. It will of course still be a Squashfs 4.0 filesystem but without fragments, and so it won't be mountable on a Squashfs 1.x system. The -always-use-fragments option tells mksquashfs to always generate fragments for files irrespective of the file length. By default only small files less than the block size are packed into fragment blocks. The ends of files which do not fit fully into a block, are NOT by default packed into fragments. To illustrate this, a 100K file has an initial 64K block and a 36K remainder. This 36K remainder is not packed into a fragment by default. This is because to do so leads to a 10 - 20% drop in sequential I/O performance, as a disk head seek is needed to seek to the initial file data and another disk seek is need to seek to the fragment block. Specify this option if you want file remainders to be packed into fragment blocks. Doing so may increase the compression obtained BUT at the expense of I/O speed. The -no-duplicates option tells mksquashfs to not check the files being added to the filesystem for duplicates. This can result in quicker filesystem generation and appending although obviously compression will suffer badly if there is a lot of duplicate files. The -b option allows the block size to be selected, both "K" and "M" postfixes are supported, this can be either 4K, 8K, 16K, 32K, 64K, 128K, 256K, 512K or 1M bytes. 3.4 Specifying the UIDs/GIDs used in the filesystem --------------------------------------------------- By default files in the generated filesystem inherit the UID and GID ownership of the original file. However, mksquashfs provides a number of options which can be used to override the ownership. The options -all-root and -root-owned (both do exactly the same thing) force all file uids/gids in the generated Squashfs filesystem to be root. This allows root owned filesystems to be built without root access on the host machine. The "-force-uid uid" option forces all files in the generated Squashfs filesystem to be owned by the specified uid. The uid can be specified either by name (i.e. "root") or by number. The "-force-gid gid" option forces all files in the generated Squashfs filesystem to be group owned by the specified gid. The gid can be specified either by name (i.e. "root") or by number. 3.5 Excluding files from the filesystem --------------------------------------- The -e and -ef options allow files/directories to be specified which are excluded from the output filesystem. The -e option takes the exclude files/directories from the command line, the -ef option takes the exlude files/directories from the specified exclude file, one file/directory per line. Two styles of exclude file matching are supported: basic exclude matching, and extended wildcard matching. Basic exclude matching is a legacy feature retained for backwards compatibility with earlier versions of Mksquashfs. Extended wildcard matching should be used in preference. 3.5.1 Basic exclude matching ---------------------------- Each exclude file is treated as an exact match of a file/directory in the source directories. If an exclude file/directory is absolute (i.e. prefixed with /, ../, or ./) the entry is treated as absolute, however, if an exclude file/directory is relative, it is treated as being relative to each of the sources in turn, i.e. %mksquashfs /tmp/source1 source2 output_fs -e ex1 /tmp/source1/ex2 out/ex3 Will generate exclude files /tmp/source1/ex2, /tmp/source1/ex1, source2/ex1, /tmp/source1/out/ex3 and source2/out/ex3. 3.5.2 Extended exclude file handling ------------------------------------ Extended exclude file matching treats each exclude file as a wildcard or regex expression. To enable wildcard matching specify the -wildcards option, and to enable regex matching specify the -regex option. In most cases the -wildcards option should be used rather than -regex because wildcard matching behaviour is significantly easier to understand! In addition to wildcards/regex expressions, exclude files can be "anchored" or "non-anchored". An anchored exclude is one which matches from the root of the directory and nowhere else, a non-anchored exclude matches anywhere. For example given the directory hierarchy "a/b/c/a/b", the anchored exclude "a/b" will match "a/b" at the root of the directory hierarchy, but it will not match the "/a/b" sub-directory within directory "c", whereas a non-anchored exclude would. A couple of examples should make this clearer. Anchored excludes 1. mksquashfs example image.sqsh -wildcards -e 'test/*.gz' Exclude all files matching "*.gz" in the top level directory "test". 2. mksquashfs example image.sqsh -wildcards -e '*/[Tt]est/example*' Exclude all files beginning with "example" inside directories called "Test" or "test", that occur inside any top level directory. Using extended wildcards, negative matching is also possible. 3. mksquashfs example image.sqsh -wildcards -e 'test/!(*data*).gz' Exclude all files matching "*.gz" in top level directory "test", except those with "data" in the name. Non-anchored excludes By default excludes match from the top level directory, but it is often useful to exclude a file matching anywhere in the source directories. For this non-anchored excludes can be used, specified by pre-fixing the exclude with "...". Examples: 1. mksquashfs example image.sqsh -wildcards -e '... *.gz' Exclude files matching "*.gz" anywhere in the source directories. For example this will match "example.gz", "test/example.gz", and "test/test/example.gz". 2. mksquashfs example image.sqsh -wildcards -e '... [Tt]est/*.gz' Exclude files matching "*.gz" inside directories called "Test" or "test" that occur anywhere in the source directories. Again, using extended wildcards, negative matching is also possible. 3. mksquashfs example image.sqsh -wildcards -e '... !(*data*).gz' Exclude all files matching "*.gz" anywhere in the source directories, except those with "data" in the name. 3.5.3 Exclude files summary --------------------------- The -e and -ef exclude options are usefully used in archiving the entire filesystem, where it is wished to avoid archiving /proc, and the filesystem being generated, i.e. %mksquashfs / /tmp/root.sqsh -e proc /tmp/root.sqsh Multiple -ef options can be specified on the command line, and the -ef option can be used in conjuction with the -e option. 3.6 Appending to squashfs filesystems ------------------------------------- Running squashfs with the destination directory containing an existing filesystem will add the source items to the existing filesystem. By default, the source items are added to the existing root directory. To make this clear... An existing filesystem "image" contains root entries "old1", and "old2". Source directory "/home/phillip/test" contains "file1", "file2" and "dir1". example 1: %mksquashfs /home/phillip/test image Will create a new "image" with root entries "old1", "old2", "file1", "file2" and "dir1" example 2: %mksquashfs /home/phillip/test image -keep-as-directory Will create a new "image" with root entries "old1", "old2", and "test". As shown in the previous section, for single source directories '-keep-as-directory' adds the source directory rather than the contents of the directory. example 3: %mksquashfs /home/phillip/test image -keep-as-directory -root-becomes original-root Will create a new "image" with root entries "original-root", and "test". The '-root-becomes' option specifies that the original root becomes a subdirectory in the new root, with the specified name. The append option with file duplicate detection, means squashfs can be used as a simple versioning archiving filesystem. A squashfs filesystem can be created with for example the linux-2.4.19 source. Appending the linux-2.4.20 source will create a filesystem with the two source trees, but only the changed files will take extra room, the unchanged files will be detected as duplicates. 3.7 Appending recovery file feature ----------------------------------- Recovery files are created when appending to existing Squashfs filesystems. This allows the original filesystem to be recovered if Mksquashfs aborts unexpectedly (i.e. power failure). The recovery files are called squashfs_recovery_xxx_yyy, where "xxx" is the name of the filesystem being appended to, and "yyy" is a number to guarantee filename uniqueness (the PID of the parent Mksquashfs process). Normally if Mksquashfs exits correctly the recovery file is deleted to avoid cluttering the filesystem. If Mksquashfs aborts, the "-recover" option can be used to recover the filesystem, giving the previously created recovery file as a parameter, i.e. mksquashfs dummy image.sqsh -recover squashfs_recovery_image.sqsh_1234 The writing of the recovery file can be disabled by specifying the "-no-recovery" option. 3.8 Pseudo file support ----------------------- Mksquashfs supports pseudo files, these allow fake files, directories, character and block devices to be specified and added to the Squashfs filesystem being built, rather than requiring them to be present in the source directories. This, for example, allows device nodes to be added to the filesystem without requiring root access. Mksquashfs 4.1 added support for "dynamic pseudo files" and a modify operation. Dynamic pseudo files allow files to be dynamically created when Mksquashfs is run, their contents being the result of running a command or piece of shell script. The modifiy operation allows the mode/uid/gid of an existing file in the source filesystem to be modified. Two Mksquashfs options are supported, -p allows one pseudo file to be specified on the command line, and -pf allows a pseudo file to be specified containing a list of pseduo definitions, one per line. 3.8.1. Creating a dynamic file ------------------------------ Pseudo definition Filename f mode uid gid command mode is the octal mode specifier, similar to that expected by chmod. uid and gid can be either specified as a decimal number, or by name. command can be an executable or a piece of shell script, and it is executed by running "/bin/sh -c command". The stdout becomes the contents of "Filename". Examples: Running a basic command ----------------------- /somedir/dmesg f 444 root root dmesg creates a file "/somedir/dmesg" containing the output from dmesg. Executing shell script ---------------------- RELEASE f 444 root root \ if [ ! -e /tmp/ver ]; then \ echo 0 > /tmp/ver; \ fi; \ ver=`cat /tmp/ver`; \ ver=$((ver +1)); \ echo $ver > /tmp/ver; \ echo -n `cat /tmp/release`; \ echo "-dev #"$ver `date` "Build host" `hostname` Creates a file RELEASE containing the release name, date, build host, and an incrementing version number. The incrementing version is a side-effect of executing the shell script, and ensures every time Mksquashfs is run a new version number is used without requiring any other shell scripting. The above example also shows that commands can be split across multiple lines using "\". Obviously as the script will be presented to the shell as a single line, a semicolon is need to separate individual shell commands within the shell script. Reading from a device (or fifo/named socket) -------------------------------------------- input f 444 root root dd if=/dev/sda1 bs=1024 count=10 Copies 10K from the device /dev/sda1 into the file input. Ordinarily Mksquashfs given a device, fifo, or named socket will place that special file within the Squashfs filesystem, the above allows input from these special files to be captured and placed in the Squashfs filesystem. 3.8.2. Creating a block or character device ------------------------------------------- Pseudo definition Filename type mode uid gid major minor Where type is either b - for block devices, and c - for character devices mode is the octal mode specifier, similar to that expected by chmod. uid and gid can be either specified as a decimal number, or by name. For example: /dev/chr_dev c 666 root root 100 1 /dev/blk_dev b 666 0 0 200 200 creates a character device "/dev/chr_dev" with major:minor 100:1 and a block device "/dev/blk_dev" with major:minor 200:200, both with root uid/gid and a mode of rw-rw-rw. 3.8.3. Creating a directory --------------------------- Pseudo definition Filename d mode uid gid mode is the octal mode specifier, similar to that expected by chmod. uid and gid can be either specified as a decimal number, or by name. For example: /pseudo_dir d 666 root root creates a directory "/pseudo_dir" with root uid/gid and mode of rw-rw-rw. 3.8.4. Modifying attributes of an existing file ----------------------------------------------- Pseudo definition Filename m mode uid gid mode is the octal mode specifier, similar to that expected by chmod. uid and gid can be either specified as a decimal number, or by name. For example: dmesg m 666 root root Changes the attributes of the file "dmesg" in the filesystem to have root uid/gid and a mode of rw-rw-rw, overriding the attributes obtained from the source filesystem. 3.9 Miscellaneous options ------------------------- The -info option displays the files/directories as they are compressed and added to the filesystem. The original uncompressed size of each file is printed, along with DUPLICATE if the file is a duplicate of a file in the filesystem. The -nopad option informs mksquashfs to not pad the filesystem to a 4K multiple. This is performed by default to enable the output filesystem file to be mounted by loopback, which requires files to be a 4K multiple. If the filesystem is being written to a block device, or is to be stored in a bootimage, the extra pad bytes are not needed. 4. UNSQUASHFS ------------- Unsquashfs allows you to decompress and extract a Squashfs filesystem without mounting it. It can extract the entire filesystem, or a specific file or directory. The Unsquashfs usage info is: SYNTAX: ./unsquashfs [options] filesystem [directories or files to extract] -v[ersion] print version, licence and copyright information -d[est] unsquash to , default "squashfs-root" -n[o-progress] don't display the progress bar -no[-xattrs] don't extract xattrs in file system -x[attrs] extract xattrs in file system (default) -u[ser-xattrs] only extract user xattrs in file system. Enables extracting xattrs -p[rocessors] use processors. By default will use number of processors available -i[nfo] print files as they are unsquashed -li[nfo] print files as they are unsquashed with file attributes (like ls -l output) -l[s] list filesystem, but don't unsquash -ll[s] list filesystem with file attributes (like ls -l output), but don't unsquash -f[orce] if file already exists then overwrite -s[tat] display filesystem superblock information -e[f] list of directories or files to extract. One per line -da[ta-queue] Set data queue to Mbytes. Default 256 Mbytes -fr[ag-queue] Set fragment queue to Mbytes. Default 256 Mbytes -r[egex] treat extract names as POSIX regular expressions rather than use the default shell wildcard expansion (globbing) Decompressors available: gzip lzo lz4 xz To extract a subset of the filesystem, the filenames or directory trees that are to be extracted can be specified on the command line. The files/directories should be specified using the full path to the files/directories as they appear within the Squashfs filesystem. The files/directories will also be extracted to those positions within the specified destination directory. The extract files can also be given in a file using the "-e[f]" option. Similarly to Mksquashfs, wildcard matching is performed on the extract files. Wildcard matching is enabled by default. Examples: 1. unsquashfs image.sqsh 'test/*.gz' Extract all files matching "*.gz" in the top level directory "test". 2. unsquashfs image.sqsh '[Tt]est/example*' Extract all files beginning with "example" inside top level directories called "Test" or "test". Using extended wildcards, negative matching is also possible. 3. unsquashfs image.sqsh 'test/!(*data*).gz' Extract all files matching "*.gz" in top level directory "test", except those with "data" in the name. 4.1 Unsquashfs options ---------------------- The "-ls" option can be used to list the contents of a filesystem without decompressing the filesystem data itself. The "-lls" option is similar but it also displays file attributes (ls -l style output). The "-info" option forces Unsquashfs to print each file as it is decompressed. The -"linfo" is similar but it also displays file attributes. The "-dest" option specifies the directory that is used to decompress the filesystem data. If this option is not given then the filesystem is decompressed to the directory "squashfs-root" in the current working directory. The "-force" option forces Unsquashfs to output to the destination directory even if files or directories already exist. This allows you to update an existing directory tree, or to Unsquashfs to a partially filled directory. Without the "-force" option, Unsquashfs will refuse to overwrite any existing files, or to create any directories if they already exist. This is done to protect data in case of mistakes, and so the "-force" option should be used with caution. The "-stat" option displays filesystem superblock information. This is useful to discover the filesystem version, byte ordering, whether it has a NFS export table, and what options were used to compress the filesystem, etc. Unsquashfs can decompress all Squashfs filesystem versions, 1.x, 2.x, 3.x and 4.0 filesystems. 5. FILESYSTEM LAYOUT -------------------- A squashfs filesystem consists of a maximum of nine parts, packed together on a byte alignment: --------------- | superblock | |---------------| | compression | | options | |---------------| | datablocks | | & fragments | |---------------| | inode table | |---------------| | directory | | table | |---------------| | fragment | | table | |---------------| | export | | table | |---------------| | uid/gid | | lookup table | |---------------| | xattr | | table | --------------- Compressed data blocks are written to the filesystem as files are read from the source directory, and checked for duplicates. Once all file data has been written the completed super-block, compression options, inode, directory, fragment, export, uid/gid lookup and xattr tables are written. 5.1 Compression options ----------------------- Compressors can optionally support compression specific options (e.g. dictionary size). If non-default compression options have been used, then these are stored here. 5.2 Inodes ---------- Metadata (inodes and directories) are compressed in 8Kbyte blocks. Each compressed block is prefixed by a two byte length, the top bit is set if the block is uncompressed. A block will be uncompressed if the -noI option is set, or if the compressed block was larger than the uncompressed block. Inodes are packed into the metadata blocks, and are not aligned to block boundaries, therefore inodes overlap compressed blocks. Inodes are identified by a 48-bit number which encodes the location of the compressed metadata block containing the inode, and the byte offset into that block where the inode is placed (). To maximise compression there are different inodes for each file type (regular file, directory, device, etc.), the inode contents and length varying with the type. To further maximise compression, two types of regular file inode and directory inode are defined: inodes optimised for frequently occurring regular files and directories, and extended types where extra information has to be stored. 5.3 Directories --------------- Like inodes, directories are packed into compressed metadata blocks, stored in a directory table. Directories are accessed using the start address of the metablock containing the directory and the offset into the decompressed block (). Directories are organised in a slightly complex way, and are not simply a list of file names. The organisation takes advantage of the fact that (in most cases) the inodes of the files will be in the same compressed metadata block, and therefore, can share the start block. Directories are therefore organised in a two level list, a directory header containing the shared start block value, and a sequence of directory entries, each of which share the shared start block. A new directory header is written once/if the inode start block changes. The directory header/directory entry list is repeated as many times as necessary. Directories are sorted, and can contain a directory index to speed up file lookup. Directory indexes store one entry per metablock, each entry storing the index/filename mapping to the first directory header in each metadata block. Directories are sorted in alphabetical order, and at lookup the index is scanned linearly looking for the first filename alphabetically larger than the filename being looked up. At this point the location of the metadata block the filename is in has been found. The general idea of the index is ensure only one metadata block needs to be decompressed to do a lookup irrespective of the length of the directory. This scheme has the advantage that it doesn't require extra memory overhead and doesn't require much extra storage on disk. 5.4 File data ------------- Regular files consist of a sequence of contiguous compressed blocks, and/or a compressed fragment block (tail-end packed block). The compressed size of each datablock is stored in a block list contained within the file inode. To speed up access to datablocks when reading 'large' files (256 Mbytes or larger), the code implements an index cache that caches the mapping from block index to datablock location on disk. The index cache allows Squashfs to handle large files (up to 1.75 TiB) while retaining a simple and space-efficient block list on disk. The cache is split into slots, caching up to eight 224 GiB files (128 KiB blocks). Larger files use multiple slots, with 1.75 TiB files using all 8 slots. The index cache is designed to be memory efficient, and by default uses 16 KiB. 5.5 Fragment lookup table ------------------------- Regular files can contain a fragment index which is mapped to a fragment location on disk and compressed size using a fragment lookup table. This fragment lookup table is itself stored compressed into metadata blocks. A second index table is used to locate these. This second index table for speed of access (and because it is small) is read at mount time and cached in memory. 5.6 Uid/gid lookup table ------------------------ For space efficiency regular files store uid and gid indexes, which are converted to 32-bit uids/gids using an id look up table. This table is stored compressed into metadata blocks. A second index table is used to locate these. This second index table for speed of access (and because it is small) is read at mount time and cached in memory. 5.7 Export table ---------------- To enable Squashfs filesystems to be exportable (via NFS etc.) filesystems can optionally (disabled with the -no-exports Mksquashfs option) contain an inode number to inode disk location lookup table. This is required to enable Squashfs to map inode numbers passed in filehandles to the inode location on disk, which is necessary when the export code reinstantiates expired/flushed inodes. This table is stored compressed into metadata blocks. A second index table is used to locate these. This second index table for speed of access (and because it is small) is read at mount time and cached in memory. 5.8 Xattr table --------------- The xattr table contains extended attributes for each inode. The xattrs for each inode are stored in a list, each list entry containing a type, name and value field. The type field encodes the xattr prefix ("user.", "trusted." etc) and it also encodes how the name/value fields should be interpreted. Currently the type indicates whether the value is stored inline (in which case the value field contains the xattr value), or if it is stored out of line (in which case the value field stores a reference to where the actual value is stored). This allows large values to be stored out of line improving scanning and lookup performance and it also allows values to be de-duplicated, the value being stored once, and all other occurences holding an out of line reference to that value. The xattr lists are packed into compressed 8K metadata blocks. To reduce overhead in inodes, rather than storing the on-disk location of the xattr list inside each inode, a 32-bit xattr id is stored. This xattr id is mapped into the location of the xattr list using a second xattr id lookup table. 6. AUTHOR INFO -------------- Squashfs was written by Phillip Lougher, email phillip@lougher.demon.co.uk, in Chepstow, Wales, UK. If you like the program, or have any problems, then please email me, as it's nice to get feedback! squashfs4.3/README-4.30000644000175000017500000001630512334252013014206 0ustar phillipphillip SQUASHFS 4.3 - A squashed read-only filesystem for Linux Copyright 2002-2014 Phillip Lougher Released under the GPL licence (version 2 or later). Welcome to Squashfs 4.3. This is the first release in over 3 years, and there are substantial improvements to stability, new compression options and compressors, speed optimisations, and new options for Mksquashfs/Unsquashfs. This is a tools only release, support for Squashfs filesystems is in mainline (2.6.29 and later). Changes in Squashfs-tools 4.3 ----------------------------- 1. Stability improvements. Better checking of user input for out of range/invalid values. Better handling of corrupted Squashfs filesystems (Mksquashfs append mode, and Unsquashfs). Better handling of buffer overflow/underflow. 2. GZIP compressor now supports compression options, allowing different compression levels to be used. 3. Rewritten LZO compressor with compression options, allowing different LZO algorithms and different compression levels to be used. 4. New LZ4 compressor (note not yet in mainline kernel) 5. Better default memory usage for Mksquashfs. Mksquashfs by default now uses 25% of physical memory. 6. Duplicate checking in Mksquashfs further optimised. With certain "problem filesystems" greater than 2x performance improvement. Filesystems with a lot of duplicates should see at least 10-20% speed improvement. 7. The -stat option in Unsquashfs now displays the compression options used to generate the original filesystem. Previously -stat only displayed the compression algorithm used. 8. The file being compressed/uncompressed in Mksquashfs/Unsquashfs is now displayed if CTRL-\ (SIGQUIT from keyboard) typed. 9. The status of the internal queues/caches in Mksquashfs/Unsquashfs is now displayed if CTRL-\ (SIGQUIT from keyboard) is typed twice within one second. Normally only useful for "power users", but it can be used to discover if there's any bottlenecks affecting performance (the bottleneck will normally be the compressors/fragment compressors). 10. Miscellaneous new options for Mksquashfs/Unsquashfs to fine tune behaviour. 11. Fixes for CVE-2012-4024 and CVE-2012-4025. Compatiblity ------------ Mksquashfs 4.3 generates 4.0 filesystems. These filesystems are fully compatible/interchangable with filesystems generated by Mksquashfs 4.0 and are mountable on 2.6.29 and later kernels. Compressors ----------- New compression options and compressors are now supported. The new options and compressors are: 1. gzip -Xcompression-level should be 1 .. 9 (default 9) -Xwindow-size should be 8 .. 15 (default 15) -Xstrategy strategy1,strategy2,...,strategyN Compress using strategy1,strategy2,...,strategyN in turn and choose the best compression. Available strategies: default, filtered, huffman_only, run_length_encoded and fixed 2. lzo -Xalgorithm Where is one of: lzo1x_1 lzo1x_1_11 lzo1x_1_12 lzo1x_1_15 lzo1x_999 (default) -Xcompression-level should be 1 .. 9 (default 8) Only applies to lzo1x_999 algorithm 3. lz4 -Xhc Compress using LZ4 High Compression The compression specific options are, obviously, specific to the compressor in question, and you should read the compressor documentation and check their web sites to understand their behaviour. In general the defaults used by Mksquashfs for each compressor are optimised to give the best performance for each compressor, where what constitutes best depends on the compressor. For gzip/xz best means highest compression (trying multiple filters/strategies can improve compression, but this is extremely expensive computationally, and hence, not suitable for the defaults), for LZO/LZ4 best means a tradeoff between compression and (de)-compression overhead (LZO/LZ4 by definition are intended for weaker processors). New Mksquashfs options ---------------------- 1. -mem Set the amount of memory used by Mksquashfs to bytes. G/M and K post-fixes are supported. By default Mksquashfs uses 25% of the physical memory. Increasing this with the -mem option can increase performance (note it does not have any effect on compression). Reducing it can prevent thrashing if the system is busy and there is not 25% of physical memory free (again, note it does not have any effect on compression). 2. -exit-on-error By default Mksquashfs treats certain errors as benign, if these errors occur Mksquashfs prints the error on the console but continues. These errors are typically failure to read a file from the source filesystem. This is deliberate, in many cases users prefer Mksquashfs to flag the error but continue rather than abort what may be hours of compression. But there are times where failure to read any file is considered critical, and users (especially in the case of automated scripts where the errors output to the console may be missed) prefer Mksquashfs to exit. The new -exit-on-error option can be used in this scenario. This option makes Mksquashfs treat all benign errors as fatal. 3. -progress By default if -info is specified, the progress bar is disabled as it gets in the way. Occasionally you might want the progress bar enabled whilst -info is enabled. This option forces Mksquashfs to output the progress bar when -info is specified. 4. -Xhelp Display the usage text for the currently selected compressor. New Unsquashfs options ---------------------- 1. -u[ser-xattrs] Only write user xattrs. This forces Unsquashfs to ignore system xattrs. This is useful when Unsquashing a filesystem as a non-root user, and the filesystem contains system xattrs which are only writable by root. Major bugs fixed ---------------- 1. If Mksquashfs ran out of space in the destination filesystem, this would not cause Mksquashfs to immediately abort, and Mksquashfs would continue to process the source filesystem. Mksquashfs now immediately aborts on out of space in the destination filesystem. 2. Unsquashfs ignored the maximum number of open files limit, and if that was lower than the default limit for Linux, it would run out of file descriptors. Unsquashfs now limits the number of open files to the limit currently in force (e.g. specified by setrlimit). 3. If huge numbers of dynamic pseudo files were specified, Mksquashfs could exceed the maximum number of open files limit. This was because Mksquashfs created all the dynamic file processes up front before commencing source filesystem reading and compression. Mksquashfs now creates the dynamic file processes on demand whilst reading and compressing the source filesystem, thus limiting the number of dynamic pseudo file processes in existence at any one time. 4. When outputting Unsquashfs used to set the permissions of directories as it recursively descended. This in hindsight had an obvious oversight, if a directory had only read permission (or was otherwise restricted), then Unsquashfs would fail to write its contents when descending into it. Fixed by setting directory permissions as Unsquashfs recursively unwinds. squashfs4.3/pseudo-file.example0000644000175000017500000000532711444674255016640 0ustar phillipphillip# Pseudo file example # Mksquashfs supports pseudo files, these allow fake files, directories, # character and block devices to be specified and added to the Squashfs # filesystem being built, rather than requiring them to be present in the # source directories. # # This, for example, allows device nodes to be added to the filesystem without # requiring root access. # Mksquashfs 4.1 adds support for "dynamic pseudo files" and a modify operation. # Dynamic pseudo files allow files to be dynamically created when Mksquashfs # is run, their contents being the result of running a command or piece of # shell script. The modifiy operation allows the mode/uid/gid of an existing # file in the source filesystem to be modified. # Two Mksquashfs options are supported, -p allows one pseudo file to be # specified #on the command line, and -pf allows a pseudo file to be specified # containing a list of pseduo definitions, one per line. # Pseudo file examples # Run mkquashfs . /tmp/img -pf pseudo-file.examples # to see their effect # Creating dynamic file examples # Create a file "dmesg" containing the output from dmesg. dmesg f 444 root root dmesg # Create a file RELEASE containing the release name, date, build host, and # an incrementing version number. The incrementing version is a side-effect # of executing the shell script, and ensures every time Mksquashfs is run a # new version number is used without requiring any other shell scripting. RELEASE f 444 root root \ if [ ! -e /tmp/ver ]; then \ echo 0 > /tmp/ver; \ fi; \ ver=`cat /tmp/ver`; \ ver=$((ver +1)); \ echo $ver > /tmp/ver; \ echo -n "release x.x"; \ echo "-dev #"$ver `date` "Build host" `hostname` # Copy 10K from the device /dev/sda1 into the file input. Ordinarily # Mksquashfs given a device, fifo, or named socket will place that special file # within the Squashfs filesystem, this allows input from these special # files to be captured and placed in the Squashfs filesystem. input f 444 root root dd if=/dev/sda1 bs=1024 count=10 # Creating a block or character device examples # Create a character device "chr_dev" with major:minor 100:1 and # a block device "blk_dev" with major:minor 200:200, both with root # uid/gid and a mode of rw-rw-rw. chr_dev c 666 root root 100 1 blk_dev b 666 0 0 200 200 # Creating a directory example # create a directory "pseudo_dir" with root uid/gid and mode of r--r--r--. pseudo_dir d 444 root root # Modifying attributes of an existing file exmaple # Change the attributes of the file "INSTALL" in the filesystem to have # root uid/gid and a mode of rw-rw-rw, overriding the attributes obtained # from the source filesystem. INSTALL m 666 root root squashfs4.3/DONATIONS0000644000175000017500000000122712332035367014313 0ustar phillipphillipHelp sponsor Squashfs development! Maintaining and improving Squashfs is a lot of work, but Squashfs is one of the only widely used Linux file systems that has no company backing. Squashfs development is funded soley by the author, partially supported by donations from companies and individuals that want to improve Squashfs for themselves and others. There's lots of exciting new improvements to Squashfs in the pipeline, and if your company is a serious user of Squashfs, please consider accelerating development of Squashfs by donating. Donatations can be made from the Squashfs sourceforge homepage, or if you prefer by contacting the author privately. squashfs4.3/OLD-READMEs/0000755000175000017500000000000012332035642014560 5ustar phillipphillipsquashfs4.3/OLD-READMEs/README-3.00000644000175000017500000000405110406111722015730 0ustar phillipphillip SQUASHFS 3.0 - A squashed read-only filesystem for Linux Copyright 2002-2006 Phillip Lougher Released under the GPL licence (version 2 or later). Welcome to the first release of Squashfs version 3.0. Squashfs 3.0 has the the following improvements to 2.x. 1. Filesystems are no longer limited to 4 GB. In theory 2^64 or 4 exabytes is now supported. 2. Files are no longer limited to 4 GB. In theory the maximum file size is 4 exabytes. 3. Metadata (inode table and directory tables) are no longer restricted to 16 Mbytes. 4. Hardlinks are now suppported. 5. Nlink counts are now supported. 6. Readdir now returns '.' and '..' entries. 7. Special support for files larger than 256 MB has been added to the Squashfs kernel code for faster read access. 8. Inode numbers are now stored within the inode rather than being computed from inode location on disk (this is not so much an improvement, but a change forced by the previously listed improvements). There is a new Unsquashfs utility (in squashfs-tools) than can be used to decompress a filesystem without mounting it. Squashfs 3.0 supports 2.x filesystems. Support for 1.x filesystems will be added in the future. 1. UNSQUASHFS ------------- Unsquashfs has the following options: SYNTAX: unsquashfs [-ls | -dest] filesystem -version print version, licence and copyright information -info print files as they are unsquashed -ls list filesystem only -dest unsquash to , default "squashfs-root" The "-ls" option can be used to list the contents of a filesystem without decompressing the filesystem data itself. The "-info" option forces Unsquashfs to print each file as it is decompressed. The "-dest" option specifies the directory that is used to decompress the filesystem data. If this option is not given then the filesystem is decompressed to the directory "squashfs-root" in the current working directory. Unsquashfs can decompress 3.0 filesystems. Support for 2.x and 1.x filesystems will be added in the future. squashfs4.3/OLD-READMEs/README-3.30000644000175000017500000001311310713004611015731 0ustar phillipphillip SQUASHFS 3.3 - A squashed read-only filesystem for Linux Copyright 2002-2007 Phillip Lougher Released under the GPL licence (version 2 or later). Welcome to another release of Squashfs. This is the 22nd release in just over five years of work. Squashfs 3.3 has lots of nice improvements, both to the filesystem itself (bigger blocks, and sparse files), but also to the Squashfs-tools Mksquashfs and Unsquashfs. As usual the CHANGES file has a detailed list of all the improvements. Following is a description of the changes to the Squashfs tools, usage guides to the new options, and a summary of the new options. 1. MKSQUASHFS - EXTENDED EXCLUDE FILE HANDLING ---------------------------------------------- 1. Extended wildcard pattern matching now supported in exclude files Enabled by specifying -wildcards option Supports both anchored and non-anchored exclude files. 1.1 Anchored excludes Similar to existing exclude files except with wildcards. Exclude file matches from root of source directories. Examples: 1. mksquashfs example image.sqsh -wildcards -e 'test/*.gz' Exclude all files matching "*.gz" in the top level directory "test". 2. mksquashfs example image.sqsh -wildcards -e '*/[Tt]est/example*' Exclude all files beginning with "example" inside directories called "Test" or "test", that occur inside any top level directory. Using extended wildcards, negative matching is also possible. 3. mksquashfs example image.sqsh -wildcards -e 'test/!(*data*).gz' Exclude all files matching "*.gz" in top level directory "test", except those with "data" in the name. 1.2 Non-anchored excludes By default excludes match from the top level directory, but it is often useful to exclude a file matching anywhere in the source directories. For this non-anchored excludes can be used, specified by pre-fixing the exclude with "...". Examples: 1. mksquashfs example image.sqsh -wildcards -e '... *.gz' Exclude files matching "*.gz" anywhere in the source directories. For example this will match "example.gz", "test/example.gz", and "test/test/example.gz". 2. mksquashfs example image.sqsh -wildcards -e '... [Tt]est/*.gz' Exclude files matching "*.gz" inside directories called "Test" or "test" that occur anywhere in the source directories. Again, using extended wildcards, negative matching is also possible. 3. mksquashfs example image.sqsh -wildcards -e '... !(*data*).gz' Exclude all files matching "*.gz" anywhere in the source directories, except those with "data" in the name. 2. Regular expression pattern matching now supported in exclude files Enabled by specifying -regex option. Identical behaviour to wild card pattern matching, except patterns are considered to be regular expressions. Supports both anchored and non-anchored exclude files. 2. MKSQUASHFS - NEW RECOVERY FILE FEATURE ----------------------------------------- Recovery files are now created when appending to existing Squashfs filesystems. This allows the original filesystem to be recovered if Mksquashfs aborts unexpectedly (i.e. power failure). The recovery files are called squashfs_recovery_xxx_yyy, where "xxx" is the name of the filesystem being appended to, and "yyy" is a number to guarantee filename uniqueness (the PID of the parent Mksquashfs process). Normally if Mksquashfs exits correctly the recovery file is deleted to avoid cluttering the filesystem. If Mksquashfs aborts, the "-recover" option can be used to recover the filesystem, giving the previously created recovery file as a parameter, i.e. mksquashfs dummy image.sqsh -recover squashfs_recovery_image.sqsh_1234 The writing of the recovery file can be disabled by specifying the "-no-recovery" option. 3. UNSQUASHFS - EXTENDED EXTRACT FILE HANDLING ---------------------------------------------- 1. Multiple extract files can now be specified on the command line, and the files/directories to be extracted can now also be given in a file. To specify a file containing the extract files use the "-e[f]" option. 2. Extended wildcard pattern matching now supported in extract files Enabled by default. Similar to existing extract files except with wildcards. Examples: 1. unsquashfs image.sqsh 'test/*.gz' Extract all files matching "*.gz" in the top level directory "test". 2. unsquashfs image.sqsh '[Tt]est/example*' Extract all files beginning with "example" inside top level directories called "Test" or "test". Using extended wildcards, negative matching is also possible. 3. unsquashfs image.sqsh 'test/!(*data*).gz' Extract all files matching "*.gz" in top level directory "test", except those with "data" in the name. 3. Regular expression pattern matching now supported in extract files Enabled by specifying -r[egex] option. Identical behaviour to wild card pattern matching, except patterns are considered to be regular expressions. 4. UNSQUASHFS - EXTENDED FILENAME PRINTING ------------------------------------------ Filename printing has been enhanced and Unquashfs can now display filenames with file attributes ('ls -l' style output). New options: -ll[s] list filesystem with file attributes, but don't unsquash -li[nfo] print files as they are unsquashed with file attributes 5. UNSQUASHFS - MISCELLANEOUS OPTIONS ------------------------------------- -s[tat] Display the filesystem superblock information. This is useful to discover the filesystem version, byte ordering, whether it has an NFS export table, and what options were used to compress the filesystem. squashfs4.3/OLD-READMEs/README-3.20000644000175000017500000000177510546200653015753 0ustar phillipphillip SQUASHFS 3.2 - A squashed read-only filesystem for Linux Copyright 2002-2007 Phillip Lougher Released under the GPL licence (version 2 or later). Welcome to Squashfs version 3.2. Squashfs 3.2 has support for NFS exporting, some improvements to the Squashfs tools (Mksquashfs and Unsquashfs), some major bug fixes, new kernel patches, and various other smaller improvements and bug fixes. Please see the CHANGES file for a detailed list. 1. MKSQUASHFS ------------- New command line options: -no-exports Squashfs now supports NFS exports. By default the additional information necessary is added to the filesystem by Mksquashfs. If you do not wish this extra information, then this option can be specified. This will save a couple of bytes per file, and the filesystem will be identical to Squashfs 3.1. -no-progress Mksquashfs by default now displays a progress bar. This option disables it. 2. UNSQUASHFS ------------- Unsquashfs now supports Squashfs 2.x filesystems. squashfs4.3/OLD-READMEs/README-2.00000644000175000017500000001440210406104324015730 0ustar phillipphillipNOTE: This the original README for version 2.0. It is retained as it contains information about the fragment design. A description of the new 2.0 mksquashfs options has been added to the main README file, and that file should now be consulted for these. SQUASHFS 2.0 - A squashed read-only filesystem for Linux Copyright 2004 Phillip Lougher (plougher@users.sourceforge.net) Released under the GPL licence (version 2 or later). Welcome to the final release of Squashfs version 2.0! A lot of changes to the filesystem have been made under the bonnet (hood). Squashfs 2.0 uses fragment blocks and larger blocks (64K) to improve compression ratio by about 5 - 20% over Squashfs 1.0 depending on the files being compressed. Using fragment blocks allows Squashfs 2.0 to achieve better compression than cloop and similar compression to tgz files while retaining the I/O efficiency of a compressed filesystem. Detailed changes: 1. Squashfs 2.0 has added the concept of fragment blocks (see later discussion). Files smaller than the file block size (64K in Squashfs 2.0) and optionally the remainder of files that do not fit fully into a block (i.e. the last 32K in a 96K file) are packed into shared fragments and compressed together. This achieves on average 5 - 20% better compression than Squashfs 1.x. 2. The maximum block size has been increased to 64K. 3. The maximum number of UIDs has been increased to 256 (from 48 in 1.x). 4. The maximum number of GIDs has been increased to 256 (from 15 in 1.x). 5. New mksquashfs -all-root, -root-owned, -force-uid, and -force-gid options. These allow the uids/gids of files in the generated filesystem to be specified, overriding the uids/gids in the source filesystem. 6. Initrds are now supported for kernels 2.6.x. 7. Removal of sleep_on() function call in 2.6.x patch, to allow Squashfs to work on the Fedora rc2 kernel. 8. AMD64, check-data and gid bug fixes. 9. Numerous small bug fixes have been made. 10. New patch for Linux 2.6.7. New Squashfs 2.0 options ------------------------ -noF or -noFragmentCompression Do not compress the fragments. Added for compatibility with noI and noD, probably not that useful. -no-fragments Do not use fragment blocks, and rather generate a filesystem similar to a Squashfs 1.x filesystem. It will of course still be a Squashfs 2.0 filesystem but without fragments, and so it won't be mountable on a Squashfs 1.x system. -always-use-fragments By default only small files less than the block size are packed into fragment blocks. The ends of files which do not fit fully into a block, are NOT by default packed into fragments. To illustrate this, a 100K file has an initial 64K block and a 36K remainder. This 36K remainder is not packed into a fragment by default. This is because to do so leads to a 10 - 20% drop in sequential I/O performance, as a disk head seek is needed to seek to the initial file data and another disk seek is need to seek to the fragment block. Specify this option if you want file remainders to be packed into fragment blocks. Doing so may increase the compression obtained BUT at the expense of I/O speed. -no-duplicates Do not detect duplicate files. -all-root -root-owned These options (both do exactly the same thing), force all file uids/gids in the generated Squashfs filesystem to be root. This allows root owned filesystems to be built without root access on the host machine. -force-uid uid This option forces all files in the generated Squashfs filesystem to be owned by the specified uid. The uid can be specified either by name (i.e. "root") or by number. -force-gid gid This option forces all files in the generated Squashfs filesystem to be group owned by the specified gid. The gid can be specified either by name (i.e. "root") or by number. Compression improvements example -------------------------------- The following is the compression results obtained compressing the 2.6.6 linux kernel source using CRAMFS, Cloop (with iso filesystem), Squashfs 1.3 and Squashfs 2.0 (results generated using big-endian filesystems). In decreasing order of size: CRAMFS 62791680 bytes (59.9M) Squashfs 1.x 51351552 bytes (48.9M) Cloop 46118681 bytes (44.0M) Squashfs 2.0 45604854 bytes (43.5M) The Squashfs 1.x filesystem is 12.6% larger than the new 2.0 filesystem. The cloop filesystem is 1.1% larger than the Squashfs 2.0 filesystem. Fragment blocks in Squashfs 2.0 ------------------------------- Squashfs like all other compressed filesystems compresses files individually on a block by block basis. This is performed to allow mounting and de-compression of files on a block by block basis without requiring the entire filesystem to be decompressed. This is in contrast to data-based compression schemes which compress without understanding the underlying filesystem (i.e. cloop and tgz files) and which, therefore, do not compress files individually. Each approach has advantages and disadvantages, data-based systems have better compression because compression is always performed at the maximum block size (64K in cloop) irrespective of the size of each file (which could be less than the block size). Compressed filesystems tend to be faster at I/O because they understand the filesystem and therefore employ better caching stategies and read less un-needed data from the filesystem. Fragment blocks in Squashfs 2.0 solves this problem by packing files (and optionally the ends of files) which are smaller than the block size into shared blocks, which are compressed together. For example five files each of 10K will be packed into one shared fragment of 50K and compressed together, rather than being compressed in five 10K blocks. This scheme produces a hybrid filesystem, retaining the I/O efficiency of a compressed filesystem, while obtaining the compression efficiency of data-based schemes by compressing small files together. Squashfs 1.x and Squashfs 2.0 compatibility ------------------------------------------- Appending to Squashfs 1.x filesystems is not supported. If you wish to append to 1.x filesystems, then either use the original mksquashfs, or convert them to Squashfs 2.0 by mounting the filesystem and running the 2.0 mksquashfs on the mounted filesystem. Mounting Squashfs 1.x filesystems IS supported by the 2.0 kernel patch. squashfs4.3/OLD-READMEs/README-3.10000644000175000017500000001207610546166374015761 0ustar phillipphillip SQUASHFS 3.1 - A squashed read-only filesystem for Linux Copyright 2002-2006 Phillip Lougher Released under the GPL licence (version 2 or later). Welcome to Squashfs version 3.1-r2. Squashfs 3.1 has major improvements to the Squashfs tools (Mksquashfs and Unsquashfs), some major bug fixes, new kernel patches, and various other smaller improvements and bug fixes. Please see the CHANGES file for a detailed list. 1. MKSQUASHFS ------------- Mksquashfs has been rewritten and it is now multi-threaded. It offers the following improvements: 1. Parallel compression. By default as many compression and fragment compression threads are created as there are available processors. This significantly speeds up performance on SMP systems. 2. File input and filesystem output is peformed in parallel on separate threads to maximise I/O performance. Even on single processor systems this speeds up performance by at least 10%. 3. Appending has been significantly improved, and files within the filesystem being appended to are no longer scanned and checksummed. This significantly improves append time for large filesystems. 4. File duplicate checking has been optimised, and split into two separate phases. Only files which are considered possible duplicates after the first phase are checksummed and cached in memory. 5. The use of swap memory was found to significantly impact performance. The amount of memory used to cache the file is now a command line option, by default this is 512 Mbytes. 1.1 NEW COMMAND LINE OPTIONS ---------------------------- The new Mksquashfs program has a couple of extra command line options which can be used to control the new features: -processors This specifies the number of processors used by Mksquashfs. By default this is the number of available processors. -read_queue This specifies the size of the file input queue used by the reader thread. This defaults to 64 Mbytes. -write_queue This specifies the size of the filesystem output queue used by the writer thread. It also specifies the maximum cache used in file duplicate detection (the output queue is shared between these tasks). This defaults to 512 Mbytes. 1.2 PERFORMANCE RESULTS ----------------------- The following results give an indication of the speed improvements. Two example filesystems were tested, a liveCD filesystem (about 1.8 Gbytes uncompressed), and my home directory consisting largely of text files (about 1.3 Gbytes uncompressed). Tests were run on a single core and a dual core system. Dual Core (AMDx2 3800+) system: Source directories on ext3. LiveCD, old mksquashfs: real 11m48.401s user 9m27.056s sys 0m15.281s LiveCD, new par_mksquashfs: real 4m8.736s user 7m11.771s sys 0m27.749s "Home", old mksquashfs: real 4m34.360s user 3m54.007s sys 0m32.155s "Home", new par_mksquashfs: real 1m27.381s user 2m7.304s sys 0m17.234s Single Core PowerBook (PowerPC G4 1.5 GHz Ubuntu Linux) Source directories on ext3. LiveCD, old mksquashs: real 11m38.472s user 9m6.137s sys 0m23.799s LiveCD, par_mksquashfs: real 10m5.572s user 8m59.921s sys 0m16.145s "Home", old mksquashfs: real 3m42.298s user 2m49.478s sys 0m13.675s "Home", new par_mksquashfs: real 3m9.178s user 2m50.699s sys 0m9.069s I'll be interested in any performance results obtained, especially from SMP machines larger than my dual-core AMD box, as this will give an indication of the scalability of the code. Obviously, I'm also interested in any problems, deadlocks, low performance etc. 2. UNSQUASHFS ------------- Unsquashfs now allows you to specify the filename or directory that is to be extracted from the Squashfs filesystem, rather than always extracting the entire filesystem. It also has a new "-force" option, and all options can be specified in a short form (-i rather than -info). The Unsquashfs usage info is now: SYNTAX: ./unsquashfs [options] filesystem [directory or file to extract] -v[ersion] print version, licence and copyright information -i[nfo] print files as they are unsquashed -l[s] list filesystem only -d[est] unsquash to , default "squashfs-root" -f[orce] if file already exists then overwrite To extract a subset of the filesystem, the filename or directory tree that is to be extracted can now be specified on the command line. The file/directory should be specified using the full path to the file/directory as it appears within the Squashfs filesystem. The file/directory will also be extracted to that position within the specified destination directory. The new "-force" option forces Unsquashfs to output to the destination directory even if files or directories already exist. This allows you to update an existing directory tree, or to Unsquashfs to a partially filled directory. Without the "-force" option, Unsquashfs will refuse to overwrite any existing files, or to create any directories if they already exist. This is done to protect data in case of mistakes, and so the "-force" option should be used with caution. squashfs4.3/OLD-READMEs/README-4.00000644000175000017500000000272111166231613015741 0ustar phillipphillip SQUASHFS 4.0 - A squashed read-only filesystem for Linux Copyright 2002-2009 Phillip Lougher Released under the GPL licence (version 2 or later). Welcome to Squashfs 4.0. This is an initial tools only release to support users of the 2.6.29 kernel, following the mainlining of Squashfs earlier this year. Later releases will probably contain kernel patches supporting 4.0 layouts for earlier kernels. New Mksquashfs options ---------------------- Mksquashfs now supports pseudo files, these allow fake directories, character and block devices to be specified and added to the Squashfs filesystem being built, rather than requiring them to be present in the source directories. This, for example, allows device nodes to be added to the filesystem without requiring root access. Two options are supported, -p allows one pseudo file to be specified on the command line, and -pf allows a pseudo file to be specified containing a list of pseduo definitions, one per line. Pseudo device nodes are specified using 7 arguments Filename type mode uid gid major minor Where type is either b - for block devices, and c - for character devices mode is the octal mode specifier, similar to that expected by chmod. Uid and gid can be either specified as a decimal number, or by name. For example: /dev/chr_dev c 666 root root 100 1 /dev/blk_dev b 444 0 0 200 200 Directories are specified using 5 arguments Filename type mode uid gid Where type is d. squashfs4.3/OLD-READMEs/README-2.10000644000175000017500000000700210406104324015727 0ustar phillipphillip SQUASHFS 2.1 - A squashed read-only filesystem for Linux Copyright 2004 Phillip Lougher (plougher@users.sourceforge.net) Released under the GPL licence (version 2 or later). Welcome to Squashfs version 2.1-r2. Squashfs 2.1 introduces indexed directories which considerably speed up directory lookup (ls, find etc.) for directories which are greater than 8K in size. All directories are now also sorted alphabetically which further speeds up directory lookup. Many smaller improvements have also been made to this release, please see the CHANGES file entry for detailed changes. 1. DIRECTORY SPEED IMPROVEMENT EXAMPLES --------------------------------------- To give an indication of the directory speed improvements a number of test results are shown here. There is in addition a new PERFORMANCE.README file which gives details of I/O and lookup performance for Squashfs 2.1 against the Zisofs, Cloop and CRAMFS filesystems. example 1: Filesystems generated from a single directory of 72,784 files (2.6 MB directory size). Each file is 10 bytes in size (the test is directory lookup and so the file size isn't an issue). The ext3 uncompressed directory size is 288 MB (presumably because of one file per block). Zisofs compressed size 153.50 MB Cloop (isofs) compressed size 1.74 MB Squashfs2.1 compressed size 612 KB (0.60 MB) Time taken to perform "ls -lR --color=always | cat > /dev/null" on filesystems mounted on hard disk. Zisofs 35 minutes 7.895 seconds (User 7.868 secs, Sys 34 mins 5.621 secs) Cloop 35 minutes 12.765 seconds (User 7.771 secs, Sys 34 mins 3.869 secs) Squashfs2.1 19 seconds (User 5.119 secs, Sys 14.547 secs) example 2: Filesystems were generated from the Ubuntu Warty livecd (original uncompressed size on ext3 is 1.4 GB). Zisofs compressed size 589.81 MB Cloop (isofs) compressed size 471.19 MB Squashfs2.0 compressed size 448.58 MB Squashfs2.1 compressed size 448.58 MB Time taken to perform "ls -lR --color=always | cat > /dev/null" on filesystems mounted on hard disk. Zisofs 49.875 seconds (User time 2.589 secs, Sys 11.194 secs) Cloop 20.797 seconds (User time 2.706 secs, Sys 13.496 secs) Squashfs2.0 16.556 seconds (User time 2.424 secs, Sys 10.371 secs) Squashfs2.1 10.143 seconds (User time 2.475 secs, Sys 4.440 secs) NOTE: the usual warnings apply to these results, they are provided for illustrative purposes only, and due to different hardware and/or file data, you may obtain different results. As such the results are provided "as is" without any warranty (either express or implied) and you assume all risks as to their quality and accuracy. 2. NEW MKSQUASHFS OPTIONS ------------------------- There is only one extra option "-2.0". This tells mksquashfs to generate a filesystem which is mountable with Squashfs version 2.0. 3. APPENDING AND MOUNTING SQUASHFS 2.0 FILESYSTEMS -------------------------------------------------- Mounting 2.0 filesystems is supported by Squashfs 2.1. In addition mksquashfs v2.1 can append to 2.0 filesystems, although the generated filesystem will still be a 2.0 filesystem. 4. DONATIONS ------------ If you find Squashfs useful then please consider making a donation, particularly if you use Squashfs in a commercial product. Please consider giving something back especially if you're making money from it. Off the Squashfs subject somewhat I'm currently looking for another job doing Linux kernel or filesystems work. If you know of any such work that can be performed from the UK then please get in touch. Thanks. squashfs4.3/OLD-READMEs/README-4.20000644000175000017500000000361211533011301015727 0ustar phillipphillip SQUASHFS 4.2 - A squashed read-only filesystem for Linux Copyright 2002-2011 Phillip Lougher Released under the GPL licence (version 2 or later). Welcome to Squashfs 4.2. This is a tools only release, support for Squashfs filesystems is in mainline (2.6.29 and later). New features in Squashfs-tools 4.2 ---------------------------------- 1. Support for XZ compression 2. Support for compressor specific options Compatiblity ------------ Mksquashfs 4.2 generates 4.0 filesystems. These filesystems are fully compatible/interchangable with filesystems generated by Mksquashfs 4.0 and are mountable on 2.6.29 and later kernels. XZ compression -------------- Squashfs now supports XZ compression. XZ support is in 2.6.38 and newer kernels. New Mksquashfs options ---------------------- -X Compression algorithms can now support compression specific options. These options are prefixed by -X, and are passed to the compressor for handling. The compression specific options supported by each compressor can be found by typing mksquashfs without any arguments. They are displayed at the end of the help message, e.g. Compressors available and compressor specific options: gzip (no options) (default) lzo (no options) xz -Xbcj filter1,filter2,...,filterN Compress using filter1,filter2,...,filterN in turn (in addition to no filter), and choose the best compression. Available filters: x86, arm, armthumb, powerpc, sparc, ia64 -Xdict-size Use as the XZ dictionary size. The dictionary size can be specified as a percentage of the block size, or as an absolute value. The dictionary size must be less than or equal to the block size and 8192 bytes or larger. It must also be storable in the xz header as either 2^n or as 2^n+2^(n+1). Example dict-sizes are 75%, 50%, 37.5%, 25%, or 32K, 16K, 8K etc. squashfs4.3/OLD-READMEs/README-AMD640000644000175000017500000000172610406104324016211 0ustar phillipphillipInformation for amd64 users --------------------------- All releases of Squashfs prior to 2.0 generate incorrect filesystems on amd64 machines. Filesystems created on amd64 machines work correctly on amd64 machines, but cannot be mounted on non-amd64 machines. Likewise, filesystems created on non-amd64 machines cannot be mounted on amd64 machines. This bug is caused by the different size of the "time_t" definition used in SquashFS filesystem structures. This bug is fixed in releases 2.0 and newer. However, all amd64 filesystems generated by previous releases will not be mountable on amd64 machines with newer releases. If you have amd64 filesystems generated with mksquashfs version 2.0-alpha or older, it is important that you recreate the filesystem. This can be performed by mounting the filesystem using a kernel with the original patch (i.e. a 2.0-alpha or older patch) and running the NEW (i.e. this release) mksquashfs tool to create a new SquashFS filesystem. squashfs4.3/OLD-READMEs/README-4.10000644000175000017500000002023411444574644015756 0ustar phillipphillip SQUASHFS 4.1 - A squashed read-only filesystem for Linux Copyright 2002-2010 Phillip Lougher Released under the GPL licence (version 2 or later). Welcome to Squashfs 4.1. This is a tools only release, support for Squashfs file systems is in mainline (2.6.29 and later). New features in Squashfs-tools 4.1 ---------------------------------- 1. Support for extended attributes 2. Support for LZMA and LZO compression 3. New pseudo file features Compatiblity ------------ Mksquashfs 4.1 generates 4.0 filesystems. These filesystems are fully compatible/interchangable with filesystems generated by Mksquashfs 4.0 and are mountable on 2.6.29 and later kernels. Extended attributes (xattrs) ---------------------------- Squashfs file systems now have extended attribute support. The extended attribute implementation has the following features: 1. Layout can store up to 2^48 bytes of compressed xattr data. 2. Number of xattrs per inode unlimited. 3. Total size of xattr data per inode 2^48 bytes of compressed data. 4. Up to 4 Gbytes of data per xattr value. 5. Inline and out-of-line xattr values supported for higher performance in xattr scanning (listxattr & getxattr), and to allow xattr value de-duplication. 6. Both whole inode xattr duplicate detection and individual xattr value duplicate detection supported. These can obviously nest, file C's xattrs can be a complete duplicate of file B, and file B's xattrs can be a partial duplicate of file A. 7. Xattr name prefix types stored, allowing the redundant "user.", "trusted." etc. characters to be eliminated and more concisely stored. 8. Support for files, directories, symbolic links, device nodes, fifos and sockets. Extended attribute support is in 2.6.35 and later kernels. File systems with extended attributes can be mounted on 2.6.29 and later kernels, the extended attributes will be ignored with a warning. LZMA and LZO compression ------------------------ Squashfs now supports LZMA and LZO compression. LZO support is in 2.6.36 and newer kernels. LZMA is not yet in mainline. New Mksquashfs options ---------------------- -comp Select compression. The compression algorithms supported by the build of Mksquashfs can be found by typing mksquashfs without any arguments. The compressors available are displayed at the end of the help message, e.g. Compressors available: gzip (default) lzma lzo The default compression used when -comp isn't specified on the command line is indicated by "(default)". -no-xattrs Don't store extended attributes -xattrs Store extended attributes The default behaviour of Mksquashfs with respect to extended attribute storage is build time selectable. The Mksquashfs help message indicates whether extended attributes are stored or not, e.g. -no-xattrs don't store extended attributes -xattrs store extended attributes (default) shows that extended attributes are stored by default, and can be disabled by the -no-xattrs option. -no-xattrs don't store extended attributes (default) -xattrs store extended attributes shows that extended attributes are not stored by default, storage can be enabled by the -xattrs option. -noX -noXattrCompression Don't compress extended attributes New Unsquashfs options ---------------------- -n[o-xattrs] Don't extract xattrs in filesystem -x[attrs] Extract xattrs in filesystem The default behaviour of Unsquashfs with respect to extended attributes is build time selectable. The Unsquashfs help message indicates whether extended attributes are stored or not, e.g. -no[-xattrs] don't extract xattrs in file system -x[attrs] extract xattrs in file system (default) shows that xattrs are extracted by default. -no[-xattrs] don't extract xattrs in file system (default) -x[attrs] extract xattrs in file system shows that xattrs are not extracted by default. New pseudo file support ----------------------- Mksquashfs supports pseudo files, these allow fake files, directories, character and block devices to be specified and added to the Squashfs filesystem being built, rather than requiring them to be present in the source directories. This, for example, allows device nodes to be added to the filesystem without requiring root access. Mksquashfs 4.1 adds support for "dynamic pseudo files" and a modify operation. Dynamic pseudo files allow files to be dynamically created when Mksquashfs is run, their contents being the result of running a command or piece of shell script. The modifiy operation allows the mode/uid/gid of an existing file in the source filesystem to be modified. Two Mksquashfs options are supported, -p allows one pseudo file to be specified on the command line, and -pf allows a pseudo file to be specified containing a list of pseduo definitions, one per line. Pseudo operations ----------------- 1. Creating a dynamic file -------------------------- Pseudo definition Filename f mode uid gid command mode is the octal mode specifier, similar to that expected by chmod. uid and gid can be either specified as a decimal number, or by name. command can be an executable or a piece of shell script, and it is executed by running "/bin/sh -c command". The stdout becomes the contents of "Filename". Examples: Running a basic command ----------------------- /somedir/dmesg f 444 root root dmesg creates a file "/somedir/dmesg" containing the output from dmesg. Executing shell script ---------------------- RELEASE f 444 root root \ if [ ! -e /tmp/ver ]; then \ echo 0 > /tmp/ver; \ fi; \ ver=`cat /tmp/ver`; \ ver=$((ver +1)); \ echo $ver > /tmp/ver; \ echo -n `cat /tmp/release`; \ echo "-dev #"$ver `date` "Build host" `hostname` Creates a file RELEASE containing the release name, date, build host, and an incrementing version number. The incrementing version is a side-effect of executing the shell script, and ensures every time Mksquashfs is run a new version number is used without requiring any other shell scripting. The above example also shows that commands can be split across multiple lines using "\". Obviously as the script will be presented to the shell as a single line, a semicolon is need to separate individual shell commands within the shell script. Reading from a device (or fifo/named socket) -------------------------------------------- input f 444 root root dd if=/dev/sda1 bs=1024 count=10 Copies 10K from the device /dev/sda1 into the file input. Ordinarily Mksquashfs given a device, fifo, or named socket will place that special file within the Squashfs filesystem, the above allows input from these special files to be captured and placed in the Squashfs filesystem. 2. Creating a block or character device --------------------------------------- Pseudo definition Filename type mode uid gid major minor Where type is either b - for block devices, and c - for character devices mode is the octal mode specifier, similar to that expected by chmod. uid and gid can be either specified as a decimal number, or by name. For example: /dev/chr_dev c 666 root root 100 1 /dev/blk_dev b 666 0 0 200 200 creates a character device "/dev/chr_dev" with major:minor 100:1 and a block device "/dev/blk_dev" with major:minor 200:200, both with root uid/gid and a mode of rw-rw-rw. 3. Creating a directory ----------------------- Pseudo definition Filename d mode uid gid mode is the octal mode specifier, similar to that expected by chmod. uid and gid can be either specified as a decimal number, or by name. For example: /pseudo_dir d 666 root root creates a directory "/pseudo_dir" with root uid/gid and mode of rw-rw-rw. 4. Modifying attributes of an existing file ------------------------------------------- Pseudo definition Filename m mode uid gid mode is the octal mode specifier, similar to that expected by chmod. uid and gid can be either specified as a decimal number, or by name. For example: dmesg m 666 root root Changes the attributes of the file "dmesg" in the filesystem to have root uid/gid and a mode of rw-rw-rw, overriding the attributes obtained from the source filesystem. squashfs4.3/COPYING0000644000175000017500000004325411352247755014102 0ustar phillipphillip GNU GENERAL PUBLIC LICENSE Version 2, June 1991 Copyright (C) 1989, 1991 Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The licenses for most software are designed to take away your freedom to share and change it. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change free software--to make sure the software is free for all its users. This General Public License applies to most of the Free Software Foundation's software and to any other program whose authors commit to using it. (Some other Free Software Foundation software is covered by the GNU Lesser General Public License instead.) You can apply it to your programs, too. When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for this service if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs; and that you know you can do these things. To protect your rights, we need to make restrictions that forbid anyone to deny you these rights or to ask you to surrender the rights. These restrictions translate to certain responsibilities for you if you distribute copies of the software, or if you modify it. For example, if you distribute copies of such a program, whether gratis or for a fee, you must give the recipients all the rights that you have. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights. We protect your rights with two steps: (1) copyright the software, and (2) offer you this license which gives you legal permission to copy, distribute and/or modify the software. Also, for each author's protection and ours, we want to make certain that everyone understands that there is no warranty for this free software. If the software is modified by someone else and passed on, we want its recipients to know that what they have is not the original, so that any problems introduced by others will not reflect on the original authors' reputations. Finally, any free program is threatened constantly by software patents. We wish to avoid the danger that redistributors of a free program will individually obtain patent licenses, in effect making the program proprietary. To prevent this, we have made it clear that any patent must be licensed for everyone's free use or not licensed at all. The precise terms and conditions for copying, distribution and modification follow. GNU GENERAL PUBLIC LICENSE TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 0. This License applies to any program or other work which contains a notice placed by the copyright holder saying it may be distributed under the terms of this General Public License. The "Program", below, refers to any such program or work, and a "work based on the Program" means either the Program or any derivative work under copyright law: that is to say, a work containing the Program or a portion of it, either verbatim or with modifications and/or translated into another language. (Hereinafter, translation is included without limitation in the term "modification".) Each licensee is addressed as "you". Activities other than copying, distribution and modification are not covered by this License; they are outside its scope. The act of running the Program is not restricted, and the output from the Program is covered only if its contents constitute a work based on the Program (independent of having been made by running the Program). Whether that is true depends on what the Program does. 1. You may copy and distribute verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice and disclaimer of warranty; keep intact all the notices that refer to this License and to the absence of any warranty; and give any other recipients of the Program a copy of this License along with the Program. You may charge a fee for the physical act of transferring a copy, and you may at your option offer warranty protection in exchange for a fee. 2. You may modify your copy or copies of the Program or any portion of it, thus forming a work based on the Program, and copy and distribute such modifications or work under the terms of Section 1 above, provided that you also meet all of these conditions: a) You must cause the modified files to carry prominent notices stating that you changed the files and the date of any change. b) You must cause any work that you distribute or publish, that in whole or in part contains or is derived from the Program or any part thereof, to be licensed as a whole at no charge to all third parties under the terms of this License. c) If the modified program normally reads commands interactively when run, you must cause it, when started running for such interactive use in the most ordinary way, to print or display an announcement including an appropriate copyright notice and a notice that there is no warranty (or else, saying that you provide a warranty) and that users may redistribute the program under these conditions, and telling the user how to view a copy of this License. (Exception: if the Program itself is interactive but does not normally print such an announcement, your work based on the Program is not required to print an announcement.) These requirements apply to the modified work as a whole. If identifiable sections of that work are not derived from the Program, and can be reasonably considered independent and separate works in themselves, then this License, and its terms, do not apply to those sections when you distribute them as separate works. But when you distribute the same sections as part of a whole which is a work based on the Program, the distribution of the whole must be on the terms of this License, whose permissions for other licensees extend to the entire whole, and thus to each and every part regardless of who wrote it. Thus, it is not the intent of this section to claim rights or contest your rights to work written entirely by you; rather, the intent is to exercise the right to control the distribution of derivative or collective works based on the Program. In addition, mere aggregation of another work not based on the Program with the Program (or with a work based on the Program) on a volume of a storage or distribution medium does not bring the other work under the scope of this License. 3. You may copy and distribute the Program (or a work based on it, under Section 2) in object code or executable form under the terms of Sections 1 and 2 above provided that you also do one of the following: a) Accompany it with the complete corresponding machine-readable source code, which must be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or, b) Accompany it with a written offer, valid for at least three years, to give any third party, for a charge no more than your cost of physically performing source distribution, a complete machine-readable copy of the corresponding source code, to be distributed under the terms of Sections 1 and 2 above on a medium customarily used for software interchange; or, c) Accompany it with the information you received as to the offer to distribute corresponding source code. (This alternative is allowed only for noncommercial distribution and only if you received the program in object code or executable form with such an offer, in accord with Subsection b above.) The source code for a work means the preferred form of the work for making modifications to it. For an executable work, complete source code means all the source code for all modules it contains, plus any associated interface definition files, plus the scripts used to control compilation and installation of the executable. However, as a special exception, the source code distributed need not include anything that is normally distributed (in either source or binary form) with the major components (compiler, kernel, and so on) of the operating system on which the executable runs, unless that component itself accompanies the executable. If distribution of executable or object code is made by offering access to copy from a designated place, then offering equivalent access to copy the source code from the same place counts as distribution of the source code, even though third parties are not compelled to copy the source along with the object code. 4. You may not copy, modify, sublicense, or distribute the Program except as expressly provided under this License. Any attempt otherwise to copy, modify, sublicense or distribute the Program is void, and will automatically terminate your rights under this License. However, parties who have received copies, or rights, from you under this License will not have their licenses terminated so long as such parties remain in full compliance. 5. You are not required to accept this License, since you have not signed it. However, nothing else grants you permission to modify or distribute the Program or its derivative works. These actions are prohibited by law if you do not accept this License. Therefore, by modifying or distributing the Program (or any work based on the Program), you indicate your acceptance of this License to do so, and all its terms and conditions for copying, distributing or modifying the Program or works based on it. 6. Each time you redistribute the Program (or any work based on the Program), the recipient automatically receives a license from the original licensor to copy, distribute or modify the Program subject to these terms and conditions. You may not impose any further restrictions on the recipients' exercise of the rights granted herein. You are not responsible for enforcing compliance by third parties to this License. 7. If, as a consequence of a court judgment or allegation of patent infringement or for any other reason (not limited to patent issues), conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot distribute so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not distribute the Program at all. For example, if a patent license would not permit royalty-free redistribution of the Program by all those who receive copies directly or indirectly through you, then the only way you could satisfy both it and this License would be to refrain entirely from distribution of the Program. If any portion of this section is held invalid or unenforceable under any particular circumstance, the balance of the section is intended to apply and the section as a whole is intended to apply in other circumstances. It is not the purpose of this section to induce you to infringe any patents or other property right claims or to contest validity of any such claims; this section has the sole purpose of protecting the integrity of the free software distribution system, which is implemented by public license practices. Many people have made generous contributions to the wide range of software distributed through that system in reliance on consistent application of that system; it is up to the author/donor to decide if he or she is willing to distribute software through any other system and a licensee cannot impose that choice. This section is intended to make thoroughly clear what is believed to be a consequence of the rest of this License. 8. If the distribution and/or use of the Program is restricted in certain countries either by patents or by copyrighted interfaces, the original copyright holder who places the Program under this License may add an explicit geographical distribution limitation excluding those countries, so that distribution is permitted only in or among countries not thus excluded. In such case, this License incorporates the limitation as if written in the body of this License. 9. The Free Software Foundation may publish revised and/or new versions of the General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies a version number of this License which applies to it and "any later version", you have the option of following the terms and conditions either of that version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of this License, you may choose any version ever published by the Free Software Foundation. 10. If you wish to incorporate parts of the Program into other free programs whose distribution conditions are different, write to the author to ask for permission. For software which is copyrighted by the Free Software Foundation, write to the Free Software Foundation; we sometimes make exceptions for this. Our decision will be guided by the two goals of preserving the free status of all derivatives of our free software and of promoting the sharing and reuse of software generally. NO WARRANTY 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. END OF TERMS AND CONDITIONS How to Apply These Terms to Your New Programs If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms. To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively convey the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. Copyright (C) This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. Also add information on how to contact you by electronic and paper mail. If the program is interactive, make it output a short notice like this when it starts in an interactive mode: Gnomovision version 69, Copyright (C) year name of author Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. This is free software, and you are welcome to redistribute it under certain conditions; type `show c' for details. The hypothetical commands `show w' and `show c' should show the appropriate parts of the General Public License. Of course, the commands you use may be called something other than `show w' and `show c'; they could even be mouse-clicks or menu items--whatever suits your program. You should also get your employer (if you work as a programmer) or your school, if any, to sign a "copyright disclaimer" for the program, if necessary. Here is a sample; alter the names: Yoyodyne, Inc., hereby disclaims all copyright interest in the program `Gnomovision' (which makes passes at compilers) written by James Hacker. , 1 April 1989 Ty Coon, President of Vice This General Public License does not permit incorporating your program into proprietary programs. If your program is a subroutine library, you may consider it more useful to permit linking proprietary applications with the library. If this is what you want to do, use the GNU Lesser General Public License instead of this License. squashfs4.3/INSTALL0000644000175000017500000000210512334225607014057 0ustar phillipphillip INSTALLING SQUASHFS The squashfs4.3.tar.gz file contains the squashfs-tools directory containing mksquashfs and unsquashfs. 1. Kernel support ----------------- This release is for 2.6.29 and newer kernels. Kernel patching is not necessary. Extended attribute support requires 2.6.35 or newer. File systems with extended attributes can be mounted on 2.6.29 and newer kernels (the extended attributes will be ignored with a warning). LZO compression support requires 2.6.36 or newer kernels. XZ compression support requires 2.6.38 or newer kernels. LZ4 support is not yet in any mainline kernel. 2. Building squashfs tools -------------------------- The squashfs-tools directory contains the mksquashfs and unsquashfs programs. These can be made by typing make (or make install to install in /usr/local/bin). By default the tools are built with GZIP compression and extended attribute support. Read the Makefile in squashfs-tools/ for instructions on building LZO, LZ4 and XZ compression support, and for instructions on disabling GZIP and extended attribute support if desired. squashfs4.3/PERFORMANCE.README0000644000175000017500000001461010406104324015440 0ustar phillipphillipGENERAL INFORMATION ON PERFORMANCE TESTS ---------------------------------------- The following performance tests were based on two file sets: the liveCD filesystem from the Ubuntu liveCD (Warty release), and the liveCD filesystem from the Damn Small Linux liveCD (release 0.8.4). The Ubuntu liveCD filesystem was used to test filesystem performance from CDROM and hard disk for Zisofs, Cloop, Squashfs 2.0 and Squashfs2.1. CRAMFS filesystem performance could not be tested for this filesystem bacause it exceeds the maximum supported size of CRAMFS. To test CRAMFS performance against Squashfs, the liveCD filesystem from Damn Small Linux was used. NOTE: the usual warnings apply to these results, they are provided for illustrative purposes only, and due to different hardware and/or file data, you may obtain different results. As such the results are provided "as is" without any warranty (either express or implied) and you assume all risks as to their quality and accuracy. 1. Ubuntu liveCD performance tests ext3 uncompressed size 1.4 GB Zisofs compressed size 589.81 MB Cloop compressed size 471.89 MB Squashfs2.0 compressed size 448.58 MB Squashfs2.1 compressed size 448.58 MB 1.1 Performance tests from CDROM 1.1.1 Directory Lookup performance Time taken to perform "ls -lR --color=alawys | cat > /dev/null" on filesystem mounted from CDROM Zisofs 49.88 seconds (User 2.60 secs, Sys 11.19 secs) Cloop 20.80 seconds (User 2.71 secs, Sys 13.50 secs) Squashfs2.0 16.56 seconds (User 2.42 secs, Sys 10.37 secs) Squashfs2.1 10.14 seconds (User 2.48 secs, Sys 4.44 secs) 1.1.2 Sequential I/O performance Time taken to perform "tar cf - | cat > /dev/null" on filesystem mounted from CDROM Zisofs 27 minutes 28.54 seconds (User 3.00 secs, Sys 1 min 4.80 secs) Cloop 5 minutes 55.72 seconds (User 2.90 secs, Sys 3 min 37.90 secs) Squashfs2.0 5 minutes 20.87 seconds (User 2.33 secs, Sys 56.98 secs) Squashfs2.1 5 minutes 15.46 seconds (user 2.28 secs, Sys 51.12 secs) 1.1.3 Random I/O performance Random access pattern generated by "find /mnt -type f -printf "%s %p\n" | sort -g | awk '{ printf $2 }' > /tmp/sort Time taken to perform "cpio -o --quiet -H newc < /tmp/sort > /dev/null" on filesystem mounted from CDROM Zisofs 101 minutes 29.65 seconds (User 5.33 secs, Sys 1 min 17.20 secs) Cloop 35 minutes 27.51 seconds (user 5.93 secs, Sys 4 mins 30.23 secs) Squashfs2.0 21 minutes 53.05 seconds (user 5.71 secs, Sys 2 mins 36.59 secs) Squashfs2.1 21 minutes 46.99 seconds (User 5.80 secs, Sys 2 mins 31.88 secs) 1.2 Performance tests from Hard disk 1.2.1 Directory Lookup performance Time taken to perform "ls -lR --color=alawys | cat > /dev/null" on filesystem mounted from Hard disk Zisofs 17.29 seconds (User 2.62 secs, Sys 11.08 secs) Cloop 16.46 seconds (User 2.63 secs, Sys 13.41 secs) Squashfs2.0 13.75 seconds (User 2.44 secs, Sys 11.00 secs) Squashfs2.1 6.94 seconds (User 2.44 secs, Sys 4.48 secs) 1.2.2 Sequential I/O performance Time taken to perform "tar cf - | cat > /dev/null" on filesystem mounted from Hard disk Zisofs 1 minute 21.47 seconds (User 2.73 secs, Sys 54.44 secs) Cloop 1 minute 34.06 seconds (user 2.85 secs, Sys 1 min 12.13 secs) Squashfs2.0 1 minute 21.22 seconds (User 2.42 secs, Sys 56.21 secs) Squashfs2.1 1 minute 15.46 seconds (User 2.36 secs, Sys 49.78 secs) 1.2.3 Random I/O performance Random access pattern generated by "find /mnt -type f -printf "%s %p\n" | sort -g | awk '{ printf $2 }' > /tmp/sort Time taken to perform "cpio -o --quiet -H newc < /tmp/sort > /dev/null" on filesystem mounted from Hard disk Zisofs 11 minutes 13.64 seconds (User 5.08 secs, Sys 52.62 secs) Cloop 5 minutes 37.93 seconds (user 6 secs, Sys 2 mins 22.38 secs) Squashfs2.0 5 minutes 7.11 seconds (user 5.63 secs, Sys 2 mins 35.23 secs) Squashfs2.1 5 minutes 1.87 seconds (User 5.71 secs, Sys 2 mins 29.98 secs) 2. Damn Small Linux liveCD performance tests ext3 uncompressed size 126 MB CRAMFS compressed size 52.19 MB Squashfs2.0 compressed size 46.52 MB Squashfs2.1 compressed size 46.52 MB 2.1 Performance tests from CDROM 2.1.1 Directory Lookup performance Time taken to perform "ls -lR --color=alawys | cat > /dev/null" on filesystem mounted from CDROM CRAMFS 10.85 seconds (User 0.39 secs, Sys 0.98 secs) Squashfs2.0 2.97 seconds (User 0.36 secs, Sys 2.15 secs) Squashfs2.1 2.43 seconds (User 0.40 secs, Sys 1.42 secs) 2.1.2 Sequential I/O performance Time taken to perform "tar cf - | cat > /dev/null" on filesystem mounted from CDROM CRAMFS 55.38 seconds (User 0.34 secs, Sys 6.98 secs) Squashfs2.0 35.99 seconds (User 0.30 secs, Sys 6.35 secs) Squashfs2.1 33.83 seconds (User 0.26 secs, Sys 5.56 secs) 2.1.3 Random I/O performance Random access pattern generated by "find /mnt -type f -printf "%s %p\n" | sort -g | awk '{ printf $2 }' > /tmp/sort Time taken to perform "cpio -o --quiet -H newc < /tmp/sort > /dev/null" on filesystem mounted from CDROM CRAMFS 3 minutes 1.68 seconds (User 0.54 secs, Sys 9.51 secs) Squashfs2.0 1 minute 39.45 seconds (User 0.57 secs, Sys 13.14 secs) Squashfs2.1 1 minute 38.41 seconds (User 0.58 secs, Sys 13.08 secs) 2.2 Performance tests from Hard disk 2.2.1 Directory Lookup performance Time taken to perform "ls -lR --color=alawys | cat > /dev/null" on filesystem mounted from Hard disk CRAMFS 1.77 seconds (User 0.53 secs, Sys 1.21 secs) Squashfs2.0 2.67 seconds (User 0.41 secs, Sys 2.25 secs) Squashfs2.1 1.87 seconds (User 0.41 secs, Sys 1.46 secs) 2.2.2 Sequential I/O performance Time taken to perform "tar cf - | cat > /dev/null" on filesystem mounted from Hard disk CRAMFS 6.80 seconds (User 0.36 secs, Sys 6.02 secs) Squashfs2.0 7.23 seconds (User 0.29 secs, Sys 6.62 secs) Squashfs2.1 6.53 seconds (User 0.31 secs, Sys 5.82 secs) 2.2.3 Random I/O performance Random access pattern generated by "find /mnt -type f -printf "%s %p\n" | sort -g | awk '{ printf $2 }' > /tmp/sort Time taken to perform "cpio -o --quiet -H newc < /tmp/sort > /dev/null" on filesystem mounted from Hard disk CRAMFS 28.55 seconds (User 0.49 secs, Sys 6.49 secs) Squashfs2.0 25.44 seconds (User 0.58 secs, Sys 13.17 secs) Squashfs2.1 24.72 seconds (User 0.56 secs, Sys 13.15 secs) squashfs4.3/squashfs-tools/0000755000175000017500000000000012334245623016023 5ustar phillipphillipsquashfs4.3/squashfs-tools/restore.c0000644000175000017500000001004412333330365017646 0ustar phillipphillip/* * Create a squashfs filesystem. This is a highly compressed read only * filesystem. * * Copyright (c) 2013, 2014 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * restore.c */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include "caches-queues-lists.h" #include "squashfs_fs.h" #include "mksquashfs.h" #include "error.h" #include "progressbar.h" #include "info.h" #define FALSE 0 #define TRUE 1 extern pthread_t reader_thread, writer_thread, main_thread; extern pthread_t *deflator_thread, *frag_deflator_thread, *frag_thread; extern struct queue *to_deflate, *to_writer, *to_frag, *to_process_frag; extern struct seq_queue *to_main; extern void restorefs(); extern int processors; static int interrupted = 0; static pthread_t restore_thread; void *restore_thrd(void *arg) { sigset_t sigmask, old_mask; int i, sig; sigemptyset(&sigmask); sigaddset(&sigmask, SIGINT); sigaddset(&sigmask, SIGTERM); sigaddset(&sigmask, SIGUSR1); pthread_sigmask(SIG_BLOCK, &sigmask, &old_mask); while(1) { sigwait(&sigmask, &sig); if((sig == SIGINT || sig == SIGTERM) && !interrupted) { ERROR("Interrupting will restore original " "filesystem!\n"); ERROR("Interrupt again to quit\n"); interrupted = TRUE; continue; } /* kill main thread/worker threads and restore */ set_progressbar_state(FALSE); disable_info(); /* first kill the reader thread */ pthread_cancel(reader_thread); pthread_join(reader_thread, NULL); /* * then flush the reader to deflator thread(s) output queue. * The deflator thread(s) will idle */ queue_flush(to_deflate); /* now kill the deflator thread(s) */ for(i = 0; i < processors; i++) pthread_cancel(deflator_thread[i]); for(i = 0; i < processors; i++) pthread_join(deflator_thread[i], NULL); /* * then flush the reader to process fragment thread(s) output * queue. The process fragment thread(s) will idle */ queue_flush(to_process_frag); /* now kill the process fragment thread(s) */ for(i = 0; i < processors; i++) pthread_cancel(frag_thread[i]); for(i = 0; i < processors; i++) pthread_join(frag_thread[i], NULL); /* * then flush the reader/deflator/process fragment to main * thread output queue. The main thread will idle */ seq_queue_flush(to_main); /* now kill the main thread */ pthread_cancel(main_thread); pthread_join(main_thread, NULL); /* then flush the main thread to fragment deflator thread(s) * queue. The fragment deflator thread(s) will idle */ queue_flush(to_frag); /* now kill the fragment deflator thread(s) */ for(i = 0; i < processors; i++) pthread_cancel(frag_deflator_thread[i]); for(i = 0; i < processors; i++) pthread_join(frag_deflator_thread[i], NULL); /* * then flush the main thread/fragment deflator thread(s) * to writer thread queue. The writer thread will idle */ queue_flush(to_writer); /* now kill the writer thread */ pthread_cancel(writer_thread); pthread_join(writer_thread, NULL); TRACE("All threads cancelled\n"); restorefs(); } } pthread_t *init_restore_thread() { pthread_create(&restore_thread, NULL, restore_thrd, NULL); return &restore_thread; } squashfs4.3/squashfs-tools/unsquashfs_info.h0000644000175000017500000000201412333330365021401 0ustar phillipphillip#ifndef UNSQUASHFS_INFO_H #define UNSQUASHFS_INFO_H /* * Create a squashfs filesystem. This is a highly compressed read only * filesystem. * * Copyright (c) 2013, 2014 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * unsquashfs_info.h */ extern void disable_info(); extern void update_info(char *); extern void init_info(); #endif squashfs4.3/squashfs-tools/lzma_xz_wrapper.c0000644000175000017500000000760712306776317021435 0ustar phillipphillip/* * Copyright (c) 2010, 2013 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * lzma_xz_wrapper.c * * Support for LZMA1 compression using XZ Utils liblzma http://tukaani.org/xz/ */ #include #include #include #include "squashfs_fs.h" #include "compressor.h" #define LZMA_PROPS_SIZE 5 #define LZMA_UNCOMP_SIZE 8 #define LZMA_HEADER_SIZE (LZMA_PROPS_SIZE + LZMA_UNCOMP_SIZE) #define LZMA_OPTIONS 5 #define MEMLIMIT (32 * 1024 * 1024) static int lzma_compress(void *dummy, void *dest, void *src, int size, int block_size, int *error) { unsigned char *d = (unsigned char *) dest; lzma_options_lzma opt; lzma_stream strm = LZMA_STREAM_INIT; int res; lzma_lzma_preset(&opt, LZMA_OPTIONS); opt.dict_size = block_size; res = lzma_alone_encoder(&strm, &opt); if(res != LZMA_OK) { lzma_end(&strm); goto failed; } strm.next_out = dest; strm.avail_out = block_size; strm.next_in = src; strm.avail_in = size; res = lzma_code(&strm, LZMA_FINISH); lzma_end(&strm); if(res == LZMA_STREAM_END) { /* * Fill in the 8 byte little endian uncompressed size field in * the LZMA header. 8 bytes is excessively large for squashfs * but this is the standard LZMA header and which is expected by * the kernel code */ d[LZMA_PROPS_SIZE] = size & 255; d[LZMA_PROPS_SIZE + 1] = (size >> 8) & 255; d[LZMA_PROPS_SIZE + 2] = (size >> 16) & 255; d[LZMA_PROPS_SIZE + 3] = (size >> 24) & 255; d[LZMA_PROPS_SIZE + 4] = 0; d[LZMA_PROPS_SIZE + 5] = 0; d[LZMA_PROPS_SIZE + 6] = 0; d[LZMA_PROPS_SIZE + 7] = 0; return (int) strm.total_out; } if(res == LZMA_OK) /* * Output buffer overflow. Return out of buffer space */ return 0; failed: /* * All other errors return failure, with the compressor * specific error code in *error */ *error = res; return -1; } static int lzma_uncompress(void *dest, void *src, int size, int outsize, int *error) { lzma_stream strm = LZMA_STREAM_INIT; int uncompressed_size = 0, res; unsigned char lzma_header[LZMA_HEADER_SIZE]; res = lzma_alone_decoder(&strm, MEMLIMIT); if(res != LZMA_OK) { lzma_end(&strm); goto failed; } memcpy(lzma_header, src, LZMA_HEADER_SIZE); uncompressed_size = lzma_header[LZMA_PROPS_SIZE] | (lzma_header[LZMA_PROPS_SIZE + 1] << 8) | (lzma_header[LZMA_PROPS_SIZE + 2] << 16) | (lzma_header[LZMA_PROPS_SIZE + 3] << 24); if(uncompressed_size > outsize) { res = 0; goto failed; } memset(lzma_header + LZMA_PROPS_SIZE, 255, LZMA_UNCOMP_SIZE); strm.next_out = dest; strm.avail_out = outsize; strm.next_in = lzma_header; strm.avail_in = LZMA_HEADER_SIZE; res = lzma_code(&strm, LZMA_RUN); if(res != LZMA_OK || strm.avail_in != 0) { lzma_end(&strm); goto failed; } strm.next_in = src + LZMA_HEADER_SIZE; strm.avail_in = size - LZMA_HEADER_SIZE; res = lzma_code(&strm, LZMA_FINISH); lzma_end(&strm); if(res == LZMA_STREAM_END || (res == LZMA_OK && strm.total_out >= uncompressed_size && strm.avail_in == 0)) return uncompressed_size; failed: *error = res; return -1; } struct compressor lzma_comp_ops = { .init = NULL, .compress = lzma_compress, .uncompress = lzma_uncompress, .options = NULL, .usage = NULL, .id = LZMA_COMPRESSION, .name = "lzma", .supported = 1 }; squashfs4.3/squashfs-tools/xattr.c0000644000175000017500000004463512333330365017342 0ustar phillipphillip/* * Create a squashfs filesystem. This is a highly compressed read only * filesystem. * * Copyright (c) 2008, 2009, 2010, 2012, 2014 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * xattr.c */ #define TRUE 1 #define FALSE 0 #include #include #include #include #include #include #include #include #include #include #include "squashfs_fs.h" #include "squashfs_swap.h" #include "mksquashfs.h" #include "xattr.h" #include "error.h" #include "progressbar.h" /* compressed xattr table */ static char *xattr_table = NULL; static unsigned int xattr_size = 0; /* cached uncompressed xattr data */ static char *data_cache = NULL; static int cache_bytes = 0, cache_size = 0; /* cached uncompressed xattr id table */ static struct squashfs_xattr_id *xattr_id_table = NULL; static int xattr_ids = 0; /* saved compressed xattr table */ unsigned int sxattr_bytes = 0, stotal_xattr_bytes = 0; /* saved cached uncompressed xattr data */ static char *sdata_cache = NULL; static int scache_bytes = 0; /* saved cached uncompressed xattr id table */ static int sxattr_ids = 0; /* xattr hash table for value duplicate detection */ static struct xattr_list *dupl_value[65536]; /* xattr hash table for id duplicate detection */ static struct dupl_id *dupl_id[65536]; /* file system globals from mksquashfs.c */ extern int no_xattrs, noX; extern long long bytes; extern int fd; extern unsigned int xattr_bytes, total_xattr_bytes; /* helper functions from mksquashfs.c */ extern unsigned short get_checksum(char *, int, unsigned short); extern void write_destination(int, long long, int, void *); extern long long generic_write_table(int, void *, int, void *, int); extern int mangle(char *, char *, int, int, int, int); extern char *pathname(struct dir_ent *); /* helper functions and definitions from read_xattrs.c */ extern int read_xattrs_from_disk(int, struct squashfs_super_block *); extern struct xattr_list *get_xattr(int, unsigned int *, int); extern struct prefix prefix_table[]; static int get_prefix(struct xattr_list *xattr, char *name) { int i; xattr->full_name = strdup(name); for(i = 0; prefix_table[i].type != -1; i++) { struct prefix *p = &prefix_table[i]; if(strncmp(xattr->full_name, p->prefix, strlen(p->prefix)) == 0) break; } if(prefix_table[i].type != -1) { xattr->name = xattr->full_name + strlen(prefix_table[i].prefix); xattr->size = strlen(xattr->name); } return prefix_table[i].type; } static int read_xattrs_from_system(char *filename, struct xattr_list **xattrs) { ssize_t size, vsize; char *xattr_names, *p; int i; struct xattr_list *xattr_list = NULL; while(1) { size = llistxattr(filename, NULL, 0); if(size <= 0) { if(size < 0 && errno != ENOTSUP) { ERROR_START("llistxattr for %s failed in " "read_attrs, because %s", filename, strerror(errno)); ERROR_EXIT(". Ignoring"); } return 0; } xattr_names = malloc(size); if(xattr_names == NULL) MEM_ERROR(); size = llistxattr(filename, xattr_names, size); if(size < 0) { free(xattr_names); if(errno == ERANGE) /* xattr list grew? Try again */ continue; else { ERROR_START("llistxattr for %s failed in " "read_attrs, because %s", filename, strerror(errno)); ERROR_EXIT(". Ignoring"); return 0; } } break; } for(i = 0, p = xattr_names; p < xattr_names + size; i++) { struct xattr_list *x = realloc(xattr_list, (i + 1) * sizeof(struct xattr_list)); if(x == NULL) MEM_ERROR(); xattr_list = x; xattr_list[i].type = get_prefix(&xattr_list[i], p); p += strlen(p) + 1; if(xattr_list[i].type == -1) { ERROR("Unrecognised xattr prefix %s\n", xattr_list[i].full_name); free(xattr_list[i].full_name); i--; continue; } while(1) { vsize = lgetxattr(filename, xattr_list[i].full_name, NULL, 0); if(vsize < 0) { ERROR_START("lgetxattr failed for %s in " "read_attrs, because %s", filename, strerror(errno)); ERROR_EXIT(". Ignoring"); free(xattr_list[i].full_name); goto failed; } xattr_list[i].value = malloc(vsize); if(xattr_list[i].value == NULL) MEM_ERROR(); vsize = lgetxattr(filename, xattr_list[i].full_name, xattr_list[i].value, vsize); if(vsize < 0) { free(xattr_list[i].value); if(errno == ERANGE) /* xattr grew? Try again */ continue; else { ERROR_START("lgetxattr failed for %s " "in read_attrs, because %s", filename, strerror(errno)); ERROR_EXIT(". Ignoring"); free(xattr_list[i].full_name); goto failed; } } break; } xattr_list[i].vsize = vsize; TRACE("read_xattrs_from_system: filename %s, xattr name %s," " vsize %d\n", filename, xattr_list[i].full_name, xattr_list[i].vsize); } free(xattr_names); *xattrs = xattr_list; return i; failed: while(--i >= 0) { free(xattr_list[i].full_name); free(xattr_list[i].value); } free(xattr_list); free(xattr_names); return 0; } static int get_xattr_size(struct xattr_list *xattr) { int size = sizeof(struct squashfs_xattr_entry) + sizeof(struct squashfs_xattr_val) + xattr->size; if(xattr->type & XATTR_VALUE_OOL) size += XATTR_VALUE_OOL_SIZE; else size += xattr->vsize; return size; } static void *get_xattr_space(unsigned int req_size, long long *disk) { int data_space; unsigned short c_byte; /* * Move and compress cached uncompressed data into xattr table. */ while(cache_bytes >= SQUASHFS_METADATA_SIZE) { if((xattr_size - xattr_bytes) < ((SQUASHFS_METADATA_SIZE << 1)) + 2) { xattr_table = realloc(xattr_table, xattr_size + (SQUASHFS_METADATA_SIZE << 1) + 2); if(xattr_table == NULL) MEM_ERROR(); xattr_size += (SQUASHFS_METADATA_SIZE << 1) + 2; } c_byte = mangle(xattr_table + xattr_bytes + BLOCK_OFFSET, data_cache, SQUASHFS_METADATA_SIZE, SQUASHFS_METADATA_SIZE, noX, 0); TRACE("Xattr block @ 0x%x, size %d\n", xattr_bytes, c_byte); SQUASHFS_SWAP_SHORTS(&c_byte, xattr_table + xattr_bytes, 1); xattr_bytes += SQUASHFS_COMPRESSED_SIZE(c_byte) + BLOCK_OFFSET; memmove(data_cache, data_cache + SQUASHFS_METADATA_SIZE, cache_bytes - SQUASHFS_METADATA_SIZE); cache_bytes -= SQUASHFS_METADATA_SIZE; } /* * Ensure there's enough space in the uncompressed data cache */ data_space = cache_size - cache_bytes; if(data_space < req_size) { int realloc_size = req_size - data_space; data_cache = realloc(data_cache, cache_size + realloc_size); if(data_cache == NULL) MEM_ERROR(); cache_size += realloc_size; } if(disk) *disk = ((long long) xattr_bytes << 16) | cache_bytes; cache_bytes += req_size; return data_cache + cache_bytes - req_size; } static struct dupl_id *check_id_dupl(struct xattr_list *xattr_list, int xattrs) { struct dupl_id *entry; int i; unsigned short checksum = 0; /* compute checksum over all xattrs */ for(i = 0; i < xattrs; i++) { struct xattr_list *xattr = &xattr_list[i]; checksum = get_checksum(xattr->full_name, strlen(xattr->full_name), checksum); checksum = get_checksum(xattr->value, xattr->vsize, checksum); } for(entry = dupl_id[checksum]; entry; entry = entry->next) { if (entry->xattrs != xattrs) continue; for(i = 0; i < xattrs; i++) { struct xattr_list *xattr = &xattr_list[i]; struct xattr_list *dup_xattr = &entry->xattr_list[i]; if(strcmp(xattr->full_name, dup_xattr->full_name)) break; if(memcmp(xattr->value, dup_xattr->value, xattr->vsize)) break; } if(i == xattrs) break; } if(entry == NULL) { /* no duplicate exists */ entry = malloc(sizeof(*entry)); if(entry == NULL) MEM_ERROR(); entry->xattrs = xattrs; entry->xattr_list = xattr_list; entry->xattr_id = SQUASHFS_INVALID_XATTR; entry->next = dupl_id[checksum]; dupl_id[checksum] = entry; } return entry; } static void check_value_dupl(struct xattr_list *xattr) { struct xattr_list *entry; if(xattr->vsize < XATTR_VALUE_OOL_SIZE) return; /* Check if this is a duplicate of an existing value */ xattr->vchecksum = get_checksum(xattr->value, xattr->vsize, 0); for(entry = dupl_value[xattr->vchecksum]; entry; entry = entry->vnext) { if(entry->vsize != xattr->vsize) continue; if(memcmp(entry->value, xattr->value, xattr->vsize) == 0) break; } if(entry == NULL) { /* * No duplicate exists, add to hash table, and mark as * requiring writing */ xattr->vnext = dupl_value[xattr->vchecksum]; dupl_value[xattr->vchecksum] = xattr; xattr->ool_value = SQUASHFS_INVALID_BLK; } else { /* * Duplicate exists, make type XATTR_VALUE_OOL, and * remember where the duplicate is */ xattr->type |= XATTR_VALUE_OOL; xattr->ool_value = entry->ool_value; /* on appending don't free duplicate values because the * duplicate value already points to the non-duplicate value */ if(xattr->value != entry->value) { free(xattr->value); xattr->value = entry->value; } } } static int get_xattr_id(int xattrs, struct xattr_list *xattr_list, long long xattr_disk, struct dupl_id *xattr_dupl) { int i, size = 0; struct squashfs_xattr_id *xattr_id; xattr_id_table = realloc(xattr_id_table, (xattr_ids + 1) * sizeof(struct squashfs_xattr_id)); if(xattr_id_table == NULL) MEM_ERROR(); /* get total uncompressed size of xattr data, needed for stat */ for(i = 0; i < xattrs; i++) size += strlen(xattr_list[i].full_name) + 1 + xattr_list[i].vsize; xattr_id = &xattr_id_table[xattr_ids]; xattr_id->xattr = xattr_disk; xattr_id->count = xattrs; xattr_id->size = size; /* * keep track of total uncompressed xattr data, needed for mksquashfs * file system summary */ total_xattr_bytes += size; xattr_dupl->xattr_id = xattr_ids ++; return xattr_dupl->xattr_id; } long long write_xattrs() { unsigned short c_byte; int i, avail_bytes; char *datap = data_cache; long long start_bytes = bytes; struct squashfs_xattr_table header; if(xattr_ids == 0) return SQUASHFS_INVALID_BLK; /* * Move and compress cached uncompressed data into xattr table. */ while(cache_bytes) { if((xattr_size - xattr_bytes) < ((SQUASHFS_METADATA_SIZE << 1)) + 2) { xattr_table = realloc(xattr_table, xattr_size + (SQUASHFS_METADATA_SIZE << 1) + 2); if(xattr_table == NULL) MEM_ERROR(); xattr_size += (SQUASHFS_METADATA_SIZE << 1) + 2; } avail_bytes = cache_bytes > SQUASHFS_METADATA_SIZE ? SQUASHFS_METADATA_SIZE : cache_bytes; c_byte = mangle(xattr_table + xattr_bytes + BLOCK_OFFSET, datap, avail_bytes, SQUASHFS_METADATA_SIZE, noX, 0); TRACE("Xattr block @ 0x%x, size %d\n", xattr_bytes, c_byte); SQUASHFS_SWAP_SHORTS(&c_byte, xattr_table + xattr_bytes, 1); xattr_bytes += SQUASHFS_COMPRESSED_SIZE(c_byte) + BLOCK_OFFSET; datap += avail_bytes; cache_bytes -= avail_bytes; } /* * Write compressed xattr table to file system */ write_destination(fd, bytes, xattr_bytes, xattr_table); bytes += xattr_bytes; /* * Swap if necessary the xattr id table */ for(i = 0; i < xattr_ids; i++) SQUASHFS_INSWAP_XATTR_ID(&xattr_id_table[i]); header.xattr_ids = xattr_ids; header.xattr_table_start = start_bytes; SQUASHFS_INSWAP_XATTR_TABLE(&header); return generic_write_table(xattr_ids * sizeof(struct squashfs_xattr_id), xattr_id_table, sizeof(header), &header, noX); } int generate_xattrs(int xattrs, struct xattr_list *xattr_list) { int total_size, i; int xattr_value_max; void *xp; long long xattr_disk; struct dupl_id *xattr_dupl; /* * check if the file xattrs are a complete duplicate of a pre-existing * id */ xattr_dupl = check_id_dupl(xattr_list, xattrs); if(xattr_dupl->xattr_id != SQUASHFS_INVALID_XATTR) return xattr_dupl->xattr_id; /* * Scan the xattr_list deciding which type to assign to each * xattr. The choice is fairly straightforward, and depends on the * size of each xattr name/value and the overall size of the * resultant xattr list stored in the xattr metadata table. * * Choices are whether to store data inline or out of line. * * The overall goal is to optimise xattr scanning and lookup, and * to enable the file system layout to scale from a couple of * small xattr name/values to a large number of large xattr * names/values without affecting performance. While hopefully * enabling the common case of a couple of small xattr name/values * to be stored efficiently * * Code repeatedly scans, doing the following * move xattr data out of line if it exceeds * xattr_value_max. Where xattr_value_max is * initially XATTR_INLINE_MAX. If the final uncompressed * xattr list is larger than XATTR_TARGET_MAX then more * aggressively move xattr data out of line by repeatedly * setting inline threshold to 1/2, then 1/4, 1/8 of * XATTR_INLINE_MAX until target achieved or there's * nothing left to move out of line */ xattr_value_max = XATTR_INLINE_MAX; while(1) { for(total_size = 0, i = 0; i < xattrs; i++) { struct xattr_list *xattr = &xattr_list[i]; xattr->type &= XATTR_PREFIX_MASK; /* all inline */ if (xattr->vsize > xattr_value_max) xattr->type |= XATTR_VALUE_OOL; total_size += get_xattr_size(xattr); } /* * If the total size of the uncompressed xattr list is <= * XATTR_TARGET_MAX we're done */ if(total_size <= XATTR_TARGET_MAX) break; if(xattr_value_max == XATTR_VALUE_OOL_SIZE) break; /* * Inline target not yet at minimum and so reduce it, and * try again */ xattr_value_max /= 2; if(xattr_value_max < XATTR_VALUE_OOL_SIZE) xattr_value_max = XATTR_VALUE_OOL_SIZE; } /* * Check xattr values for duplicates */ for(i = 0; i < xattrs; i++) { check_value_dupl(&xattr_list[i]); } /* * Add each out of line value to the file system xattr table * if it doesn't already exist as a duplicate */ for(i = 0; i < xattrs; i++) { struct xattr_list *xattr = &xattr_list[i]; if((xattr->type & XATTR_VALUE_OOL) && (xattr->ool_value == SQUASHFS_INVALID_BLK)) { struct squashfs_xattr_val val; int size = sizeof(val) + xattr->vsize; xp = get_xattr_space(size, &xattr->ool_value); val.vsize = xattr->vsize; SQUASHFS_SWAP_XATTR_VAL(&val, xp); memcpy(xp + sizeof(val), xattr->value, xattr->vsize); } } /* * Create xattr list and add to file system xattr table */ get_xattr_space(0, &xattr_disk); for(i = 0; i < xattrs; i++) { struct xattr_list *xattr = &xattr_list[i]; struct squashfs_xattr_entry entry; struct squashfs_xattr_val val; xp = get_xattr_space(sizeof(entry) + xattr->size, NULL); entry.type = xattr->type; entry.size = xattr->size; SQUASHFS_SWAP_XATTR_ENTRY(&entry, xp); memcpy(xp + sizeof(entry), xattr->name, xattr->size); if(xattr->type & XATTR_VALUE_OOL) { int size = sizeof(val) + XATTR_VALUE_OOL_SIZE; xp = get_xattr_space(size, NULL); val.vsize = XATTR_VALUE_OOL_SIZE; SQUASHFS_SWAP_XATTR_VAL(&val, xp); SQUASHFS_SWAP_LONG_LONGS(&xattr->ool_value, xp + sizeof(val), 1); } else { int size = sizeof(val) + xattr->vsize; xp = get_xattr_space(size, &xattr->ool_value); val.vsize = xattr->vsize; SQUASHFS_SWAP_XATTR_VAL(&val, xp); memcpy(xp + sizeof(val), xattr->value, xattr->vsize); } } /* * Add to xattr id lookup table */ return get_xattr_id(xattrs, xattr_list, xattr_disk, xattr_dupl); } int read_xattrs(void *d) { struct dir_ent *dir_ent = d; struct inode_info *inode = dir_ent->inode; char *filename = pathname(dir_ent); struct xattr_list *xattr_list; int xattrs; if(no_xattrs || IS_PSEUDO(inode) || inode->root_entry) return SQUASHFS_INVALID_XATTR; xattrs = read_xattrs_from_system(filename, &xattr_list); if(xattrs == 0) return SQUASHFS_INVALID_XATTR; return generate_xattrs(xattrs, xattr_list); } /* * Add the existing xattr ids and xattr metadata in the file system being * appended to, to the in-memory xattr cache. This allows duplicate checking to * take place against the xattrs already in the file system being appended to, * and ensures the pre-existing xattrs are written out along with any new xattrs */ int get_xattrs(int fd, struct squashfs_super_block *sBlk) { int ids, res, i, id; unsigned int count; TRACE("get_xattrs\n"); res = read_xattrs_from_disk(fd, sBlk); if(res == SQUASHFS_INVALID_BLK || res == 0) goto done; ids = res; /* * for each xattr id read and construct its list of xattr * name:value pairs, and add them to the in-memory xattr cache */ for(i = 0; i < ids; i++) { struct xattr_list *xattr_list = get_xattr(i, &count, 0); if(xattr_list == NULL) { res = 0; goto done; } id = generate_xattrs(count, xattr_list); /* * Sanity check, the new xattr id should be the same as the * xattr id in the original file system */ if(id != i) { ERROR("BUG, different xattr_id in get_xattrs\n"); res = 0; goto done; } } done: return res; } /* * Save current state of xattrs, needed for restoring state in the event of an * abort in appending */ void save_xattrs() { /* save the current state of the compressed xattr data */ sxattr_bytes = xattr_bytes; stotal_xattr_bytes = total_xattr_bytes; /* * save the current state of the cached uncompressed xattr data. * Note we have to save the contents of the data cache because future * operations will delete the current contents */ sdata_cache = malloc(cache_bytes); if(sdata_cache == NULL) MEM_ERROR(); memcpy(sdata_cache, data_cache, cache_bytes); scache_bytes = cache_bytes; /* save the current state of the xattr id table */ sxattr_ids = xattr_ids; } /* * Restore xattrs in the event of an abort in appending */ void restore_xattrs() { /* restore the state of the compressed xattr data */ xattr_bytes = sxattr_bytes; total_xattr_bytes = stotal_xattr_bytes; /* restore the state of the uncomoressed xattr data */ memcpy(data_cache, sdata_cache, scache_bytes); cache_bytes = scache_bytes; /* restore the state of the xattr id table */ xattr_ids = sxattr_ids; } squashfs4.3/squashfs-tools/unsquash-2.c0000644000175000017500000001737412306776317020221 0ustar phillipphillip/* * Unsquash a squashfs filesystem. This is a highly compressed read only * filesystem. * * Copyright (c) 2009, 2010, 2013 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * unsquash-2.c */ #include "unsquashfs.h" #include "squashfs_compat.h" static squashfs_fragment_entry_2 *fragment_table; void read_block_list_2(unsigned int *block_list, char *block_ptr, int blocks) { TRACE("read_block_list: blocks %d\n", blocks); if(swap) { unsigned int sblock_list[blocks]; memcpy(sblock_list, block_ptr, blocks * sizeof(unsigned int)); SQUASHFS_SWAP_INTS_3(block_list, sblock_list, blocks); } else memcpy(block_list, block_ptr, blocks * sizeof(unsigned int)); } int read_fragment_table_2(long long *directory_table_end) { int res, i; int bytes = SQUASHFS_FRAGMENT_BYTES_2(sBlk.s.fragments); int indexes = SQUASHFS_FRAGMENT_INDEXES_2(sBlk.s.fragments); unsigned int fragment_table_index[indexes]; TRACE("read_fragment_table: %d fragments, reading %d fragment indexes " "from 0x%llx\n", sBlk.s.fragments, indexes, sBlk.s.fragment_table_start); if(sBlk.s.fragments == 0) { *directory_table_end = sBlk.s.fragment_table_start; return TRUE; } fragment_table = malloc(bytes); if(fragment_table == NULL) EXIT_UNSQUASH("read_fragment_table: failed to allocate " "fragment table\n"); if(swap) { unsigned int sfragment_table_index[indexes]; res = read_fs_bytes(fd, sBlk.s.fragment_table_start, SQUASHFS_FRAGMENT_INDEX_BYTES_2(sBlk.s.fragments), sfragment_table_index); if(res == FALSE) { ERROR("read_fragment_table: failed to read fragment " "table index\n"); return FALSE; } SQUASHFS_SWAP_FRAGMENT_INDEXES_2(fragment_table_index, sfragment_table_index, indexes); } else { res = read_fs_bytes(fd, sBlk.s.fragment_table_start, SQUASHFS_FRAGMENT_INDEX_BYTES_2(sBlk.s.fragments), fragment_table_index); if(res == FALSE) { ERROR("read_fragment_table: failed to read fragment " "table index\n"); return FALSE; } } for(i = 0; i < indexes; i++) { int expected = (i + 1) != indexes ? SQUASHFS_METADATA_SIZE : bytes & (SQUASHFS_METADATA_SIZE - 1); int length = read_block(fd, fragment_table_index[i], NULL, expected, ((char *) fragment_table) + (i * SQUASHFS_METADATA_SIZE)); TRACE("Read fragment table block %d, from 0x%x, length %d\n", i, fragment_table_index[i], length); if(length == FALSE) { ERROR("read_fragment_table: failed to read fragment " "table block\n"); return FALSE; } } if(swap) { squashfs_fragment_entry_2 sfragment; for(i = 0; i < sBlk.s.fragments; i++) { SQUASHFS_SWAP_FRAGMENT_ENTRY_2((&sfragment), (&fragment_table[i])); memcpy((char *) &fragment_table[i], (char *) &sfragment, sizeof(squashfs_fragment_entry_2)); } } *directory_table_end = fragment_table_index[0]; return TRUE; } void read_fragment_2(unsigned int fragment, long long *start_block, int *size) { TRACE("read_fragment: reading fragment %d\n", fragment); squashfs_fragment_entry_2 *fragment_entry = &fragment_table[fragment]; *start_block = fragment_entry->start_block; *size = fragment_entry->size; } struct inode *read_inode_2(unsigned int start_block, unsigned int offset) { static union squashfs_inode_header_2 header; long long start = sBlk.s.inode_table_start + start_block; int bytes = lookup_entry(inode_table_hash, start); char *block_ptr = inode_table + bytes + offset; static struct inode i; TRACE("read_inode: reading inode [%d:%d]\n", start_block, offset); if(bytes == -1) EXIT_UNSQUASH("read_inode: inode table block %lld not found\n", start); if(swap) { squashfs_base_inode_header_2 sinode; memcpy(&sinode, block_ptr, sizeof(header.base)); SQUASHFS_SWAP_BASE_INODE_HEADER_2(&header.base, &sinode, sizeof(squashfs_base_inode_header_2)); } else memcpy(&header.base, block_ptr, sizeof(header.base)); i.xattr = SQUASHFS_INVALID_XATTR; i.uid = (uid_t) uid_table[header.base.uid]; i.gid = header.base.guid == SQUASHFS_GUIDS ? i.uid : (uid_t) guid_table[header.base.guid]; i.mode = lookup_type[header.base.inode_type] | header.base.mode; i.type = header.base.inode_type; i.time = sBlk.s.mkfs_time; i.inode_number = inode_number++; switch(header.base.inode_type) { case SQUASHFS_DIR_TYPE: { squashfs_dir_inode_header_2 *inode = &header.dir; if(swap) { squashfs_dir_inode_header_2 sinode; memcpy(&sinode, block_ptr, sizeof(header.dir)); SQUASHFS_SWAP_DIR_INODE_HEADER_2(&header.dir, &sinode); } else memcpy(&header.dir, block_ptr, sizeof(header.dir)); i.data = inode->file_size; i.offset = inode->offset; i.start = inode->start_block; i.time = inode->mtime; break; } case SQUASHFS_LDIR_TYPE: { squashfs_ldir_inode_header_2 *inode = &header.ldir; if(swap) { squashfs_ldir_inode_header_2 sinode; memcpy(&sinode, block_ptr, sizeof(header.ldir)); SQUASHFS_SWAP_LDIR_INODE_HEADER_2(&header.ldir, &sinode); } else memcpy(&header.ldir, block_ptr, sizeof(header.ldir)); i.data = inode->file_size; i.offset = inode->offset; i.start = inode->start_block; i.time = inode->mtime; break; } case SQUASHFS_FILE_TYPE: { squashfs_reg_inode_header_2 *inode = &header.reg; if(swap) { squashfs_reg_inode_header_2 sinode; memcpy(&sinode, block_ptr, sizeof(sinode)); SQUASHFS_SWAP_REG_INODE_HEADER_2(inode, &sinode); } else memcpy(inode, block_ptr, sizeof(*inode)); i.data = inode->file_size; i.time = inode->mtime; i.frag_bytes = inode->fragment == SQUASHFS_INVALID_FRAG ? 0 : inode->file_size % sBlk.s.block_size; i.fragment = inode->fragment; i.offset = inode->offset; i.blocks = inode->fragment == SQUASHFS_INVALID_FRAG ? (i.data + sBlk.s.block_size - 1) >> sBlk.s.block_log : i.data >> sBlk.s.block_log; i.start = inode->start_block; i.sparse = 0; i.block_ptr = block_ptr + sizeof(*inode); break; } case SQUASHFS_SYMLINK_TYPE: { squashfs_symlink_inode_header_2 *inodep = &header.symlink; if(swap) { squashfs_symlink_inode_header_2 sinodep; memcpy(&sinodep, block_ptr, sizeof(sinodep)); SQUASHFS_SWAP_SYMLINK_INODE_HEADER_2(inodep, &sinodep); } else memcpy(inodep, block_ptr, sizeof(*inodep)); i.symlink = malloc(inodep->symlink_size + 1); if(i.symlink == NULL) EXIT_UNSQUASH("read_inode: failed to malloc " "symlink data\n"); strncpy(i.symlink, block_ptr + sizeof(squashfs_symlink_inode_header_2), inodep->symlink_size); i.symlink[inodep->symlink_size] = '\0'; i.data = inodep->symlink_size; break; } case SQUASHFS_BLKDEV_TYPE: case SQUASHFS_CHRDEV_TYPE: { squashfs_dev_inode_header_2 *inodep = &header.dev; if(swap) { squashfs_dev_inode_header_2 sinodep; memcpy(&sinodep, block_ptr, sizeof(sinodep)); SQUASHFS_SWAP_DEV_INODE_HEADER_2(inodep, &sinodep); } else memcpy(inodep, block_ptr, sizeof(*inodep)); i.data = inodep->rdev; break; } case SQUASHFS_FIFO_TYPE: case SQUASHFS_SOCKET_TYPE: i.data = 0; break; default: EXIT_UNSQUASH("Unknown inode type %d in " "read_inode_header_2!\n", header.base.inode_type); } return &i; } squashfs4.3/squashfs-tools/squashfs_fs.h0000644000175000017500000003140312333330365020517 0ustar phillipphillip#ifndef SQUASHFS_FS #define SQUASHFS_FS /* * Squashfs * * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2012, * 2013, 2014 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * squashfs_fs.h */ #define SQUASHFS_CACHED_FRAGMENTS CONFIG_SQUASHFS_FRAGMENT_CACHE_SIZE #define SQUASHFS_MAJOR 4 #define SQUASHFS_MINOR 0 #define SQUASHFS_MAGIC 0x73717368 #define SQUASHFS_MAGIC_SWAP 0x68737173 #define SQUASHFS_START 0 /* size of metadata (inode and directory) blocks */ #define SQUASHFS_METADATA_SIZE 8192 #define SQUASHFS_METADATA_LOG 13 /* default size of data blocks */ #define SQUASHFS_FILE_SIZE 131072 #define SQUASHFS_FILE_MAX_SIZE 1048576 #define SQUASHFS_FILE_MAX_LOG 20 /* Max number of uids and gids */ #define SQUASHFS_IDS 65536 /* Max length of filename (not 255) */ #define SQUASHFS_NAME_LEN 256 #define SQUASHFS_INVALID ((long long) 0xffffffffffff) #define SQUASHFS_INVALID_FRAG ((unsigned int) 0xffffffff) #define SQUASHFS_INVALID_XATTR ((unsigned int) 0xffffffff) #define SQUASHFS_INVALID_BLK ((long long) -1) #define SQUASHFS_USED_BLK ((long long) -2) /* Filesystem flags */ #define SQUASHFS_NOI 0 #define SQUASHFS_NOD 1 #define SQUASHFS_CHECK 2 #define SQUASHFS_NOF 3 #define SQUASHFS_NO_FRAG 4 #define SQUASHFS_ALWAYS_FRAG 5 #define SQUASHFS_DUPLICATE 6 #define SQUASHFS_EXPORT 7 #define SQUASHFS_NOX 8 #define SQUASHFS_NO_XATTR 9 #define SQUASHFS_COMP_OPT 10 #define SQUASHFS_BIT(flag, bit) ((flag >> bit) & 1) #define SQUASHFS_UNCOMPRESSED_INODES(flags) SQUASHFS_BIT(flags, \ SQUASHFS_NOI) #define SQUASHFS_UNCOMPRESSED_DATA(flags) SQUASHFS_BIT(flags, \ SQUASHFS_NOD) #define SQUASHFS_UNCOMPRESSED_FRAGMENTS(flags) SQUASHFS_BIT(flags, \ SQUASHFS_NOF) #define SQUASHFS_NO_FRAGMENTS(flags) SQUASHFS_BIT(flags, \ SQUASHFS_NO_FRAG) #define SQUASHFS_ALWAYS_FRAGMENTS(flags) SQUASHFS_BIT(flags, \ SQUASHFS_ALWAYS_FRAG) #define SQUASHFS_DUPLICATES(flags) SQUASHFS_BIT(flags, \ SQUASHFS_DUPLICATE) #define SQUASHFS_EXPORTABLE(flags) SQUASHFS_BIT(flags, \ SQUASHFS_EXPORT) #define SQUASHFS_UNCOMPRESSED_XATTRS(flags) SQUASHFS_BIT(flags, \ SQUASHFS_NOX) #define SQUASHFS_NO_XATTRS(flags) SQUASHFS_BIT(flags, \ SQUASHFS_NO_XATTR) #define SQUASHFS_COMP_OPTS(flags) SQUASHFS_BIT(flags, \ SQUASHFS_COMP_OPT) #define SQUASHFS_MKFLAGS(noi, nod, nof, nox, no_frag, always_frag, \ duplicate_checking, exportable, no_xattr, comp_opt) (noi | \ (nod << 1) | (nof << 3) | (no_frag << 4) | \ (always_frag << 5) | (duplicate_checking << 6) | \ (exportable << 7) | (nox << 8) | (no_xattr << 9) | \ (comp_opt << 10)) /* Max number of types and file types */ #define SQUASHFS_DIR_TYPE 1 #define SQUASHFS_FILE_TYPE 2 #define SQUASHFS_SYMLINK_TYPE 3 #define SQUASHFS_BLKDEV_TYPE 4 #define SQUASHFS_CHRDEV_TYPE 5 #define SQUASHFS_FIFO_TYPE 6 #define SQUASHFS_SOCKET_TYPE 7 #define SQUASHFS_LDIR_TYPE 8 #define SQUASHFS_LREG_TYPE 9 #define SQUASHFS_LSYMLINK_TYPE 10 #define SQUASHFS_LBLKDEV_TYPE 11 #define SQUASHFS_LCHRDEV_TYPE 12 #define SQUASHFS_LFIFO_TYPE 13 #define SQUASHFS_LSOCKET_TYPE 14 /* Xattr types */ #define SQUASHFS_XATTR_USER 0 #define SQUASHFS_XATTR_TRUSTED 1 #define SQUASHFS_XATTR_SECURITY 2 #define SQUASHFS_XATTR_VALUE_OOL 256 #define SQUASHFS_XATTR_PREFIX_MASK 0xff /* Flag whether block is compressed or uncompressed, bit is set if block is * uncompressed */ #define SQUASHFS_COMPRESSED_BIT (1 << 15) #define SQUASHFS_COMPRESSED_SIZE(B) (((B) & ~SQUASHFS_COMPRESSED_BIT) ? \ (B) & ~SQUASHFS_COMPRESSED_BIT : SQUASHFS_COMPRESSED_BIT) #define SQUASHFS_COMPRESSED(B) (!((B) & SQUASHFS_COMPRESSED_BIT)) #define SQUASHFS_COMPRESSED_BIT_BLOCK (1 << 24) #define SQUASHFS_COMPRESSED_SIZE_BLOCK(B) ((B) & \ ~SQUASHFS_COMPRESSED_BIT_BLOCK) #define SQUASHFS_COMPRESSED_BLOCK(B) (!((B) & SQUASHFS_COMPRESSED_BIT_BLOCK)) /* * Inode number ops. Inodes consist of a compressed block number, and an * uncompressed offset within that block */ #define SQUASHFS_INODE_BLK(a) ((unsigned int) ((a) >> 16)) #define SQUASHFS_INODE_OFFSET(a) ((unsigned int) ((a) & 0xffff)) #define SQUASHFS_MKINODE(A, B) ((squashfs_inode)(((squashfs_inode) (A)\ << 16) + (B))) /* Compute 32 bit VFS inode number from squashfs inode number */ #define SQUASHFS_MK_VFS_INODE(a, b) ((unsigned int) (((a) << 8) + \ ((b) >> 2) + 1)) /* Translate between VFS mode and squashfs mode */ #define SQUASHFS_MODE(a) ((a) & 0xfff) /* fragment and fragment table defines */ #define SQUASHFS_FRAGMENT_BYTES(A) ((A) * \ sizeof(struct squashfs_fragment_entry)) #define SQUASHFS_FRAGMENT_INDEX(A) (SQUASHFS_FRAGMENT_BYTES(A) / \ SQUASHFS_METADATA_SIZE) #define SQUASHFS_FRAGMENT_INDEX_OFFSET(A) (SQUASHFS_FRAGMENT_BYTES(A) % \ SQUASHFS_METADATA_SIZE) #define SQUASHFS_FRAGMENT_INDEXES(A) ((SQUASHFS_FRAGMENT_BYTES(A) + \ SQUASHFS_METADATA_SIZE - 1) / \ SQUASHFS_METADATA_SIZE) #define SQUASHFS_FRAGMENT_INDEX_BYTES(A) (SQUASHFS_FRAGMENT_INDEXES(A) *\ sizeof(long long)) /* inode lookup table defines */ #define SQUASHFS_LOOKUP_BYTES(A) ((A) * sizeof(squashfs_inode)) #define SQUASHFS_LOOKUP_BLOCK(A) (SQUASHFS_LOOKUP_BYTES(A) / \ SQUASHFS_METADATA_SIZE) #define SQUASHFS_LOOKUP_BLOCK_OFFSET(A) (SQUASHFS_LOOKUP_BYTES(A) % \ SQUASHFS_METADATA_SIZE) #define SQUASHFS_LOOKUP_BLOCKS(A) ((SQUASHFS_LOOKUP_BYTES(A) + \ SQUASHFS_METADATA_SIZE - 1) / \ SQUASHFS_METADATA_SIZE) #define SQUASHFS_LOOKUP_BLOCK_BYTES(A) (SQUASHFS_LOOKUP_BLOCKS(A) *\ sizeof(long long)) /* uid lookup table defines */ #define SQUASHFS_ID_BYTES(A) ((A) * sizeof(unsigned int)) #define SQUASHFS_ID_BLOCK(A) (SQUASHFS_ID_BYTES(A) / \ SQUASHFS_METADATA_SIZE) #define SQUASHFS_ID_BLOCK_OFFSET(A) (SQUASHFS_ID_BYTES(A) % \ SQUASHFS_METADATA_SIZE) #define SQUASHFS_ID_BLOCKS(A) ((SQUASHFS_ID_BYTES(A) + \ SQUASHFS_METADATA_SIZE - 1) / \ SQUASHFS_METADATA_SIZE) #define SQUASHFS_ID_BLOCK_BYTES(A) (SQUASHFS_ID_BLOCKS(A) *\ sizeof(long long)) /* xattr id lookup table defines */ #define SQUASHFS_XATTR_BYTES(A) ((A) * sizeof(struct squashfs_xattr_id)) #define SQUASHFS_XATTR_BLOCK(A) (SQUASHFS_XATTR_BYTES(A) / \ SQUASHFS_METADATA_SIZE) #define SQUASHFS_XATTR_BLOCK_OFFSET(A) (SQUASHFS_XATTR_BYTES(A) % \ SQUASHFS_METADATA_SIZE) #define SQUASHFS_XATTR_BLOCKS(A) ((SQUASHFS_XATTR_BYTES(A) + \ SQUASHFS_METADATA_SIZE - 1) / \ SQUASHFS_METADATA_SIZE) #define SQUASHFS_XATTR_BLOCK_BYTES(A) (SQUASHFS_XATTR_BLOCKS(A) *\ sizeof(long long)) #define SQUASHFS_XATTR_BLK(A) ((unsigned int) ((A) >> 16)) #define SQUASHFS_XATTR_OFFSET(A) ((unsigned int) ((A) & 0xffff)) /* cached data constants for filesystem */ #define SQUASHFS_CACHED_BLKS 8 #define SQUASHFS_MAX_FILE_SIZE_LOG 64 #define SQUASHFS_MAX_FILE_SIZE ((long long) 1 << \ (SQUASHFS_MAX_FILE_SIZE_LOG - 2)) #define SQUASHFS_MARKER_BYTE 0xff /* meta index cache */ #define SQUASHFS_META_INDEXES (SQUASHFS_METADATA_SIZE / sizeof(unsigned int)) #define SQUASHFS_META_ENTRIES 31 #define SQUASHFS_META_NUMBER 8 #define SQUASHFS_SLOTS 4 struct meta_entry { long long data_block; unsigned int index_block; unsigned short offset; unsigned short pad; }; struct meta_index { unsigned int inode_number; unsigned int offset; unsigned short entries; unsigned short skip; unsigned short locked; unsigned short pad; struct meta_entry meta_entry[SQUASHFS_META_ENTRIES]; }; /* * definitions for structures on disk */ typedef long long squashfs_block; typedef long long squashfs_inode; #define ZLIB_COMPRESSION 1 #define LZMA_COMPRESSION 2 #define LZO_COMPRESSION 3 #define XZ_COMPRESSION 4 #define LZ4_COMPRESSION 5 struct squashfs_super_block { unsigned int s_magic; unsigned int inodes; int mkfs_time /* time of filesystem creation */; unsigned int block_size; unsigned int fragments; unsigned short compression; unsigned short block_log; unsigned short flags; unsigned short no_ids; unsigned short s_major; unsigned short s_minor; squashfs_inode root_inode; long long bytes_used; long long id_table_start; long long xattr_id_table_start; long long inode_table_start; long long directory_table_start; long long fragment_table_start; long long lookup_table_start; }; struct squashfs_dir_index { unsigned int index; unsigned int start_block; unsigned int size; unsigned char name[0]; }; struct squashfs_base_inode_header { unsigned short inode_type; unsigned short mode; unsigned short uid; unsigned short guid; int mtime; unsigned int inode_number; }; struct squashfs_ipc_inode_header { unsigned short inode_type; unsigned short mode; unsigned short uid; unsigned short guid; int mtime; unsigned int inode_number; unsigned int nlink; }; struct squashfs_lipc_inode_header { unsigned short inode_type; unsigned short mode; unsigned short uid; unsigned short guid; int mtime; unsigned int inode_number; unsigned int nlink; unsigned int xattr; }; struct squashfs_dev_inode_header { unsigned short inode_type; unsigned short mode; unsigned short uid; unsigned short guid; int mtime; unsigned int inode_number; unsigned int nlink; unsigned int rdev; }; struct squashfs_ldev_inode_header { unsigned short inode_type; unsigned short mode; unsigned short uid; unsigned short guid; int mtime; unsigned int inode_number; unsigned int nlink; unsigned int rdev; unsigned int xattr; }; struct squashfs_symlink_inode_header { unsigned short inode_type; unsigned short mode; unsigned short uid; unsigned short guid; int mtime; unsigned int inode_number; unsigned int nlink; unsigned int symlink_size; char symlink[0]; }; struct squashfs_reg_inode_header { unsigned short inode_type; unsigned short mode; unsigned short uid; unsigned short guid; int mtime; unsigned int inode_number; unsigned int start_block; unsigned int fragment; unsigned int offset; unsigned int file_size; unsigned int block_list[0]; }; struct squashfs_lreg_inode_header { unsigned short inode_type; unsigned short mode; unsigned short uid; unsigned short guid; int mtime; unsigned int inode_number; squashfs_block start_block; long long file_size; long long sparse; unsigned int nlink; unsigned int fragment; unsigned int offset; unsigned int xattr; unsigned int block_list[0]; }; struct squashfs_dir_inode_header { unsigned short inode_type; unsigned short mode; unsigned short uid; unsigned short guid; int mtime; unsigned int inode_number; unsigned int start_block; unsigned int nlink; unsigned short file_size; unsigned short offset; unsigned int parent_inode; }; struct squashfs_ldir_inode_header { unsigned short inode_type; unsigned short mode; unsigned short uid; unsigned short guid; int mtime; unsigned int inode_number; unsigned int nlink; unsigned int file_size; unsigned int start_block; unsigned int parent_inode; unsigned short i_count; unsigned short offset; unsigned int xattr; struct squashfs_dir_index index[0]; }; union squashfs_inode_header { struct squashfs_base_inode_header base; struct squashfs_dev_inode_header dev; struct squashfs_ldev_inode_header ldev; struct squashfs_symlink_inode_header symlink; struct squashfs_reg_inode_header reg; struct squashfs_lreg_inode_header lreg; struct squashfs_dir_inode_header dir; struct squashfs_ldir_inode_header ldir; struct squashfs_ipc_inode_header ipc; struct squashfs_lipc_inode_header lipc; }; struct squashfs_dir_entry { unsigned short offset; short inode_number; unsigned short type; unsigned short size; char name[0]; }; struct squashfs_dir_header { unsigned int count; unsigned int start_block; unsigned int inode_number; }; struct squashfs_fragment_entry { long long start_block; unsigned int size; unsigned int unused; }; struct squashfs_xattr_entry { unsigned short type; unsigned short size; }; struct squashfs_xattr_val { unsigned int vsize; }; struct squashfs_xattr_id { long long xattr; unsigned int count; unsigned int size; }; struct squashfs_xattr_table { long long xattr_table_start; unsigned int xattr_ids; unsigned int unused; }; #endif squashfs4.3/squashfs-tools/xattr.h0000644000175000017500000000705112333330365017336 0ustar phillipphillip#ifndef XATTR_H #define XATTR_H /* * Create a squashfs filesystem. This is a highly compressed read only * filesystem. * * Copyright (c) 2010, 2012, 2013, 2014 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * xattr.h */ #define XATTR_VALUE_OOL SQUASHFS_XATTR_VALUE_OOL #define XATTR_PREFIX_MASK SQUASHFS_XATTR_PREFIX_MASK #define XATTR_VALUE_OOL_SIZE sizeof(long long) /* maximum size of xattr value data that will be inlined */ #define XATTR_INLINE_MAX 128 /* the target size of an inode's xattr name:value list. If it * exceeds this, then xattr value data will be successively out of lined * until it meets the target */ #define XATTR_TARGET_MAX 65536 #define IS_XATTR(a) (a != SQUASHFS_INVALID_XATTR) struct xattr_list { char *name; char *full_name; int size; int vsize; void *value; int type; long long ool_value; unsigned short vchecksum; struct xattr_list *vnext; }; struct dupl_id { struct xattr_list *xattr_list; int xattrs; int xattr_id; struct dupl_id *next; }; struct prefix { char *prefix; int type; }; extern int generate_xattrs(int, struct xattr_list *); #ifdef XATTR_SUPPORT extern int get_xattrs(int, struct squashfs_super_block *); extern int read_xattrs(void *); extern long long write_xattrs(); extern void save_xattrs(); extern void restore_xattrs(); extern unsigned int xattr_bytes, total_xattr_bytes; extern void write_xattr(char *, unsigned int); extern int read_xattrs_from_disk(int, struct squashfs_super_block *); extern struct xattr_list *get_xattr(int, unsigned int *, int); extern void free_xattr(struct xattr_list *, int); #else static inline int get_xattrs(int fd, struct squashfs_super_block *sBlk) { if(sBlk->xattr_id_table_start != SQUASHFS_INVALID_BLK) { fprintf(stderr, "Xattrs in filesystem! These are not " "supported on this version of Squashfs\n"); return 0; } else return SQUASHFS_INVALID_BLK; } static inline int read_xattrs(void *dir_ent) { return SQUASHFS_INVALID_XATTR; } static inline long long write_xattrs() { return SQUASHFS_INVALID_BLK; } static inline void save_xattrs() { } static inline void restore_xattrs() { } static inline void write_xattr(char *pathname, unsigned int xattr) { } static inline int read_xattrs_from_disk(int fd, struct squashfs_super_block *sBlk) { if(sBlk->xattr_id_table_start != SQUASHFS_INVALID_BLK) { fprintf(stderr, "Xattrs in filesystem! These are not " "supported on this version of Squashfs\n"); return 0; } else return SQUASHFS_INVALID_BLK; } static inline struct xattr_list *get_xattr(int i, unsigned int *count, int j) { return NULL; } #endif #ifdef XATTR_SUPPORT #ifdef XATTR_DEFAULT #define NOXOPT_STR #define XOPT_STR " (default)" #define XATTR_DEF 0 #else #define NOXOPT_STR " (default)" #define XOPT_STR #define XATTR_DEF 1 #endif #else #define NOXOPT_STR " (default)" #define XOPT_STR " (unsupported)" #define XATTR_DEF 1 #endif #endif squashfs4.3/squashfs-tools/xz_wrapper.h0000644000175000017500000000313112306776317020403 0ustar phillipphillip#ifndef XZ_WRAPPER_H #define XZ_WRAPPER_H /* * Squashfs * * Copyright (c) 2010 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * xz_wrapper.h * */ #ifndef linux #define __BYTE_ORDER BYTE_ORDER #define __BIG_ENDIAN BIG_ENDIAN #define __LITTLE_ENDIAN LITTLE_ENDIAN #else #include #endif #if __BYTE_ORDER == __BIG_ENDIAN extern unsigned int inswap_le32(unsigned int); #define SQUASHFS_INSWAP_COMP_OPTS(s) { \ (s)->dictionary_size = inswap_le32((s)->dictionary_size); \ (s)->flags = inswap_le32((s)->flags); \ } #else #define SQUASHFS_INSWAP_COMP_OPTS(s) #endif #define MEMLIMIT (32 * 1024 * 1024) struct bcj { char *name; lzma_vli id; int selected; }; struct filter { void *buffer; lzma_filter filter[3]; size_t length; }; struct xz_stream { struct filter *filter; int filters; int dictionary_size; lzma_options_lzma opt; }; struct comp_opts { int dictionary_size; int flags; }; #endif squashfs4.3/squashfs-tools/caches-queues-lists.h0000644000175000017500000001276112333330365022067 0ustar phillipphillip#ifndef CACHES_QUEUES_LISTS_H #define CACHES_QUEUES_LISTS_H /* * Create a squashfs filesystem. This is a highly compressed read only * filesystem. * * Copyright (c) 2013, 2014 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * caches-queues-lists.h */ #define INSERT_LIST(NAME, TYPE) \ void insert_##NAME##_list(TYPE **list, TYPE *entry) { \ if(*list) { \ entry->NAME##_next = *list; \ entry->NAME##_prev = (*list)->NAME##_prev; \ (*list)->NAME##_prev->NAME##_next = entry; \ (*list)->NAME##_prev = entry; \ } else { \ *list = entry; \ entry->NAME##_prev = entry->NAME##_next = entry; \ } \ } #define REMOVE_LIST(NAME, TYPE) \ void remove_##NAME##_list(TYPE **list, TYPE *entry) { \ if(entry->NAME##_prev == entry && entry->NAME##_next == entry) { \ /* only this entry in the list */ \ *list = NULL; \ } else if(entry->NAME##_prev != NULL && entry->NAME##_next != NULL) { \ /* more than one entry in the list */ \ entry->NAME##_next->NAME##_prev = entry->NAME##_prev; \ entry->NAME##_prev->NAME##_next = entry->NAME##_next; \ if(*list == entry) \ *list = entry->NAME##_next; \ } \ entry->NAME##_prev = entry->NAME##_next = NULL; \ } #define INSERT_HASH_TABLE(NAME, TYPE, HASH_FUNCTION, FIELD, LINK) \ void insert_##NAME##_hash_table(TYPE *container, struct file_buffer *entry) \ { \ int hash = HASH_FUNCTION(entry->FIELD); \ \ entry->LINK##_next = container->hash_table[hash]; \ container->hash_table[hash] = entry; \ entry->LINK##_prev = NULL; \ if(entry->LINK##_next) \ entry->LINK##_next->LINK##_prev = entry; \ } #define REMOVE_HASH_TABLE(NAME, TYPE, HASH_FUNCTION, FIELD, LINK) \ void remove_##NAME##_hash_table(TYPE *container, struct file_buffer *entry) \ { \ if(entry->LINK##_prev) \ entry->LINK##_prev->LINK##_next = entry->LINK##_next; \ else \ container->hash_table[HASH_FUNCTION(entry->FIELD)] = \ entry->LINK##_next; \ if(entry->LINK##_next) \ entry->LINK##_next->LINK##_prev = entry->LINK##_prev; \ \ entry->LINK##_prev = entry->LINK##_next = NULL; \ } #define HASH_SIZE 65536 #define CALCULATE_HASH(n) ((n) & 0xffff) /* struct describing a cache entry passed between threads */ struct file_buffer { union { long long index; long long sequence; }; long long file_size; union { long long block; unsigned short checksum; }; struct cache *cache; union { struct file_info *dupl_start; struct file_buffer *hash_next; }; union { int duplicate; struct file_buffer *hash_prev; }; union { struct { struct file_buffer *free_next; struct file_buffer *free_prev; }; struct { struct file_buffer *seq_next; struct file_buffer *seq_prev; }; }; int size; int c_byte; char used; char fragment; char error; char locked; char wait_on_unlock; char noD; char data[0]; }; /* struct describing queues used to pass data between threads */ struct queue { int size; int readp; int writep; pthread_mutex_t mutex; pthread_cond_t empty; pthread_cond_t full; void **data; }; /* * struct describing seq_queues used to pass data between the read * thread and the deflate and main threads */ struct seq_queue { int fragment_count; int block_count; struct file_buffer *hash_table[HASH_SIZE]; pthread_mutex_t mutex; pthread_cond_t wait; }; /* Cache status struct. Caches are used to keep track of memory buffers passed between different threads */ struct cache { int max_buffers; int count; int buffer_size; int noshrink_lookup; int first_freelist; union { int used; int max_count; }; pthread_mutex_t mutex; pthread_cond_t wait_for_free; pthread_cond_t wait_for_unlock; struct file_buffer *free_list; struct file_buffer *hash_table[HASH_SIZE]; }; extern struct queue *queue_init(int); extern void queue_put(struct queue *, void *); extern void *queue_get(struct queue *); extern int queue_empty(struct queue *); extern void queue_flush(struct queue *); extern void dump_queue(struct queue *); extern struct seq_queue *seq_queue_init(); extern void seq_queue_put(struct seq_queue *, struct file_buffer *); extern void dump_seq_queue(struct seq_queue *, int); extern struct file_buffer *seq_queue_get(struct seq_queue *); extern void seq_queue_flush(struct seq_queue *); extern struct cache *cache_init(int, int, int, int); extern struct file_buffer *cache_lookup(struct cache *, long long); extern struct file_buffer *cache_get(struct cache *, long long); extern struct file_buffer *cache_get_nohash(struct cache *); extern void cache_hash(struct file_buffer *, long long); extern void cache_block_put(struct file_buffer *); extern void dump_cache(struct cache *); extern struct file_buffer *cache_get_nowait(struct cache *, long long); extern struct file_buffer *cache_lookup_nowait(struct cache *, long long, char *); extern void cache_wait_unlock(struct file_buffer *); extern void cache_unlock(struct file_buffer *); extern int first_freelist; #endif squashfs4.3/squashfs-tools/gzip_wrapper.c0000644000175000017500000003112412334035501020671 0ustar phillipphillip/* * Copyright (c) 2009, 2010, 2013, 2014 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * gzip_wrapper.c * * Support for ZLIB compression http://www.zlib.net */ #include #include #include #include #include "squashfs_fs.h" #include "gzip_wrapper.h" #include "compressor.h" static struct strategy strategy[] = { { "default", Z_DEFAULT_STRATEGY, 0 }, { "filtered", Z_FILTERED, 0 }, { "huffman_only", Z_HUFFMAN_ONLY, 0 }, { "run_length_encoded", Z_RLE, 0 }, { "fixed", Z_FIXED, 0 }, { NULL, 0, 0 } }; static int strategy_count = 0; /* default compression level */ static int compression_level = GZIP_DEFAULT_COMPRESSION_LEVEL; /* default window size */ static int window_size = GZIP_DEFAULT_WINDOW_SIZE; /* * This function is called by the options parsing code in mksquashfs.c * to parse any -X compressor option. * * This function returns: * >=0 (number of additional args parsed) on success * -1 if the option was unrecognised, or * -2 if the option was recognised, but otherwise bad in * some way (e.g. invalid parameter) * * Note: this function sets internal compressor state, but does not * pass back the results of the parsing other than success/failure. * The gzip_dump_options() function is called later to get the options in * a format suitable for writing to the filesystem. */ static int gzip_options(char *argv[], int argc) { if(strcmp(argv[0], "-Xcompression-level") == 0) { if(argc < 2) { fprintf(stderr, "gzip: -Xcompression-level missing " "compression level\n"); fprintf(stderr, "gzip: -Xcompression-level it " "should be 1 >= n <= 9\n"); goto failed; } compression_level = atoi(argv[1]); if(compression_level < 1 || compression_level > 9) { fprintf(stderr, "gzip: -Xcompression-level invalid, it " "should be 1 >= n <= 9\n"); goto failed; } return 1; } else if(strcmp(argv[0], "-Xwindow-size") == 0) { if(argc < 2) { fprintf(stderr, "gzip: -Xwindow-size missing window " " size\n"); fprintf(stderr, "gzip: -Xwindow-size \n"); goto failed; } window_size = atoi(argv[1]); if(window_size < 8 || window_size > 15) { fprintf(stderr, "gzip: -Xwindow-size invalid, it " "should be 8 >= n <= 15\n"); goto failed; } return 1; } else if(strcmp(argv[0], "-Xstrategy") == 0) { char *name; int i; if(argc < 2) { fprintf(stderr, "gzip: -Xstrategy missing " "strategies\n"); goto failed; } name = argv[1]; while(name[0] != '\0') { for(i = 0; strategy[i].name; i++) { int n = strlen(strategy[i].name); if((strncmp(name, strategy[i].name, n) == 0) && (name[n] == '\0' || name[n] == ',')) { if(strategy[i].selected == 0) { strategy[i].selected = 1; strategy_count++; } name += name[n] == ',' ? n + 1 : n; break; } } if(strategy[i].name == NULL) { fprintf(stderr, "gzip: -Xstrategy unrecognised " "strategy\n"); goto failed; } } return 1; } return -1; failed: return -2; } /* * This function is called after all options have been parsed. * It is used to do post-processing on the compressor options using * values that were not expected to be known at option parse time. * * This function returns 0 on successful post processing, or * -1 on error */ static int gzip_options_post(int block_size) { if(strategy_count == 1 && strategy[0].selected) { strategy_count = 0; strategy[0].selected = 0; } return 0; } /* * This function is called by mksquashfs to dump the parsed * compressor options in a format suitable for writing to the * compressor options field in the filesystem (stored immediately * after the superblock). * * This function returns a pointer to the compression options structure * to be stored (and the size), or NULL if there are no compression * options * */ static void *gzip_dump_options(int block_size, int *size) { static struct gzip_comp_opts comp_opts; int i, strategies = 0; /* * If default compression options of: * compression-level: 8 and * window-size: 15 and * strategy_count == 0 then * don't store a compression options structure (this is compatible * with the legacy implementation of GZIP for Squashfs) */ if(compression_level == GZIP_DEFAULT_COMPRESSION_LEVEL && window_size == GZIP_DEFAULT_WINDOW_SIZE && strategy_count == 0) return NULL; for(i = 0; strategy[i].name; i++) strategies |= strategy[i].selected << i; comp_opts.compression_level = compression_level; comp_opts.window_size = window_size; comp_opts.strategy = strategies; SQUASHFS_INSWAP_COMP_OPTS(&comp_opts); *size = sizeof(comp_opts); return &comp_opts; } /* * This function is a helper specifically for the append mode of * mksquashfs. Its purpose is to set the internal compressor state * to the stored compressor options in the passed compressor options * structure. * * In effect this function sets up the compressor options * to the same state they were when the filesystem was originally * generated, this is to ensure on appending, the compressor uses * the same compression options that were used to generate the * original filesystem. * * Note, even if there are no compressor options, this function is still * called with an empty compressor structure (size == 0), to explicitly * set the default options, this is to ensure any user supplied * -X options on the appending mksquashfs command line are over-ridden * * This function returns 0 on sucessful extraction of options, and * -1 on error */ static int gzip_extract_options(int block_size, void *buffer, int size) { struct gzip_comp_opts *comp_opts = buffer; int i; if(size == 0) { /* Set default values */ compression_level = GZIP_DEFAULT_COMPRESSION_LEVEL; window_size = GZIP_DEFAULT_WINDOW_SIZE; strategy_count = 0; return 0; } /* we expect a comp_opts structure of sufficient size to be present */ if(size < sizeof(*comp_opts)) goto failed; SQUASHFS_INSWAP_COMP_OPTS(comp_opts); /* Check comp_opts structure for correctness */ if(comp_opts->compression_level < 1 || comp_opts->compression_level > 9) { fprintf(stderr, "gzip: bad compression level in " "compression options structure\n"); goto failed; } compression_level = comp_opts->compression_level; if(comp_opts->window_size < 8 || comp_opts->window_size > 15) { fprintf(stderr, "gzip: bad window size in " "compression options structure\n"); goto failed; } window_size = comp_opts->window_size; strategy_count = 0; for(i = 0; strategy[i].name; i++) { if((comp_opts->strategy >> i) & 1) { strategy[i].selected = 1; strategy_count ++; } else strategy[i].selected = 0; } return 0; failed: fprintf(stderr, "gzip: error reading stored compressor options from " "filesystem!\n"); return -1; } void gzip_display_options(void *buffer, int size) { struct gzip_comp_opts *comp_opts = buffer; int i, printed; /* we expect a comp_opts structure of sufficient size to be present */ if(size < sizeof(*comp_opts)) goto failed; SQUASHFS_INSWAP_COMP_OPTS(comp_opts); /* Check comp_opts structure for correctness */ if(comp_opts->compression_level < 1 || comp_opts->compression_level > 9) { fprintf(stderr, "gzip: bad compression level in " "compression options structure\n"); goto failed; } printf("\tcompression-level %d\n", comp_opts->compression_level); if(comp_opts->window_size < 8 || comp_opts->window_size > 15) { fprintf(stderr, "gzip: bad window size in " "compression options structure\n"); goto failed; } printf("\twindow-size %d\n", comp_opts->window_size); for(i = 0, printed = 0; strategy[i].name; i++) { if((comp_opts->strategy >> i) & 1) { if(printed) printf(", "); else printf("\tStrategies selected: "); printf("%s", strategy[i].name); printed = 1; } } if(!printed) printf("\tStrategies selected: default\n"); else printf("\n"); return; failed: fprintf(stderr, "gzip: error reading stored compressor options from " "filesystem!\n"); } /* * This function is called by mksquashfs to initialise the * compressor, before compress() is called. * * This function returns 0 on success, and * -1 on error */ static int gzip_init(void **strm, int block_size, int datablock) { int i, j, res; struct gzip_stream *stream; if(!datablock || !strategy_count) { stream = malloc(sizeof(*stream) + sizeof(struct gzip_strategy)); if(stream == NULL) goto failed; stream->strategies = 1; stream->strategy[0].strategy = Z_DEFAULT_STRATEGY; } else { stream = malloc(sizeof(*stream) + sizeof(struct gzip_strategy) * strategy_count); if(stream == NULL) goto failed; memset(stream->strategy, 0, sizeof(struct gzip_strategy) * strategy_count); stream->strategies = strategy_count; for(i = 0, j = 0; strategy[i].name; i++) { if(!strategy[i].selected) continue; stream->strategy[j].strategy = strategy[i].strategy; if(j) { stream->strategy[j].buffer = malloc(block_size); if(stream->strategy[j].buffer == NULL) goto failed2; } j++; } } stream->stream.zalloc = Z_NULL; stream->stream.zfree = Z_NULL; stream->stream.opaque = 0; res = deflateInit2(&stream->stream, compression_level, Z_DEFLATED, window_size, 8, stream->strategy[0].strategy); if(res != Z_OK) goto failed2; *strm = stream; return 0; failed2: for(i = 1; i < stream->strategies; i++) free(stream->strategy[i].buffer); free(stream); failed: return -1; } static int gzip_compress(void *strm, void *d, void *s, int size, int block_size, int *error) { int i, res; struct gzip_stream *stream = strm; struct gzip_strategy *selected = NULL; stream->strategy[0].buffer = d; for(i = 0; i < stream->strategies; i++) { struct gzip_strategy *strategy = &stream->strategy[i]; res = deflateReset(&stream->stream); if(res != Z_OK) goto failed; stream->stream.next_in = s; stream->stream.avail_in = size; stream->stream.next_out = strategy->buffer; stream->stream.avail_out = block_size; if(stream->strategies > 1) { res = deflateParams(&stream->stream, compression_level, strategy->strategy); if(res != Z_OK) goto failed; } res = deflate(&stream->stream, Z_FINISH); strategy->length = stream->stream.total_out; if(res == Z_STREAM_END) { if(!selected || selected->length > strategy->length) selected = strategy; } else if(res != Z_OK) goto failed; } if(!selected) /* * Output buffer overflow. Return out of buffer space */ return 0; if(selected->buffer != d) memcpy(d, selected->buffer, selected->length); return (int) selected->length; failed: /* * All other errors return failure, with the compressor * specific error code in *error */ *error = res; return -1; } static int gzip_uncompress(void *d, void *s, int size, int outsize, int *error) { int res; unsigned long bytes = outsize; res = uncompress(d, &bytes, s, size); if(res == Z_OK) return (int) bytes; else { *error = res; return -1; } } void gzip_usage() { fprintf(stderr, "\t -Xcompression-level \n"); fprintf(stderr, "\t\t should be 1 .. 9 (default " "%d)\n", GZIP_DEFAULT_COMPRESSION_LEVEL); fprintf(stderr, "\t -Xwindow-size \n"); fprintf(stderr, "\t\t should be 8 .. 15 (default " "%d)\n", GZIP_DEFAULT_WINDOW_SIZE); fprintf(stderr, "\t -Xstrategy strategy1,strategy2,...,strategyN\n"); fprintf(stderr, "\t\tCompress using strategy1,strategy2,...,strategyN" " in turn\n"); fprintf(stderr, "\t\tand choose the best compression.\n"); fprintf(stderr, "\t\tAvailable strategies: default, filtered, " "huffman_only,\n\t\trun_length_encoded and fixed\n"); } struct compressor gzip_comp_ops = { .init = gzip_init, .compress = gzip_compress, .uncompress = gzip_uncompress, .options = gzip_options, .options_post = gzip_options_post, .dump_options = gzip_dump_options, .extract_options = gzip_extract_options, .display_options = gzip_display_options, .usage = gzip_usage, .id = ZLIB_COMPRESSION, .name = "gzip", .supported = 1 }; squashfs4.3/squashfs-tools/compressor.c0000644000175000017500000000545312306776316020401 0ustar phillipphillip/* * * Copyright (c) 2009, 2010, 2011 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * compressor.c */ #include #include #include "compressor.h" #include "squashfs_fs.h" #ifndef GZIP_SUPPORT static struct compressor gzip_comp_ops = { ZLIB_COMPRESSION, "gzip" }; #else extern struct compressor gzip_comp_ops; #endif #ifndef LZMA_SUPPORT static struct compressor lzma_comp_ops = { LZMA_COMPRESSION, "lzma" }; #else extern struct compressor lzma_comp_ops; #endif #ifndef LZO_SUPPORT static struct compressor lzo_comp_ops = { LZO_COMPRESSION, "lzo" }; #else extern struct compressor lzo_comp_ops; #endif #ifndef LZ4_SUPPORT static struct compressor lz4_comp_ops = { LZ4_COMPRESSION, "lz4" }; #else extern struct compressor lz4_comp_ops; #endif #ifndef XZ_SUPPORT static struct compressor xz_comp_ops = { XZ_COMPRESSION, "xz" }; #else extern struct compressor xz_comp_ops; #endif static struct compressor unknown_comp_ops = { 0, "unknown" }; struct compressor *compressor[] = { &gzip_comp_ops, &lzma_comp_ops, &lzo_comp_ops, &lz4_comp_ops, &xz_comp_ops, &unknown_comp_ops }; struct compressor *lookup_compressor(char *name) { int i; for(i = 0; compressor[i]->id; i++) if(strcmp(compressor[i]->name, name) == 0) break; return compressor[i]; } struct compressor *lookup_compressor_id(int id) { int i; for(i = 0; compressor[i]->id; i++) if(id == compressor[i]->id) break; return compressor[i]; } void display_compressors(char *indent, char *def_comp) { int i; for(i = 0; compressor[i]->id; i++) if(compressor[i]->supported) fprintf(stderr, "%s\t%s%s\n", indent, compressor[i]->name, strcmp(compressor[i]->name, def_comp) == 0 ? " (default)" : ""); } void display_compressor_usage(char *def_comp) { int i; for(i = 0; compressor[i]->id; i++) if(compressor[i]->supported) { char *str = strcmp(compressor[i]->name, def_comp) == 0 ? " (default)" : ""; if(compressor[i]->usage) { fprintf(stderr, "\t%s%s\n", compressor[i]->name, str); compressor[i]->usage(); } else fprintf(stderr, "\t%s (no options)%s\n", compressor[i]->name, str); } } squashfs4.3/squashfs-tools/lzo_wrapper.c0000644000175000017500000002740412306776317020552 0ustar phillipphillip/* * Copyright (c) 2013, 2014 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * lzo_wrapper.c * * Support for LZO compression http://www.oberhumer.com/opensource/lzo */ #include #include #include #include #include #include "squashfs_fs.h" #include "lzo_wrapper.h" #include "compressor.h" static struct lzo_algorithm lzo[] = { { "lzo1x_1", LZO1X_1_MEM_COMPRESS, lzo1x_1_compress }, { "lzo1x_1_11", LZO1X_1_11_MEM_COMPRESS, lzo1x_1_11_compress }, { "lzo1x_1_12", LZO1X_1_12_MEM_COMPRESS, lzo1x_1_12_compress }, { "lzo1x_1_15", LZO1X_1_15_MEM_COMPRESS, lzo1x_1_15_compress }, { "lzo1x_999", LZO1X_999_MEM_COMPRESS, lzo1x_999_wrapper }, { NULL, 0, NULL } }; /* default LZO compression algorithm and compression level */ static int algorithm = SQUASHFS_LZO1X_999; static int compression_level = SQUASHFS_LZO1X_999_COMP_DEFAULT; /* user specified compression level */ static int user_comp_level = -1; /* * This function is called by the options parsing code in mksquashfs.c * to parse any -X compressor option. * * This function returns: * >=0 (number of additional args parsed) on success * -1 if the option was unrecognised, or * -2 if the option was recognised, but otherwise bad in * some way (e.g. invalid parameter) * * Note: this function sets internal compressor state, but does not * pass back the results of the parsing other than success/failure. * The lzo_dump_options() function is called later to get the options in * a format suitable for writing to the filesystem. */ static int lzo_options(char *argv[], int argc) { int i; if(strcmp(argv[0], "-Xalgorithm") == 0) { if(argc < 2) { fprintf(stderr, "lzo: -Xalgorithm missing algorithm\n"); fprintf(stderr, "lzo: -Xalgorithm \n"); goto failed2; } for(i = 0; lzo[i].name; i++) { if(strcmp(argv[1], lzo[i].name) == 0) { algorithm = i; return 1; } } fprintf(stderr, "lzo: -Xalgorithm unrecognised algorithm\n"); goto failed2; } else if(strcmp(argv[0], "-Xcompression-level") == 0) { if(argc < 2) { fprintf(stderr, "lzo: -Xcompression-level missing " "compression level\n"); fprintf(stderr, "lzo: -Xcompression-level it " "should be 1 >= n <= 9\n"); goto failed; } user_comp_level = atoi(argv[1]); if(user_comp_level < 1 || user_comp_level > 9) { fprintf(stderr, "lzo: -Xcompression-level invalid, it " "should be 1 >= n <= 9\n"); goto failed; } return 1; } return -1; failed: return -2; failed2: fprintf(stderr, "lzo: compression algorithm should be one of:\n"); for(i = 0; lzo[i].name; i++) fprintf(stderr, "\t%s\n", lzo[i].name); return -2; } /* * This function is called after all options have been parsed. * It is used to do post-processing on the compressor options using * values that were not expected to be known at option parse time. * * In this case the LZO algorithm may not be known until after the * compression level has been set (-Xalgorithm used after -Xcompression-level) * * This function returns 0 on successful post processing, or * -1 on error */ static int lzo_options_post(int block_size) { /* * Use of compression level only makes sense for * LZO1X_999 algorithm */ if(user_comp_level != -1) { if(algorithm != SQUASHFS_LZO1X_999) { fprintf(stderr, "lzo: -Xcompression-level not " "supported by selected %s algorithm\n", lzo[algorithm].name); fprintf(stderr, "lzo: -Xcompression-level is only " "applicable for the lzo1x_999 algorithm\n"); goto failed; } compression_level = user_comp_level; } return 0; failed: return -1; } /* * This function is called by mksquashfs to dump the parsed * compressor options in a format suitable for writing to the * compressor options field in the filesystem (stored immediately * after the superblock). * * This function returns a pointer to the compression options structure * to be stored (and the size), or NULL if there are no compression * options * */ static void *lzo_dump_options(int block_size, int *size) { static struct lzo_comp_opts comp_opts; /* * If default compression options of SQUASHFS_LZO1X_999 and * compression level of SQUASHFS_LZO1X_999_COMP_DEFAULT then * don't store a compression options structure (this is compatible * with the legacy implementation of LZO for Squashfs) */ if(algorithm == SQUASHFS_LZO1X_999 && compression_level == SQUASHFS_LZO1X_999_COMP_DEFAULT) return NULL; comp_opts.algorithm = algorithm; comp_opts.compression_level = algorithm == SQUASHFS_LZO1X_999 ? compression_level : 0; SQUASHFS_INSWAP_COMP_OPTS(&comp_opts); *size = sizeof(comp_opts); return &comp_opts; } /* * This function is a helper specifically for the append mode of * mksquashfs. Its purpose is to set the internal compressor state * to the stored compressor options in the passed compressor options * structure. * * In effect this function sets up the compressor options * to the same state they were when the filesystem was originally * generated, this is to ensure on appending, the compressor uses * the same compression options that were used to generate the * original filesystem. * * Note, even if there are no compressor options, this function is still * called with an empty compressor structure (size == 0), to explicitly * set the default options, this is to ensure any user supplied * -X options on the appending mksquashfs command line are over-ridden * * This function returns 0 on sucessful extraction of options, and * -1 on error */ static int lzo_extract_options(int block_size, void *buffer, int size) { struct lzo_comp_opts *comp_opts = buffer; if(size == 0) { /* Set default values */ algorithm = SQUASHFS_LZO1X_999; compression_level = SQUASHFS_LZO1X_999_COMP_DEFAULT; return 0; } /* we expect a comp_opts structure of sufficient size to be present */ if(size < sizeof(*comp_opts)) goto failed; SQUASHFS_INSWAP_COMP_OPTS(comp_opts); /* Check comp_opts structure for correctness */ switch(comp_opts->algorithm) { case SQUASHFS_LZO1X_1: case SQUASHFS_LZO1X_1_11: case SQUASHFS_LZO1X_1_12: case SQUASHFS_LZO1X_1_15: if(comp_opts->compression_level != 0) { fprintf(stderr, "lzo: bad compression level in " "compression options structure\n"); goto failed; } break; case SQUASHFS_LZO1X_999: if(comp_opts->compression_level < 1 || comp_opts->compression_level > 9) { fprintf(stderr, "lzo: bad compression level in " "compression options structure\n"); goto failed; } compression_level = comp_opts->compression_level; break; default: fprintf(stderr, "lzo: bad algorithm in compression options " "structure\n"); goto failed; } algorithm = comp_opts->algorithm; return 0; failed: fprintf(stderr, "lzo: error reading stored compressor options from " "filesystem!\n"); return -1; } void lzo_display_options(void *buffer, int size) { struct lzo_comp_opts *comp_opts = buffer; /* we expect a comp_opts structure of sufficient size to be present */ if(size < sizeof(*comp_opts)) goto failed; SQUASHFS_INSWAP_COMP_OPTS(comp_opts); /* Check comp_opts structure for correctness */ switch(comp_opts->algorithm) { case SQUASHFS_LZO1X_1: case SQUASHFS_LZO1X_1_11: case SQUASHFS_LZO1X_1_12: case SQUASHFS_LZO1X_1_15: printf("\talgorithm %s\n", lzo[comp_opts->algorithm].name); break; case SQUASHFS_LZO1X_999: if(comp_opts->compression_level < 1 || comp_opts->compression_level > 9) { fprintf(stderr, "lzo: bad compression level in " "compression options structure\n"); goto failed; } printf("\talgorithm %s\n", lzo[comp_opts->algorithm].name); printf("\tcompression level %d\n", comp_opts->compression_level); break; default: fprintf(stderr, "lzo: bad algorithm in compression options " "structure\n"); goto failed; } return; failed: fprintf(stderr, "lzo: error reading stored compressor options from " "filesystem!\n"); } /* * This function is called by mksquashfs to initialise the * compressor, before compress() is called. * * This function returns 0 on success, and * -1 on error */ static int squashfs_lzo_init(void **strm, int block_size, int datablock) { struct lzo_stream *stream; stream = *strm = malloc(sizeof(struct lzo_stream)); if(stream == NULL) goto failed; stream->workspace = malloc(lzo[algorithm].size); if(stream->workspace == NULL) goto failed2; stream->buffer = malloc(LZO_MAX_EXPANSION(block_size)); if(stream->buffer != NULL) return 0; free(stream->workspace); failed2: free(stream); failed: return -1; } static int lzo_compress(void *strm, void *dest, void *src, int size, int block_size, int *error) { int res; lzo_uint compsize, orig_size = size; struct lzo_stream *stream = strm; res = lzo[algorithm].compress(src, size, stream->buffer, &compsize, stream->workspace); if(res != LZO_E_OK) goto failed; /* Successful compression, however, we need to check that * the compressed size is not larger than the available * buffer space. Normally in other compressor APIs they take * a destination buffer size, and overflows return an error. * With LZO it lacks a destination size and so we must output * to a temporary buffer large enough to accomodate any * result, and explictly check here for overflow */ if(compsize > block_size) return 0; res = lzo1x_optimize(stream->buffer, compsize, src, &orig_size, NULL); if (res != LZO_E_OK || orig_size != size) goto failed; memcpy(dest, stream->buffer, compsize); return compsize; failed: /* fail, compressor specific error code returned in error */ *error = res; return -1; } static int lzo_uncompress(void *dest, void *src, int size, int outsize, int *error) { int res; lzo_uint outlen = outsize; res = lzo1x_decompress_safe(src, size, dest, &outlen, NULL); if(res != LZO_E_OK) { *error = res; return -1; } return outlen; } void lzo_usage() { int i; fprintf(stderr, "\t -Xalgorithm \n"); fprintf(stderr, "\t\tWhere is one of:\n"); for(i = 0; lzo[i].name; i++) fprintf(stderr, "\t\t\t%s%s\n", lzo[i].name, i == SQUASHFS_LZO1X_999 ? " (default)" : ""); fprintf(stderr, "\t -Xcompression-level \n"); fprintf(stderr, "\t\t should be 1 .. 9 (default " "%d)\n", SQUASHFS_LZO1X_999_COMP_DEFAULT); fprintf(stderr, "\t\tOnly applies to lzo1x_999 algorithm\n"); } /* * Helper function for lzo1x_999 compression algorithm. * All other lzo1x_xxx compressors do not take a compression level, * so we need to wrap lzo1x_999 to pass the compression level which * is applicable to it */ int lzo1x_999_wrapper(const lzo_bytep src, lzo_uint src_len, lzo_bytep dst, lzo_uintp compsize, lzo_voidp workspace) { return lzo1x_999_compress_level(src, src_len, dst, compsize, workspace, NULL, 0, 0, compression_level); } struct compressor lzo_comp_ops = { .init = squashfs_lzo_init, .compress = lzo_compress, .uncompress = lzo_uncompress, .options = lzo_options, .options_post = lzo_options_post, .dump_options = lzo_dump_options, .extract_options = lzo_extract_options, .display_options = lzo_display_options, .usage = lzo_usage, .id = LZO_COMPRESSION, .name = "lzo", .supported = 1 }; squashfs4.3/squashfs-tools/progressbar.h0000644000175000017500000000226312333330365020525 0ustar phillipphillip#ifndef PROGRESSBAR_H #define PROGRESSBAR_H /* * Create a squashfs filesystem. This is a highly compressed read only * filesystem. * * Copyright (c) 2012, 2013, 2014 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * progressbar.h */ extern void inc_progress_bar(); extern void dec_progress_bar(int count); extern void progress_bar_size(int count); extern void enable_progress_bar(); extern void disable_progress_bar(); extern void init_progress_bar(); extern void set_progressbar_state(int); #endif squashfs4.3/squashfs-tools/mksquashfs.c0000644000175000017500000045024712334244254020367 0ustar phillipphillip/* * Create a squashfs filesystem. This is a highly compressed read only * filesystem. * * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, * 2012, 2013, 2014 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * mksquashfs.c */ #define FALSE 0 #define TRUE 1 #define MAX_LINE 16384 #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifndef linux #define __BYTE_ORDER BYTE_ORDER #define __BIG_ENDIAN BIG_ENDIAN #define __LITTLE_ENDIAN LITTLE_ENDIAN #include #else #include #include #endif #include "squashfs_fs.h" #include "squashfs_swap.h" #include "mksquashfs.h" #include "sort.h" #include "pseudo.h" #include "compressor.h" #include "xattr.h" #include "action.h" #include "error.h" #include "progressbar.h" #include "info.h" #include "caches-queues-lists.h" #include "read_fs.h" #include "restore.h" #include "process_fragments.h" int delete = FALSE; int fd; struct squashfs_super_block sBlk; /* filesystem flags for building */ int comp_opts = FALSE; int no_xattrs = XATTR_DEF; int noX = FALSE; int duplicate_checking = TRUE; int noF = FALSE; int no_fragments = FALSE; int always_use_fragments = FALSE; int noI = FALSE; int noD = FALSE; int silent = TRUE; int exportable = TRUE; int sparse_files = TRUE; int old_exclude = TRUE; int use_regex = FALSE; int nopad = FALSE; int exit_on_error = FALSE; long long global_uid = -1, global_gid = -1; /* superblock attributes */ int block_size = SQUASHFS_FILE_SIZE, block_log; unsigned int id_count = 0; int file_count = 0, sym_count = 0, dev_count = 0, dir_count = 0, fifo_count = 0, sock_count = 0; /* write position within data section */ long long bytes = 0, total_bytes = 0; /* in memory directory table - possibly compressed */ char *directory_table = NULL; unsigned int directory_bytes = 0, directory_size = 0, total_directory_bytes = 0; /* cached directory table */ char *directory_data_cache = NULL; unsigned int directory_cache_bytes = 0, directory_cache_size = 0; /* in memory inode table - possibly compressed */ char *inode_table = NULL; unsigned int inode_bytes = 0, inode_size = 0, total_inode_bytes = 0; /* cached inode table */ char *data_cache = NULL; unsigned int cache_bytes = 0, cache_size = 0, inode_count = 0; /* inode lookup table */ squashfs_inode *inode_lookup_table = NULL; /* in memory directory data */ #define I_COUNT_SIZE 128 #define DIR_ENTRIES 32 #define INODE_HASH_SIZE 65536 #define INODE_HASH_MASK (INODE_HASH_SIZE - 1) #define INODE_HASH(dev, ino) (ino & INODE_HASH_MASK) struct cached_dir_index { struct squashfs_dir_index index; char *name; }; struct directory { unsigned int start_block; unsigned int size; unsigned char *buff; unsigned char *p; unsigned int entry_count; unsigned char *entry_count_p; unsigned int i_count; unsigned int i_size; struct cached_dir_index *index; unsigned char *index_count_p; unsigned int inode_number; }; struct inode_info *inode_info[INODE_HASH_SIZE]; /* hash tables used to do fast duplicate searches in duplicate check */ struct file_info *dupl[65536]; int dup_files = 0; /* exclude file handling */ /* list of exclude dirs/files */ struct exclude_info { dev_t st_dev; ino_t st_ino; }; #define EXCLUDE_SIZE 8192 int exclude = 0; struct exclude_info *exclude_paths = NULL; int old_excluded(char *filename, struct stat *buf); struct path_entry { char *name; regex_t *preg; struct pathname *paths; }; struct pathname { int names; struct path_entry *name; }; struct pathnames { int count; struct pathname *path[0]; }; #define PATHS_ALLOC_SIZE 10 struct pathnames *paths = NULL; struct pathname *path = NULL; struct pathname *stickypath = NULL; int excluded(char *name, struct pathnames *paths, struct pathnames **new); int fragments = 0; #define FRAG_SIZE 32768 struct squashfs_fragment_entry *fragment_table = NULL; int fragments_outstanding = 0; int fragments_locked = FALSE; /* current inode number for directories and non directories */ unsigned int inode_no = 1; unsigned int root_inode_number = 0; /* list of source dirs/files */ int source = 0; char **source_path; /* list of root directory entries read from original filesystem */ int old_root_entries = 0; struct old_root_entry_info { char *name; struct inode_info inode; }; struct old_root_entry_info *old_root_entry; /* restore orignal filesystem state if appending to existing filesystem is * cancelled */ int appending = FALSE; char *sdata_cache, *sdirectory_data_cache, *sdirectory_compressed; long long sbytes, stotal_bytes; unsigned int sinode_bytes, scache_bytes, sdirectory_bytes, sdirectory_cache_bytes, sdirectory_compressed_bytes, stotal_inode_bytes, stotal_directory_bytes, sinode_count = 0, sfile_count, ssym_count, sdev_count, sdir_count, sfifo_count, ssock_count, sdup_files; int sfragments; int threads; /* flag whether destination file is a block device */ int block_device = FALSE; /* flag indicating whether files are sorted using sort list(s) */ int sorted = FALSE; /* save destination file name for deleting on error */ char *destination_file = NULL; /* recovery file for abnormal exit on appending */ char *recovery_file = NULL; int recover = TRUE; struct id *id_hash_table[ID_ENTRIES]; struct id *id_table[SQUASHFS_IDS], *sid_table[SQUASHFS_IDS]; unsigned int uid_count = 0, guid_count = 0; unsigned int sid_count = 0, suid_count = 0, sguid_count = 0; struct cache *reader_buffer, *fragment_buffer, *reserve_cache; struct cache *bwriter_buffer, *fwriter_buffer; struct queue *to_reader, *to_deflate, *to_writer, *from_writer, *to_frag, *locked_fragment, *to_process_frag; struct seq_queue *to_main; pthread_t reader_thread, writer_thread, main_thread; pthread_t *deflator_thread, *frag_deflator_thread, *frag_thread; pthread_t *restore_thread = NULL; pthread_mutex_t fragment_mutex = PTHREAD_MUTEX_INITIALIZER; pthread_mutex_t pos_mutex = PTHREAD_MUTEX_INITIALIZER; pthread_mutex_t dup_mutex = PTHREAD_MUTEX_INITIALIZER; /* user options that control parallelisation */ int processors = -1; int bwriter_size; /* compression operations */ struct compressor *comp = NULL; int compressor_opt_parsed = FALSE; void *stream = NULL; /* xattr stats */ unsigned int xattr_bytes = 0, total_xattr_bytes = 0; /* fragment to file mapping used when appending */ int append_fragments = 0; struct append_file **file_mapping; static char *read_from_disk(long long start, unsigned int avail_bytes); void add_old_root_entry(char *name, squashfs_inode inode, int inode_number, int type); struct file_info *duplicate(long long file_size, long long bytes, unsigned int **block_list, long long *start, struct fragment **fragment, struct file_buffer *file_buffer, int blocks, unsigned short checksum, int checksum_flag); struct dir_info *dir_scan1(char *, char *, struct pathnames *, struct dir_ent *(_readdir)(struct dir_info *), int); void dir_scan2(struct dir_info *dir, struct pseudo *pseudo); void dir_scan3(struct dir_info *root, struct dir_info *dir); void dir_scan4(struct dir_info *dir); void dir_scan5(struct dir_info *dir); void dir_scan6(squashfs_inode *inode, struct dir_info *dir_info); struct file_info *add_non_dup(long long file_size, long long bytes, unsigned int *block_list, long long start, struct fragment *fragment, unsigned short checksum, unsigned short fragment_checksum, int checksum_flag, int checksum_frag_flag); long long generic_write_table(int, void *, int, void *, int); void restorefs(); struct dir_info *scan1_opendir(char *pathname, char *subpath, int depth); void write_filesystem_tables(struct squashfs_super_block *sBlk, int nopad); unsigned short get_checksum_mem(char *buff, int bytes); int get_physical_memory(); void prep_exit() { if(restore_thread) { if(pthread_self() == *restore_thread) { /* * Recursive failure when trying to restore filesystem! * Nothing to do except to exit, otherwise we'll just * appear to hang. The user should be able to restore * from the recovery file (which is why it was added, in * case of catastrophic failure in Mksquashfs) */ exit(1); } else { /* signal the restore thread to restore */ pthread_kill(*restore_thread, SIGUSR1); pthread_exit(NULL); } } else if(delete) { if(destination_file && !block_device) unlink(destination_file); } else if(recovery_file) unlink(recovery_file); } int add_overflow(int a, int b) { return (INT_MAX - a) < b; } int shift_overflow(int a, int shift) { return (INT_MAX >> shift) < a; } int multiply_overflow(int a, int multiplier) { return (INT_MAX / multiplier) < a; } int multiply_overflowll(long long a, int multiplier) { return (LLONG_MAX / multiplier) < a; } #define MKINODE(A) ((squashfs_inode)(((squashfs_inode) inode_bytes << 16) \ + (((char *)A) - data_cache))) void restorefs() { ERROR("Exiting - restoring original filesystem!\n\n"); bytes = sbytes; memcpy(data_cache, sdata_cache, cache_bytes = scache_bytes); memcpy(directory_data_cache, sdirectory_data_cache, sdirectory_cache_bytes); directory_cache_bytes = sdirectory_cache_bytes; inode_bytes = sinode_bytes; directory_bytes = sdirectory_bytes; memcpy(directory_table + directory_bytes, sdirectory_compressed, sdirectory_compressed_bytes); directory_bytes += sdirectory_compressed_bytes; total_bytes = stotal_bytes; total_inode_bytes = stotal_inode_bytes; total_directory_bytes = stotal_directory_bytes; inode_count = sinode_count; file_count = sfile_count; sym_count = ssym_count; dev_count = sdev_count; dir_count = sdir_count; fifo_count = sfifo_count; sock_count = ssock_count; dup_files = sdup_files; fragments = sfragments; id_count = sid_count; restore_xattrs(); write_filesystem_tables(&sBlk, nopad); exit(1); } void sighandler() { EXIT_MKSQUASHFS(); } int mangle2(void *strm, char *d, char *s, int size, int block_size, int uncompressed, int data_block) { int error, c_byte = 0; if(!uncompressed) { c_byte = compressor_compress(comp, strm, d, s, size, block_size, &error); if(c_byte == -1) BAD_ERROR("mangle2:: %s compress failed with error " "code %d\n", comp->name, error); } if(c_byte == 0 || c_byte >= size) { memcpy(d, s, size); return size | (data_block ? SQUASHFS_COMPRESSED_BIT_BLOCK : SQUASHFS_COMPRESSED_BIT); } return c_byte; } int mangle(char *d, char *s, int size, int block_size, int uncompressed, int data_block) { return mangle2(stream, d, s, size, block_size, uncompressed, data_block); } void *get_inode(int req_size) { int data_space; unsigned short c_byte; while(cache_bytes >= SQUASHFS_METADATA_SIZE) { if((inode_size - inode_bytes) < ((SQUASHFS_METADATA_SIZE << 1)) + 2) { void *it = realloc(inode_table, inode_size + (SQUASHFS_METADATA_SIZE << 1) + 2); if(it == NULL) MEM_ERROR(); inode_table = it; inode_size += (SQUASHFS_METADATA_SIZE << 1) + 2; } c_byte = mangle(inode_table + inode_bytes + BLOCK_OFFSET, data_cache, SQUASHFS_METADATA_SIZE, SQUASHFS_METADATA_SIZE, noI, 0); TRACE("Inode block @ 0x%x, size %d\n", inode_bytes, c_byte); SQUASHFS_SWAP_SHORTS(&c_byte, inode_table + inode_bytes, 1); inode_bytes += SQUASHFS_COMPRESSED_SIZE(c_byte) + BLOCK_OFFSET; total_inode_bytes += SQUASHFS_METADATA_SIZE + BLOCK_OFFSET; memmove(data_cache, data_cache + SQUASHFS_METADATA_SIZE, cache_bytes - SQUASHFS_METADATA_SIZE); cache_bytes -= SQUASHFS_METADATA_SIZE; } data_space = (cache_size - cache_bytes); if(data_space < req_size) { int realloc_size = cache_size == 0 ? ((req_size + SQUASHFS_METADATA_SIZE) & ~(SQUASHFS_METADATA_SIZE - 1)) : req_size - data_space; void *dc = realloc(data_cache, cache_size + realloc_size); if(dc == NULL) MEM_ERROR(); cache_size += realloc_size; data_cache = dc; } cache_bytes += req_size; return data_cache + cache_bytes - req_size; } int read_bytes(int fd, void *buff, int bytes) { int res, count; for(count = 0; count < bytes; count += res) { res = read(fd, buff + count, bytes - count); if(res < 1) { if(res == 0) goto bytes_read; else if(errno != EINTR) { ERROR("Read failed because %s\n", strerror(errno)); return -1; } else res = 0; } } bytes_read: return count; } int read_fs_bytes(int fd, long long byte, int bytes, void *buff) { off_t off = byte; int res = 1; TRACE("read_fs_bytes: reading from position 0x%llx, bytes %d\n", byte, bytes); pthread_cleanup_push((void *) pthread_mutex_unlock, &pos_mutex); pthread_mutex_lock(&pos_mutex); if(lseek(fd, off, SEEK_SET) == -1) { ERROR("read_fs_bytes: Lseek on destination failed because %s, " "offset=0x%llx\n", strerror(errno), off); res = 0; } else if(read_bytes(fd, buff, bytes) < bytes) { ERROR("Read on destination failed\n"); res = 0; } pthread_cleanup_pop(1); return res; } int write_bytes(int fd, void *buff, int bytes) { int res, count; for(count = 0; count < bytes; count += res) { res = write(fd, buff + count, bytes - count); if(res == -1) { if(errno != EINTR) { ERROR("Write failed because %s\n", strerror(errno)); return -1; } res = 0; } } return 0; } void write_destination(int fd, long long byte, int bytes, void *buff) { off_t off = byte; pthread_cleanup_push((void *) pthread_mutex_unlock, &pos_mutex); pthread_mutex_lock(&pos_mutex); if(lseek(fd, off, SEEK_SET) == -1) { ERROR("write_destination: Lseek on destination " "failed because %s, offset=0x%llx\n", strerror(errno), off); BAD_ERROR("Probably out of space on output %s\n", block_device ? "block device" : "filesystem"); } if(write_bytes(fd, buff, bytes) == -1) BAD_ERROR("Failed to write to output %s\n", block_device ? "block device" : "filesystem"); pthread_cleanup_pop(1); } long long write_inodes() { unsigned short c_byte; int avail_bytes; char *datap = data_cache; long long start_bytes = bytes; while(cache_bytes) { if(inode_size - inode_bytes < ((SQUASHFS_METADATA_SIZE << 1) + 2)) { void *it = realloc(inode_table, inode_size + ((SQUASHFS_METADATA_SIZE << 1) + 2)); if(it == NULL) MEM_ERROR(); inode_size += (SQUASHFS_METADATA_SIZE << 1) + 2; inode_table = it; } avail_bytes = cache_bytes > SQUASHFS_METADATA_SIZE ? SQUASHFS_METADATA_SIZE : cache_bytes; c_byte = mangle(inode_table + inode_bytes + BLOCK_OFFSET, datap, avail_bytes, SQUASHFS_METADATA_SIZE, noI, 0); TRACE("Inode block @ 0x%x, size %d\n", inode_bytes, c_byte); SQUASHFS_SWAP_SHORTS(&c_byte, inode_table + inode_bytes, 1); inode_bytes += SQUASHFS_COMPRESSED_SIZE(c_byte) + BLOCK_OFFSET; total_inode_bytes += avail_bytes + BLOCK_OFFSET; datap += avail_bytes; cache_bytes -= avail_bytes; } write_destination(fd, bytes, inode_bytes, inode_table); bytes += inode_bytes; return start_bytes; } long long write_directories() { unsigned short c_byte; int avail_bytes; char *directoryp = directory_data_cache; long long start_bytes = bytes; while(directory_cache_bytes) { if(directory_size - directory_bytes < ((SQUASHFS_METADATA_SIZE << 1) + 2)) { void *dt = realloc(directory_table, directory_size + ((SQUASHFS_METADATA_SIZE << 1) + 2)); if(dt == NULL) MEM_ERROR(); directory_size += (SQUASHFS_METADATA_SIZE << 1) + 2; directory_table = dt; } avail_bytes = directory_cache_bytes > SQUASHFS_METADATA_SIZE ? SQUASHFS_METADATA_SIZE : directory_cache_bytes; c_byte = mangle(directory_table + directory_bytes + BLOCK_OFFSET, directoryp, avail_bytes, SQUASHFS_METADATA_SIZE, noI, 0); TRACE("Directory block @ 0x%x, size %d\n", directory_bytes, c_byte); SQUASHFS_SWAP_SHORTS(&c_byte, directory_table + directory_bytes, 1); directory_bytes += SQUASHFS_COMPRESSED_SIZE(c_byte) + BLOCK_OFFSET; total_directory_bytes += avail_bytes + BLOCK_OFFSET; directoryp += avail_bytes; directory_cache_bytes -= avail_bytes; } write_destination(fd, bytes, directory_bytes, directory_table); bytes += directory_bytes; return start_bytes; } long long write_id_table() { unsigned int id_bytes = SQUASHFS_ID_BYTES(id_count); unsigned int p[id_count]; int i; TRACE("write_id_table: ids %d, id_bytes %d\n", id_count, id_bytes); for(i = 0; i < id_count; i++) { TRACE("write_id_table: id index %d, id %d", i, id_table[i]->id); SQUASHFS_SWAP_INTS(&id_table[i]->id, p + i, 1); } return generic_write_table(id_bytes, p, 0, NULL, noI); } struct id *get_id(unsigned int id) { int hash = ID_HASH(id); struct id *entry = id_hash_table[hash]; for(; entry; entry = entry->next) if(entry->id == id) break; return entry; } struct id *create_id(unsigned int id) { int hash = ID_HASH(id); struct id *entry = malloc(sizeof(struct id)); if(entry == NULL) MEM_ERROR(); entry->id = id; entry->index = id_count ++; entry->flags = 0; entry->next = id_hash_table[hash]; id_hash_table[hash] = entry; id_table[entry->index] = entry; return entry; } unsigned int get_uid(unsigned int uid) { struct id *entry = get_id(uid); if(entry == NULL) { if(id_count == SQUASHFS_IDS) BAD_ERROR("Out of uids!\n"); entry = create_id(uid); } if((entry->flags & ISA_UID) == 0) { entry->flags |= ISA_UID; uid_count ++; } return entry->index; } unsigned int get_guid(unsigned int guid) { struct id *entry = get_id(guid); if(entry == NULL) { if(id_count == SQUASHFS_IDS) BAD_ERROR("Out of gids!\n"); entry = create_id(guid); } if((entry->flags & ISA_GID) == 0) { entry->flags |= ISA_GID; guid_count ++; } return entry->index; } #define ALLOC_SIZE 128 char *_pathname(struct dir_ent *dir_ent, char *pathname, int *size) { if(pathname == NULL) { pathname = malloc(ALLOC_SIZE); if(pathname == NULL) MEM_ERROR(); } for(;;) { int res = snprintf(pathname, *size, "%s/%s", dir_ent->our_dir->pathname, dir_ent->source_name ? : dir_ent->name); if(res < 0) BAD_ERROR("snprintf failed in pathname\n"); else if(res >= *size) { /* * pathname is too small to contain the result, so * increase it and try again */ *size = (res + ALLOC_SIZE) & ~(ALLOC_SIZE - 1); pathname = realloc(pathname, *size); if(pathname == NULL) MEM_ERROR(); } else break; } return pathname; } char *pathname(struct dir_ent *dir_ent) { static char *pathname = NULL; static int size = ALLOC_SIZE; if (dir_ent->nonstandard_pathname) return dir_ent->nonstandard_pathname; return pathname = _pathname(dir_ent, pathname, &size); } char *pathname_reader(struct dir_ent *dir_ent) { static char *pathname = NULL; static int size = ALLOC_SIZE; if (dir_ent->nonstandard_pathname) return dir_ent->nonstandard_pathname; return pathname = _pathname(dir_ent, pathname, &size); } char *subpathname(struct dir_ent *dir_ent) { static char *subpath = NULL; static int size = ALLOC_SIZE; int res; if(subpath == NULL) { subpath = malloc(ALLOC_SIZE); if(subpath == NULL) MEM_ERROR(); } for(;;) { if(dir_ent->our_dir->subpath[0] != '\0') res = snprintf(subpath, size, "%s/%s", dir_ent->our_dir->subpath, dir_ent->name); else res = snprintf(subpath, size, "/%s", dir_ent->name); if(res < 0) BAD_ERROR("snprintf failed in subpathname\n"); else if(res >= size) { /* * subpath is too small to contain the result, so * increase it and try again */ size = (res + ALLOC_SIZE) & ~(ALLOC_SIZE - 1); subpath = realloc(subpath, size); if(subpath == NULL) MEM_ERROR(); } else break; } return subpath; } inline unsigned int get_inode_no(struct inode_info *inode) { return inode->inode_number; } inline unsigned int get_parent_no(struct dir_info *dir) { return dir->depth ? get_inode_no(dir->dir_ent->inode) : inode_no; } int create_inode(squashfs_inode *i_no, struct dir_info *dir_info, struct dir_ent *dir_ent, int type, long long byte_size, long long start_block, unsigned int offset, unsigned int *block_list, struct fragment *fragment, struct directory *dir_in, long long sparse) { struct stat *buf = &dir_ent->inode->buf; union squashfs_inode_header inode_header; struct squashfs_base_inode_header *base = &inode_header.base; void *inode; char *filename = pathname(dir_ent); int nlink = dir_ent->inode->nlink; int xattr = read_xattrs(dir_ent); switch(type) { case SQUASHFS_FILE_TYPE: if(dir_ent->inode->nlink > 1 || byte_size >= (1LL << 32) || start_block >= (1LL << 32) || sparse || IS_XATTR(xattr)) type = SQUASHFS_LREG_TYPE; break; case SQUASHFS_DIR_TYPE: if(dir_info->dir_is_ldir || IS_XATTR(xattr)) type = SQUASHFS_LDIR_TYPE; break; case SQUASHFS_SYMLINK_TYPE: if(IS_XATTR(xattr)) type = SQUASHFS_LSYMLINK_TYPE; break; case SQUASHFS_BLKDEV_TYPE: if(IS_XATTR(xattr)) type = SQUASHFS_LBLKDEV_TYPE; break; case SQUASHFS_CHRDEV_TYPE: if(IS_XATTR(xattr)) type = SQUASHFS_LCHRDEV_TYPE; break; case SQUASHFS_FIFO_TYPE: if(IS_XATTR(xattr)) type = SQUASHFS_LFIFO_TYPE; break; case SQUASHFS_SOCKET_TYPE: if(IS_XATTR(xattr)) type = SQUASHFS_LSOCKET_TYPE; break; } base->mode = SQUASHFS_MODE(buf->st_mode); base->uid = get_uid((unsigned int) global_uid == -1 ? buf->st_uid : global_uid); base->inode_type = type; base->guid = get_guid((unsigned int) global_gid == -1 ? buf->st_gid : global_gid); base->mtime = buf->st_mtime; base->inode_number = get_inode_no(dir_ent->inode); if(type == SQUASHFS_FILE_TYPE) { int i; struct squashfs_reg_inode_header *reg = &inode_header.reg; size_t off = offsetof(struct squashfs_reg_inode_header, block_list); inode = get_inode(sizeof(*reg) + offset * sizeof(unsigned int)); reg->file_size = byte_size; reg->start_block = start_block; reg->fragment = fragment->index; reg->offset = fragment->offset; SQUASHFS_SWAP_REG_INODE_HEADER(reg, inode); SQUASHFS_SWAP_INTS(block_list, inode + off, offset); TRACE("File inode, file_size %lld, start_block 0x%llx, blocks " "%d, fragment %d, offset %d, size %d\n", byte_size, start_block, offset, fragment->index, fragment->offset, fragment->size); for(i = 0; i < offset; i++) TRACE("Block %d, size %d\n", i, block_list[i]); } else if(type == SQUASHFS_LREG_TYPE) { int i; struct squashfs_lreg_inode_header *reg = &inode_header.lreg; size_t off = offsetof(struct squashfs_lreg_inode_header, block_list); inode = get_inode(sizeof(*reg) + offset * sizeof(unsigned int)); reg->nlink = nlink; reg->file_size = byte_size; reg->start_block = start_block; reg->fragment = fragment->index; reg->offset = fragment->offset; if(sparse && sparse >= byte_size) sparse = byte_size - 1; reg->sparse = sparse; reg->xattr = xattr; SQUASHFS_SWAP_LREG_INODE_HEADER(reg, inode); SQUASHFS_SWAP_INTS(block_list, inode + off, offset); TRACE("Long file inode, file_size %lld, start_block 0x%llx, " "blocks %d, fragment %d, offset %d, size %d, nlink %d" "\n", byte_size, start_block, offset, fragment->index, fragment->offset, fragment->size, nlink); for(i = 0; i < offset; i++) TRACE("Block %d, size %d\n", i, block_list[i]); } else if(type == SQUASHFS_LDIR_TYPE) { int i; unsigned char *p; struct squashfs_ldir_inode_header *dir = &inode_header.ldir; struct cached_dir_index *index = dir_in->index; unsigned int i_count = dir_in->i_count; unsigned int i_size = dir_in->i_size; if(byte_size >= 1 << 27) BAD_ERROR("directory greater than 2^27-1 bytes!\n"); inode = get_inode(sizeof(*dir) + i_size); dir->inode_type = SQUASHFS_LDIR_TYPE; dir->nlink = dir_ent->dir->directory_count + 2; dir->file_size = byte_size; dir->offset = offset; dir->start_block = start_block; dir->i_count = i_count; dir->parent_inode = get_parent_no(dir_ent->our_dir); dir->xattr = xattr; SQUASHFS_SWAP_LDIR_INODE_HEADER(dir, inode); p = inode + offsetof(struct squashfs_ldir_inode_header, index); for(i = 0; i < i_count; i++) { SQUASHFS_SWAP_DIR_INDEX(&index[i].index, p); p += offsetof(struct squashfs_dir_index, name); memcpy(p, index[i].name, index[i].index.size + 1); p += index[i].index.size + 1; } TRACE("Long directory inode, file_size %lld, start_block " "0x%llx, offset 0x%x, nlink %d\n", byte_size, start_block, offset, dir_ent->dir->directory_count + 2); } else if(type == SQUASHFS_DIR_TYPE) { struct squashfs_dir_inode_header *dir = &inode_header.dir; inode = get_inode(sizeof(*dir)); dir->nlink = dir_ent->dir->directory_count + 2; dir->file_size = byte_size; dir->offset = offset; dir->start_block = start_block; dir->parent_inode = get_parent_no(dir_ent->our_dir); SQUASHFS_SWAP_DIR_INODE_HEADER(dir, inode); TRACE("Directory inode, file_size %lld, start_block 0x%llx, " "offset 0x%x, nlink %d\n", byte_size, start_block, offset, dir_ent->dir->directory_count + 2); } else if(type == SQUASHFS_CHRDEV_TYPE || type == SQUASHFS_BLKDEV_TYPE) { struct squashfs_dev_inode_header *dev = &inode_header.dev; unsigned int major = major(buf->st_rdev); unsigned int minor = minor(buf->st_rdev); if(major > 0xfff) { ERROR("Major %d out of range in device node %s, " "truncating to %d\n", major, filename, major & 0xfff); major &= 0xfff; } if(minor > 0xfffff) { ERROR("Minor %d out of range in device node %s, " "truncating to %d\n", minor, filename, minor & 0xfffff); minor &= 0xfffff; } inode = get_inode(sizeof(*dev)); dev->nlink = nlink; dev->rdev = (major << 8) | (minor & 0xff) | ((minor & ~0xff) << 12); SQUASHFS_SWAP_DEV_INODE_HEADER(dev, inode); TRACE("Device inode, rdev 0x%x, nlink %d\n", dev->rdev, nlink); } else if(type == SQUASHFS_LCHRDEV_TYPE || type == SQUASHFS_LBLKDEV_TYPE) { struct squashfs_ldev_inode_header *dev = &inode_header.ldev; unsigned int major = major(buf->st_rdev); unsigned int minor = minor(buf->st_rdev); if(major > 0xfff) { ERROR("Major %d out of range in device node %s, " "truncating to %d\n", major, filename, major & 0xfff); major &= 0xfff; } if(minor > 0xfffff) { ERROR("Minor %d out of range in device node %s, " "truncating to %d\n", minor, filename, minor & 0xfffff); minor &= 0xfffff; } inode = get_inode(sizeof(*dev)); dev->nlink = nlink; dev->rdev = (major << 8) | (minor & 0xff) | ((minor & ~0xff) << 12); dev->xattr = xattr; SQUASHFS_SWAP_LDEV_INODE_HEADER(dev, inode); TRACE("Device inode, rdev 0x%x, nlink %d\n", dev->rdev, nlink); } else if(type == SQUASHFS_SYMLINK_TYPE) { struct squashfs_symlink_inode_header *symlink = &inode_header.symlink; int byte; char buff[65536]; /* overflow safe */ size_t off = offsetof(struct squashfs_symlink_inode_header, symlink); byte = readlink(filename, buff, 65536); if(byte == -1) { ERROR_START("Failed to read symlink %s", filename); ERROR_EXIT(", creating empty symlink\n"); byte = 0; } if(byte == 65536) { ERROR_START("Symlink %s is greater than 65536 bytes!", filename); ERROR_EXIT(" Creating empty symlink\n"); byte = 0; } inode = get_inode(sizeof(*symlink) + byte); symlink->nlink = nlink; symlink->symlink_size = byte; SQUASHFS_SWAP_SYMLINK_INODE_HEADER(symlink, inode); strncpy(inode + off, buff, byte); TRACE("Symbolic link inode, symlink_size %d, nlink %d\n", byte, nlink); } else if(type == SQUASHFS_LSYMLINK_TYPE) { struct squashfs_symlink_inode_header *symlink = &inode_header.symlink; int byte; char buff[65536]; /* overflow safe */ size_t off = offsetof(struct squashfs_symlink_inode_header, symlink); byte = readlink(filename, buff, 65536); if(byte == -1) { ERROR_START("Failed to read symlink %s", filename); ERROR_EXIT(", creating empty symlink\n"); byte = 0; } if(byte == 65536) { ERROR_START("Symlink %s is greater than 65536 bytes!", filename); ERROR_EXIT(" Creating empty symlink\n"); byte = 0; } inode = get_inode(sizeof(*symlink) + byte + sizeof(unsigned int)); symlink->nlink = nlink; symlink->symlink_size = byte; SQUASHFS_SWAP_SYMLINK_INODE_HEADER(symlink, inode); strncpy(inode + off, buff, byte); SQUASHFS_SWAP_INTS(&xattr, inode + off + byte, 1); TRACE("Symbolic link inode, symlink_size %d, nlink %d\n", byte, nlink); } else if(type == SQUASHFS_FIFO_TYPE || type == SQUASHFS_SOCKET_TYPE) { struct squashfs_ipc_inode_header *ipc = &inode_header.ipc; inode = get_inode(sizeof(*ipc)); ipc->nlink = nlink; SQUASHFS_SWAP_IPC_INODE_HEADER(ipc, inode); TRACE("ipc inode, type %s, nlink %d\n", type == SQUASHFS_FIFO_TYPE ? "fifo" : "socket", nlink); } else if(type == SQUASHFS_LFIFO_TYPE || type == SQUASHFS_LSOCKET_TYPE) { struct squashfs_lipc_inode_header *ipc = &inode_header.lipc; inode = get_inode(sizeof(*ipc)); ipc->nlink = nlink; ipc->xattr = xattr; SQUASHFS_SWAP_LIPC_INODE_HEADER(ipc, inode); TRACE("ipc inode, type %s, nlink %d\n", type == SQUASHFS_FIFO_TYPE ? "fifo" : "socket", nlink); } else BAD_ERROR("Unrecognised inode %d in create_inode\n", type); *i_no = MKINODE(inode); inode_count ++; TRACE("Created inode 0x%llx, type %d, uid %d, guid %d\n", *i_no, type, base->uid, base->guid); return TRUE; } void add_dir(squashfs_inode inode, unsigned int inode_number, char *name, int type, struct directory *dir) { unsigned char *buff; struct squashfs_dir_entry idir; unsigned int start_block = inode >> 16; unsigned int offset = inode & 0xffff; unsigned int size = strlen(name); size_t name_off = offsetof(struct squashfs_dir_entry, name); if(size > SQUASHFS_NAME_LEN) { size = SQUASHFS_NAME_LEN; ERROR("Filename is greater than %d characters, truncating! ..." "\n", SQUASHFS_NAME_LEN); } if(dir->p + sizeof(struct squashfs_dir_entry) + size + sizeof(struct squashfs_dir_header) >= dir->buff + dir->size) { buff = realloc(dir->buff, dir->size += SQUASHFS_METADATA_SIZE); if(buff == NULL) MEM_ERROR(); dir->p = (dir->p - dir->buff) + buff; if(dir->entry_count_p) dir->entry_count_p = (dir->entry_count_p - dir->buff + buff); dir->index_count_p = dir->index_count_p - dir->buff + buff; dir->buff = buff; } if(dir->entry_count == 256 || start_block != dir->start_block || ((dir->entry_count_p != NULL) && ((dir->p + sizeof(struct squashfs_dir_entry) + size - dir->index_count_p) > SQUASHFS_METADATA_SIZE)) || ((long long) inode_number - dir->inode_number) > 32767 || ((long long) inode_number - dir->inode_number) < -32768) { if(dir->entry_count_p) { struct squashfs_dir_header dir_header; if((dir->p + sizeof(struct squashfs_dir_entry) + size - dir->index_count_p) > SQUASHFS_METADATA_SIZE) { if(dir->i_count % I_COUNT_SIZE == 0) { dir->index = realloc(dir->index, (dir->i_count + I_COUNT_SIZE) * sizeof(struct cached_dir_index)); if(dir->index == NULL) MEM_ERROR(); } dir->index[dir->i_count].index.index = dir->p - dir->buff; dir->index[dir->i_count].index.size = size - 1; dir->index[dir->i_count++].name = name; dir->i_size += sizeof(struct squashfs_dir_index) + size; dir->index_count_p = dir->p; } dir_header.count = dir->entry_count - 1; dir_header.start_block = dir->start_block; dir_header.inode_number = dir->inode_number; SQUASHFS_SWAP_DIR_HEADER(&dir_header, dir->entry_count_p); } dir->entry_count_p = dir->p; dir->start_block = start_block; dir->entry_count = 0; dir->inode_number = inode_number; dir->p += sizeof(struct squashfs_dir_header); } idir.offset = offset; idir.type = type; idir.size = size - 1; idir.inode_number = ((long long) inode_number - dir->inode_number); SQUASHFS_SWAP_DIR_ENTRY(&idir, dir->p); strncpy((char *) dir->p + name_off, name, size); dir->p += sizeof(struct squashfs_dir_entry) + size; dir->entry_count ++; } void write_dir(squashfs_inode *inode, struct dir_info *dir_info, struct directory *dir) { unsigned int dir_size = dir->p - dir->buff; int data_space = directory_cache_size - directory_cache_bytes; unsigned int directory_block, directory_offset, i_count, index; unsigned short c_byte; if(data_space < dir_size) { int realloc_size = directory_cache_size == 0 ? ((dir_size + SQUASHFS_METADATA_SIZE) & ~(SQUASHFS_METADATA_SIZE - 1)) : dir_size - data_space; void *dc = realloc(directory_data_cache, directory_cache_size + realloc_size); if(dc == NULL) MEM_ERROR(); directory_cache_size += realloc_size; directory_data_cache = dc; } if(dir_size) { struct squashfs_dir_header dir_header; dir_header.count = dir->entry_count - 1; dir_header.start_block = dir->start_block; dir_header.inode_number = dir->inode_number; SQUASHFS_SWAP_DIR_HEADER(&dir_header, dir->entry_count_p); memcpy(directory_data_cache + directory_cache_bytes, dir->buff, dir_size); } directory_offset = directory_cache_bytes; directory_block = directory_bytes; directory_cache_bytes += dir_size; i_count = 0; index = SQUASHFS_METADATA_SIZE - directory_offset; while(1) { while(i_count < dir->i_count && dir->index[i_count].index.index < index) dir->index[i_count++].index.start_block = directory_bytes; index += SQUASHFS_METADATA_SIZE; if(directory_cache_bytes < SQUASHFS_METADATA_SIZE) break; if((directory_size - directory_bytes) < ((SQUASHFS_METADATA_SIZE << 1) + 2)) { void *dt = realloc(directory_table, directory_size + (SQUASHFS_METADATA_SIZE << 1) + 2); if(dt == NULL) MEM_ERROR(); directory_size += SQUASHFS_METADATA_SIZE << 1; directory_table = dt; } c_byte = mangle(directory_table + directory_bytes + BLOCK_OFFSET, directory_data_cache, SQUASHFS_METADATA_SIZE, SQUASHFS_METADATA_SIZE, noI, 0); TRACE("Directory block @ 0x%x, size %d\n", directory_bytes, c_byte); SQUASHFS_SWAP_SHORTS(&c_byte, directory_table + directory_bytes, 1); directory_bytes += SQUASHFS_COMPRESSED_SIZE(c_byte) + BLOCK_OFFSET; total_directory_bytes += SQUASHFS_METADATA_SIZE + BLOCK_OFFSET; memmove(directory_data_cache, directory_data_cache + SQUASHFS_METADATA_SIZE, directory_cache_bytes - SQUASHFS_METADATA_SIZE); directory_cache_bytes -= SQUASHFS_METADATA_SIZE; } create_inode(inode, dir_info, dir_info->dir_ent, SQUASHFS_DIR_TYPE, dir_size + 3, directory_block, directory_offset, NULL, NULL, dir, 0); #ifdef SQUASHFS_TRACE { unsigned char *dirp; int count; TRACE("Directory contents of inode 0x%llx\n", *inode); dirp = dir->buff; while(dirp < dir->p) { char buffer[SQUASHFS_NAME_LEN + 1]; struct squashfs_dir_entry idir, *idirp; struct squashfs_dir_header dirh; SQUASHFS_SWAP_DIR_HEADER((struct squashfs_dir_header *) dirp, &dirh); count = dirh.count + 1; dirp += sizeof(struct squashfs_dir_header); TRACE("\tStart block 0x%x, count %d\n", dirh.start_block, count); while(count--) { idirp = (struct squashfs_dir_entry *) dirp; SQUASHFS_SWAP_DIR_ENTRY(idirp, &idir); strncpy(buffer, idirp->name, idir.size + 1); buffer[idir.size + 1] = '\0'; TRACE("\t\tname %s, inode offset 0x%x, type " "%d\n", buffer, idir.offset, idir.type); dirp += sizeof(struct squashfs_dir_entry) + idir.size + 1; } } } #endif dir_count ++; } static struct file_buffer *get_fragment(struct fragment *fragment) { struct squashfs_fragment_entry *disk_fragment; struct file_buffer *buffer, *compressed_buffer; long long start_block; int res, size, index = fragment->index; char locked; /* * Lookup fragment block in cache. * If the fragment block doesn't exist, then get the compressed version * from the writer cache or off disk, and decompress it. * * This routine has two things which complicate the code: * * 1. Multiple threads can simultaneously lookup/create the * same buffer. This means a buffer needs to be "locked" * when it is being filled in, to prevent other threads from * using it when it is not ready. This is because we now do * fragment duplicate checking in parallel. * 2. We have two caches which need to be checked for the * presence of fragment blocks: the normal fragment cache * and a "reserve" cache. The reserve cache is used to * prevent an unnecessary pipeline stall when the fragment cache * is full of fragments waiting to be compressed. */ if(fragment->index == SQUASHFS_INVALID_FRAG) return NULL; pthread_cleanup_push((void *) pthread_mutex_unlock, &dup_mutex); pthread_mutex_lock(&dup_mutex); again: buffer = cache_lookup_nowait(fragment_buffer, index, &locked); if(buffer) { pthread_mutex_unlock(&dup_mutex); if(locked) /* got a buffer being filled in. Wait for it */ cache_wait_unlock(buffer); goto finished; } /* not in fragment cache, is it in the reserve cache? */ buffer = cache_lookup_nowait(reserve_cache, index, &locked); if(buffer) { pthread_mutex_unlock(&dup_mutex); if(locked) /* got a buffer being filled in. Wait for it */ cache_wait_unlock(buffer); goto finished; } /* in neither cache, try to get it from the fragment cache */ buffer = cache_get_nowait(fragment_buffer, index); if(!buffer) { /* * no room, get it from the reserve cache, this is * dimensioned so it will always have space (no more than * processors + 1 can have an outstanding reserve buffer) */ buffer = cache_get_nowait(reserve_cache, index); if(!buffer) { /* failsafe */ ERROR("no space in reserve cache\n"); goto again; } } pthread_mutex_unlock(&dup_mutex); compressed_buffer = cache_lookup(fwriter_buffer, index); pthread_cleanup_push((void *) pthread_mutex_unlock, &fragment_mutex); pthread_mutex_lock(&fragment_mutex); disk_fragment = &fragment_table[index]; size = SQUASHFS_COMPRESSED_SIZE_BLOCK(disk_fragment->size); start_block = disk_fragment->start_block; pthread_cleanup_pop(1); if(SQUASHFS_COMPRESSED_BLOCK(disk_fragment->size)) { int error; char *data; if(compressed_buffer) data = compressed_buffer->data; else { data = read_from_disk(start_block, size); if(data == NULL) { ERROR("Failed to read fragment from output" " filesystem\n"); BAD_ERROR("Output filesystem corrupted?\n"); } } res = compressor_uncompress(comp, buffer->data, data, size, block_size, &error); if(res == -1) BAD_ERROR("%s uncompress failed with error code %d\n", comp->name, error); } else if(compressed_buffer) memcpy(buffer->data, compressed_buffer->data, size); else { res = read_fs_bytes(fd, start_block, size, buffer->data); if(res == 0) { ERROR("Failed to read fragment from output " "filesystem\n"); BAD_ERROR("Output filesystem corrupted?\n"); } } cache_unlock(buffer); cache_block_put(compressed_buffer); finished: pthread_cleanup_pop(0); return buffer; } unsigned short get_fragment_checksum(struct file_info *file) { struct file_buffer *frag_buffer; struct append_file *append; int res, index = file->fragment->index; unsigned short checksum; if(index == SQUASHFS_INVALID_FRAG) return 0; pthread_cleanup_push((void *) pthread_mutex_unlock, &dup_mutex); pthread_mutex_lock(&dup_mutex); res = file->have_frag_checksum; checksum = file->fragment_checksum; pthread_cleanup_pop(1); if(res) return checksum; frag_buffer = get_fragment(file->fragment); pthread_cleanup_push((void *) pthread_mutex_unlock, &dup_mutex); for(append = file_mapping[index]; append; append = append->next) { int offset = append->file->fragment->offset; int size = append->file->fragment->size; unsigned short cksum = get_checksum_mem(frag_buffer->data + offset, size); if(file == append->file) checksum = cksum; pthread_mutex_lock(&dup_mutex); append->file->fragment_checksum = cksum; append->file->have_frag_checksum = TRUE; pthread_mutex_unlock(&dup_mutex); } cache_block_put(frag_buffer); pthread_cleanup_pop(0); return checksum; } void lock_fragments() { pthread_cleanup_push((void *) pthread_mutex_unlock, &fragment_mutex); pthread_mutex_lock(&fragment_mutex); fragments_locked = TRUE; pthread_cleanup_pop(1); } void unlock_fragments() { int frg, size; struct file_buffer *write_buffer; pthread_cleanup_push((void *) pthread_mutex_unlock, &fragment_mutex); pthread_mutex_lock(&fragment_mutex); /* * Note queue_empty() is inherently racy with respect to concurrent * queue get and pushes. We avoid this because we're holding the * fragment_mutex which ensures no other threads can be using the * queue at this time. */ while(!queue_empty(locked_fragment)) { write_buffer = queue_get(locked_fragment); frg = write_buffer->block; size = SQUASHFS_COMPRESSED_SIZE_BLOCK(fragment_table[frg].size); fragment_table[frg].start_block = bytes; write_buffer->block = bytes; bytes += size; fragments_outstanding --; queue_put(to_writer, write_buffer); TRACE("fragment_locked writing fragment %d, compressed size %d" "\n", frg, size); } fragments_locked = FALSE; pthread_cleanup_pop(1); } /* Called with the fragment_mutex locked */ void add_pending_fragment(struct file_buffer *write_buffer, int c_byte, int fragment) { fragment_table[fragment].size = c_byte; write_buffer->block = fragment; queue_put(locked_fragment, write_buffer); } void write_fragment(struct file_buffer *fragment) { if(fragment == NULL) return; pthread_cleanup_push((void *) pthread_mutex_unlock, &fragment_mutex); pthread_mutex_lock(&fragment_mutex); fragment_table[fragment->block].unused = 0; fragments_outstanding ++; queue_put(to_frag, fragment); pthread_cleanup_pop(1); } struct file_buffer *allocate_fragment() { struct file_buffer *fragment = cache_get(fragment_buffer, fragments); pthread_cleanup_push((void *) pthread_mutex_unlock, &fragment_mutex); pthread_mutex_lock(&fragment_mutex); if(fragments % FRAG_SIZE == 0) { void *ft = realloc(fragment_table, (fragments + FRAG_SIZE) * sizeof(struct squashfs_fragment_entry)); if(ft == NULL) MEM_ERROR(); fragment_table = ft; } fragment->size = 0; fragment->block = fragments ++; pthread_cleanup_pop(1); return fragment; } static struct fragment empty_fragment = {SQUASHFS_INVALID_FRAG, 0, 0}; void free_fragment(struct fragment *fragment) { if(fragment != &empty_fragment) free(fragment); } struct fragment *get_and_fill_fragment(struct file_buffer *file_buffer, struct dir_ent *dir_ent) { struct fragment *ffrg; struct file_buffer **fragment; if(file_buffer == NULL || file_buffer->size == 0) return &empty_fragment; fragment = eval_frag_actions(dir_ent); if((*fragment) && (*fragment)->size + file_buffer->size > block_size) { write_fragment(*fragment); *fragment = NULL; } ffrg = malloc(sizeof(struct fragment)); if(ffrg == NULL) MEM_ERROR(); if(*fragment == NULL) *fragment = allocate_fragment(); ffrg->index = (*fragment)->block; ffrg->offset = (*fragment)->size; ffrg->size = file_buffer->size; memcpy((*fragment)->data + (*fragment)->size, file_buffer->data, file_buffer->size); (*fragment)->size += file_buffer->size; return ffrg; } long long generic_write_table(int length, void *buffer, int length2, void *buffer2, int uncompressed) { int meta_blocks = (length + SQUASHFS_METADATA_SIZE - 1) / SQUASHFS_METADATA_SIZE; long long *list, start_bytes; int compressed_size, i, list_size = meta_blocks * sizeof(long long); unsigned short c_byte; char cbuffer[(SQUASHFS_METADATA_SIZE << 2) + 2]; #ifdef SQUASHFS_TRACE long long obytes = bytes; int olength = length; #endif list = malloc(list_size); if(list == NULL) MEM_ERROR(); for(i = 0; i < meta_blocks; i++) { int avail_bytes = length > SQUASHFS_METADATA_SIZE ? SQUASHFS_METADATA_SIZE : length; c_byte = mangle(cbuffer + BLOCK_OFFSET, buffer + i * SQUASHFS_METADATA_SIZE , avail_bytes, SQUASHFS_METADATA_SIZE, uncompressed, 0); SQUASHFS_SWAP_SHORTS(&c_byte, cbuffer, 1); list[i] = bytes; compressed_size = SQUASHFS_COMPRESSED_SIZE(c_byte) + BLOCK_OFFSET; TRACE("block %d @ 0x%llx, compressed size %d\n", i, bytes, compressed_size); write_destination(fd, bytes, compressed_size, cbuffer); bytes += compressed_size; total_bytes += avail_bytes; length -= avail_bytes; } start_bytes = bytes; if(length2) { write_destination(fd, bytes, length2, buffer2); bytes += length2; total_bytes += length2; } SQUASHFS_INSWAP_LONG_LONGS(list, meta_blocks); write_destination(fd, bytes, list_size, list); bytes += list_size; total_bytes += list_size; TRACE("generic_write_table: total uncompressed %d compressed %lld\n", olength, bytes - obytes); free(list); return start_bytes; } long long write_fragment_table() { unsigned int frag_bytes = SQUASHFS_FRAGMENT_BYTES(fragments); int i; TRACE("write_fragment_table: fragments %d, frag_bytes %d\n", fragments, frag_bytes); for(i = 0; i < fragments; i++) { TRACE("write_fragment_table: fragment %d, start_block 0x%llx, " "size %d\n", i, fragment_table[i].start_block, fragment_table[i].size); SQUASHFS_INSWAP_FRAGMENT_ENTRY(&fragment_table[i]); } return generic_write_table(frag_bytes, fragment_table, 0, NULL, noF); } char read_from_file_buffer[SQUASHFS_FILE_MAX_SIZE]; static char *read_from_disk(long long start, unsigned int avail_bytes) { int res; res = read_fs_bytes(fd, start, avail_bytes, read_from_file_buffer); if(res == 0) return NULL; return read_from_file_buffer; } char read_from_file_buffer2[SQUASHFS_FILE_MAX_SIZE]; char *read_from_disk2(long long start, unsigned int avail_bytes) { int res; res = read_fs_bytes(fd, start, avail_bytes, read_from_file_buffer2); if(res == 0) return NULL; return read_from_file_buffer2; } /* * Compute 16 bit BSD checksum over the data */ unsigned short get_checksum(char *buff, int bytes, unsigned short chksum) { unsigned char *b = (unsigned char *) buff; while(bytes --) { chksum = (chksum & 1) ? (chksum >> 1) | 0x8000 : chksum >> 1; chksum += *b++; } return chksum; } unsigned short get_checksum_disk(long long start, long long l, unsigned int *blocks) { unsigned short chksum = 0; unsigned int bytes; struct file_buffer *write_buffer; int i; for(i = 0; l; i++) { bytes = SQUASHFS_COMPRESSED_SIZE_BLOCK(blocks[i]); if(bytes == 0) /* sparse block */ continue; write_buffer = cache_lookup(bwriter_buffer, start); if(write_buffer) { chksum = get_checksum(write_buffer->data, bytes, chksum); cache_block_put(write_buffer); } else { void *data = read_from_disk(start, bytes); if(data == NULL) { ERROR("Failed to checksum data from output" " filesystem\n"); BAD_ERROR("Output filesystem corrupted?\n"); } chksum = get_checksum(data, bytes, chksum); } l -= bytes; start += bytes; } return chksum; } unsigned short get_checksum_mem(char *buff, int bytes) { return get_checksum(buff, bytes, 0); } unsigned short get_checksum_mem_buffer(struct file_buffer *file_buffer) { if(file_buffer == NULL) return 0; else return get_checksum(file_buffer->data, file_buffer->size, 0); } #define DUP_HASH(a) (a & 0xffff) void add_file(long long start, long long file_size, long long file_bytes, unsigned int *block_listp, int blocks, unsigned int fragment, int offset, int bytes) { struct fragment *frg; unsigned int *block_list = block_listp; struct file_info *dupl_ptr = dupl[DUP_HASH(file_size)]; struct append_file *append_file; struct file_info *file; if(!duplicate_checking || file_size == 0) return; for(; dupl_ptr; dupl_ptr = dupl_ptr->next) { if(file_size != dupl_ptr->file_size) continue; if(blocks != 0 && start != dupl_ptr->start) continue; if(fragment != dupl_ptr->fragment->index) continue; if(fragment != SQUASHFS_INVALID_FRAG && (offset != dupl_ptr->fragment->offset || bytes != dupl_ptr->fragment->size)) continue; return; } frg = malloc(sizeof(struct fragment)); if(frg == NULL) MEM_ERROR(); frg->index = fragment; frg->offset = offset; frg->size = bytes; file = add_non_dup(file_size, file_bytes, block_list, start, frg, 0, 0, FALSE, FALSE); if(fragment == SQUASHFS_INVALID_FRAG) return; append_file = malloc(sizeof(struct append_file)); if(append_file == NULL) MEM_ERROR(); append_file->file = file; append_file->next = file_mapping[fragment]; file_mapping[fragment] = append_file; } int pre_duplicate(long long file_size) { struct file_info *dupl_ptr = dupl[DUP_HASH(file_size)]; for(; dupl_ptr; dupl_ptr = dupl_ptr->next) if(dupl_ptr->file_size == file_size) return TRUE; return FALSE; } struct file_info *add_non_dup(long long file_size, long long bytes, unsigned int *block_list, long long start, struct fragment *fragment, unsigned short checksum, unsigned short fragment_checksum, int checksum_flag, int checksum_frag_flag) { struct file_info *dupl_ptr = malloc(sizeof(struct file_info)); if(dupl_ptr == NULL) MEM_ERROR(); dupl_ptr->file_size = file_size; dupl_ptr->bytes = bytes; dupl_ptr->block_list = block_list; dupl_ptr->start = start; dupl_ptr->fragment = fragment; dupl_ptr->checksum = checksum; dupl_ptr->fragment_checksum = fragment_checksum; dupl_ptr->have_frag_checksum = checksum_frag_flag; dupl_ptr->have_checksum = checksum_flag; pthread_cleanup_push((void *) pthread_mutex_unlock, &dup_mutex); pthread_mutex_lock(&dup_mutex); dupl_ptr->next = dupl[DUP_HASH(file_size)]; dupl[DUP_HASH(file_size)] = dupl_ptr; dup_files ++; pthread_cleanup_pop(1); return dupl_ptr; } struct fragment *frag_duplicate(struct file_buffer *file_buffer, char *dont_put) { struct file_info *dupl_ptr; struct file_buffer *buffer; struct file_info *dupl_start = file_buffer->dupl_start; long long file_size = file_buffer->file_size; unsigned short checksum = file_buffer->checksum; int res; if(file_buffer->duplicate) { TRACE("Found duplicate file, fragment %d, size %d, offset %d, " "checksum 0x%x\n", dupl_start->fragment->index, file_size, dupl_start->fragment->offset, checksum); *dont_put = TRUE; return dupl_start->fragment; } else { *dont_put = FALSE; dupl_ptr = dupl[DUP_HASH(file_size)]; } for(; dupl_ptr && dupl_ptr != dupl_start; dupl_ptr = dupl_ptr->next) { if(file_size == dupl_ptr->file_size && file_size == dupl_ptr->fragment->size) { if(get_fragment_checksum(dupl_ptr) == checksum) { buffer = get_fragment(dupl_ptr->fragment); res = memcmp(file_buffer->data, buffer->data + dupl_ptr->fragment->offset, file_size); cache_block_put(buffer); if(res == 0) break; } } } if(!dupl_ptr || dupl_ptr == dupl_start) return NULL; TRACE("Found duplicate file, fragment %d, size %d, offset %d, " "checksum 0x%x\n", dupl_ptr->fragment->index, file_size, dupl_ptr->fragment->offset, checksum); return dupl_ptr->fragment; } struct file_info *duplicate(long long file_size, long long bytes, unsigned int **block_list, long long *start, struct fragment **fragment, struct file_buffer *file_buffer, int blocks, unsigned short checksum, int checksum_flag) { struct file_info *dupl_ptr = dupl[DUP_HASH(file_size)]; int frag_bytes = file_buffer ? file_buffer->size : 0; unsigned short fragment_checksum = file_buffer ? file_buffer->checksum : 0; for(; dupl_ptr; dupl_ptr = dupl_ptr->next) if(file_size == dupl_ptr->file_size && bytes == dupl_ptr->bytes && frag_bytes == dupl_ptr->fragment->size) { long long target_start, dup_start = dupl_ptr->start; int block; if(memcmp(*block_list, dupl_ptr->block_list, blocks * sizeof(unsigned int)) != 0) continue; if(checksum_flag == FALSE) { checksum = get_checksum_disk(*start, bytes, *block_list); checksum_flag = TRUE; } if(!dupl_ptr->have_checksum) { dupl_ptr->checksum = get_checksum_disk(dupl_ptr->start, dupl_ptr->bytes, dupl_ptr->block_list); dupl_ptr->have_checksum = TRUE; } if(checksum != dupl_ptr->checksum || fragment_checksum != get_fragment_checksum(dupl_ptr)) continue; target_start = *start; for(block = 0; block < blocks; block ++) { int size = SQUASHFS_COMPRESSED_SIZE_BLOCK ((*block_list)[block]); struct file_buffer *target_buffer = NULL; struct file_buffer *dup_buffer = NULL; char *target_data, *dup_data; int res; if(size == 0) continue; target_buffer = cache_lookup(bwriter_buffer, target_start); if(target_buffer) target_data = target_buffer->data; else { target_data = read_from_disk(target_start, size); if(target_data == NULL) { ERROR("Failed to read data from" " output filesystem\n"); BAD_ERROR("Output filesystem" " corrupted?\n"); } } dup_buffer = cache_lookup(bwriter_buffer, dup_start); if(dup_buffer) dup_data = dup_buffer->data; else { dup_data = read_from_disk2(dup_start, size); if(dup_data == NULL) { ERROR("Failed to read data from" " output filesystem\n"); BAD_ERROR("Output filesystem" " corrupted?\n"); } } res = memcmp(target_data, dup_data, size); cache_block_put(target_buffer); cache_block_put(dup_buffer); if(res != 0) break; target_start += size; dup_start += size; } if(block == blocks) { struct file_buffer *frag_buffer = get_fragment(dupl_ptr->fragment); if(frag_bytes == 0 || memcmp(file_buffer->data, frag_buffer->data + dupl_ptr->fragment->offset, frag_bytes) == 0) { TRACE("Found duplicate file, start " "0x%llx, size %lld, checksum " "0x%x, fragment %d, size %d, " "offset %d, checksum 0x%x\n", dupl_ptr->start, dupl_ptr->bytes, dupl_ptr->checksum, dupl_ptr->fragment->index, frag_bytes, dupl_ptr->fragment->offset, fragment_checksum); *block_list = dupl_ptr->block_list; *start = dupl_ptr->start; *fragment = dupl_ptr->fragment; cache_block_put(frag_buffer); return 0; } cache_block_put(frag_buffer); } } return add_non_dup(file_size, bytes, *block_list, *start, *fragment, checksum, fragment_checksum, checksum_flag, TRUE); } inline int is_fragment(struct inode_info *inode) { int file_size = inode->buf.st_size; /* * If this block is to be compressed differently to the * fragment compression then it cannot be a fragment */ if(inode->noF != noF) return FALSE; return !inode->no_fragments && file_size && (file_size < block_size || (inode->always_use_fragments && file_size & (block_size - 1))); } void put_file_buffer(struct file_buffer *file_buffer) { /* * Decide where to send the file buffer: * - compressible non-fragment blocks go to the deflate threads, * - fragments go to the process fragment threads, * - all others go directly to the main thread */ if(file_buffer->error) { file_buffer->fragment = 0; seq_queue_put(to_main, file_buffer); } else if (file_buffer->file_size == 0) seq_queue_put(to_main, file_buffer); else if(file_buffer->fragment) queue_put(to_process_frag, file_buffer); else queue_put(to_deflate, file_buffer); } static int seq = 0; void reader_read_process(struct dir_ent *dir_ent) { long long bytes = 0; struct inode_info *inode = dir_ent->inode; struct file_buffer *prev_buffer = NULL, *file_buffer; int status, byte, res, child; int file = pseudo_exec_file(get_pseudo_file(inode->pseudo_id), &child); if(!file) { file_buffer = cache_get_nohash(reader_buffer); file_buffer->sequence = seq ++; goto read_err; } while(1) { file_buffer = cache_get_nohash(reader_buffer); file_buffer->sequence = seq ++; file_buffer->noD = inode->noD; byte = read_bytes(file, file_buffer->data, block_size); if(byte == -1) goto read_err2; file_buffer->size = byte; file_buffer->file_size = -1; file_buffer->error = FALSE; file_buffer->fragment = FALSE; bytes += byte; if(byte == 0) break; /* * Update progress bar size. This is done * on every block rather than waiting for all blocks to be * read incase write_file_process() is running in parallel * with this. Otherwise the current progress bar position * may get ahead of the progress bar size. */ progress_bar_size(1); if(prev_buffer) put_file_buffer(prev_buffer); prev_buffer = file_buffer; } /* * Update inode file size now that the size of the dynamic pseudo file * is known. This is needed for the -info option. */ inode->buf.st_size = bytes; res = waitpid(child, &status, 0); close(file); if(res == -1 || !WIFEXITED(status) || WEXITSTATUS(status) != 0) goto read_err; if(prev_buffer == NULL) prev_buffer = file_buffer; else { cache_block_put(file_buffer); seq --; } prev_buffer->file_size = bytes; prev_buffer->fragment = is_fragment(inode); put_file_buffer(prev_buffer); return; read_err2: close(file); read_err: if(prev_buffer) { cache_block_put(file_buffer); seq --; file_buffer = prev_buffer; } file_buffer->error = TRUE; put_file_buffer(file_buffer); } void reader_read_file(struct dir_ent *dir_ent) { struct stat *buf = &dir_ent->inode->buf, buf2; struct file_buffer *file_buffer; int blocks, file, res; long long bytes, read_size; struct inode_info *inode = dir_ent->inode; if(inode->read) return; inode->read = TRUE; again: bytes = 0; read_size = buf->st_size; blocks = (read_size + block_size - 1) >> block_log; file = open(pathname_reader(dir_ent), O_RDONLY); if(file == -1) { file_buffer = cache_get_nohash(reader_buffer); file_buffer->sequence = seq ++; goto read_err2; } do { file_buffer = cache_get_nohash(reader_buffer); file_buffer->file_size = read_size; file_buffer->sequence = seq ++; file_buffer->noD = inode->noD; file_buffer->error = FALSE; /* * Always try to read block_size bytes from the file rather * than expected bytes (which will be less than the block_size * at the file tail) to check that the file hasn't grown * since being stated. If it is longer (or shorter) than * expected, then restat, and try again. Note the special * case where the file is an exact multiple of the block_size * is dealt with later. */ file_buffer->size = read_bytes(file, file_buffer->data, block_size); if(file_buffer->size == -1) goto read_err; bytes += file_buffer->size; if(blocks > 1) { /* non-tail block should be exactly block_size */ if(file_buffer->size < block_size) goto restat; file_buffer->fragment = FALSE; put_file_buffer(file_buffer); } } while(-- blocks > 0); /* Overall size including tail should match */ if(read_size != bytes) goto restat; if(read_size && read_size % block_size == 0) { /* * Special case where we've not tried to read past the end of * the file. We expect to get EOF, i.e. the file isn't larger * than we expect. */ char buffer; int res; res = read_bytes(file, &buffer, 1); if(res == -1) goto read_err; if(res != 0) goto restat; } file_buffer->fragment = is_fragment(inode); put_file_buffer(file_buffer); close(file); return; restat: res = fstat(file, &buf2); if(res == -1) { ERROR("Cannot stat dir/file %s because %s\n", pathname_reader(dir_ent), strerror(errno)); goto read_err; } if(read_size != buf2.st_size) { close(file); memcpy(buf, &buf2, sizeof(struct stat)); file_buffer->error = 2; put_file_buffer(file_buffer); goto again; } read_err: close(file); read_err2: file_buffer->error = TRUE; put_file_buffer(file_buffer); } void reader_scan(struct dir_info *dir) { struct dir_ent *dir_ent = dir->list; for(; dir_ent; dir_ent = dir_ent->next) { struct stat *buf = &dir_ent->inode->buf; if(dir_ent->inode->root_entry) continue; if(IS_PSEUDO_PROCESS(dir_ent->inode)) { reader_read_process(dir_ent); continue; } switch(buf->st_mode & S_IFMT) { case S_IFREG: reader_read_file(dir_ent); break; case S_IFDIR: reader_scan(dir_ent->dir); break; } } } void *reader(void *arg) { if(!sorted) reader_scan(queue_get(to_reader)); else { int i; struct priority_entry *entry; queue_get(to_reader); for(i = 65535; i >= 0; i--) for(entry = priority_list[i]; entry; entry = entry->next) reader_read_file(entry->dir); } pthread_exit(NULL); } void *writer(void *arg) { while(1) { struct file_buffer *file_buffer = queue_get(to_writer); off_t off; if(file_buffer == NULL) { queue_put(from_writer, NULL); continue; } off = file_buffer->block; pthread_cleanup_push((void *) pthread_mutex_unlock, &pos_mutex); pthread_mutex_lock(&pos_mutex); if(lseek(fd, off, SEEK_SET) == -1) { ERROR("writer: Lseek on destination failed because " "%s, offset=0x%llx\n", strerror(errno), off); BAD_ERROR("Probably out of space on output " "%s\n", block_device ? "block device" : "filesystem"); } if(write_bytes(fd, file_buffer->data, file_buffer->size) == -1) BAD_ERROR("Failed to write to output %s\n", block_device ? "block device" : "filesystem"); pthread_cleanup_pop(1); cache_block_put(file_buffer); } } int all_zero(struct file_buffer *file_buffer) { int i; long entries = file_buffer->size / sizeof(long); long *p = (long *) file_buffer->data; for(i = 0; i < entries && p[i] == 0; i++); if(i == entries) { for(i = file_buffer->size & ~(sizeof(long) - 1); i < file_buffer->size && file_buffer->data[i] == 0; i++); return i == file_buffer->size; } return 0; } void *deflator(void *arg) { struct file_buffer *write_buffer = cache_get_nohash(bwriter_buffer); void *stream = NULL; int res; res = compressor_init(comp, &stream, block_size, 1); if(res) BAD_ERROR("deflator:: compressor_init failed\n"); while(1) { struct file_buffer *file_buffer = queue_get(to_deflate); if(sparse_files && all_zero(file_buffer)) { file_buffer->c_byte = 0; seq_queue_put(to_main, file_buffer); } else { write_buffer->c_byte = mangle2(stream, write_buffer->data, file_buffer->data, file_buffer->size, block_size, file_buffer->noD, 1); write_buffer->sequence = file_buffer->sequence; write_buffer->file_size = file_buffer->file_size; write_buffer->block = file_buffer->block; write_buffer->size = SQUASHFS_COMPRESSED_SIZE_BLOCK (write_buffer->c_byte); write_buffer->fragment = FALSE; write_buffer->error = FALSE; cache_block_put(file_buffer); seq_queue_put(to_main, write_buffer); write_buffer = cache_get_nohash(bwriter_buffer); } } } void *frag_deflator(void *arg) { void *stream = NULL; int res; res = compressor_init(comp, &stream, block_size, 1); if(res) BAD_ERROR("frag_deflator:: compressor_init failed\n"); pthread_cleanup_push((void *) pthread_mutex_unlock, &fragment_mutex); while(1) { int c_byte, compressed_size; struct file_buffer *file_buffer = queue_get(to_frag); struct file_buffer *write_buffer = cache_get(fwriter_buffer, file_buffer->block); c_byte = mangle2(stream, write_buffer->data, file_buffer->data, file_buffer->size, block_size, noF, 1); compressed_size = SQUASHFS_COMPRESSED_SIZE_BLOCK(c_byte); write_buffer->size = compressed_size; pthread_mutex_lock(&fragment_mutex); if(fragments_locked == FALSE) { fragment_table[file_buffer->block].size = c_byte; fragment_table[file_buffer->block].start_block = bytes; write_buffer->block = bytes; bytes += compressed_size; fragments_outstanding --; pthread_mutex_unlock(&fragment_mutex); queue_put(to_writer, write_buffer); TRACE("Writing fragment %lld, uncompressed size %d, " "compressed size %d\n", file_buffer->block, file_buffer->size, compressed_size); } else { add_pending_fragment(write_buffer, c_byte, file_buffer->block); pthread_mutex_unlock(&fragment_mutex); } cache_block_put(file_buffer); } pthread_cleanup_pop(0); } struct file_buffer *get_file_buffer() { struct file_buffer *file_buffer = seq_queue_get(to_main); return file_buffer; } void write_file_empty(squashfs_inode *inode, struct dir_ent *dir_ent, struct file_buffer *file_buffer, int *duplicate_file) { file_count ++; *duplicate_file = FALSE; cache_block_put(file_buffer); create_inode(inode, NULL, dir_ent, SQUASHFS_FILE_TYPE, 0, 0, 0, NULL, &empty_fragment, NULL, 0); } void write_file_frag(squashfs_inode *inode, struct dir_ent *dir_ent, struct file_buffer *file_buffer, int *duplicate_file) { int size = file_buffer->file_size; struct fragment *fragment; unsigned short checksum = file_buffer->checksum; char dont_put; fragment = frag_duplicate(file_buffer, &dont_put); *duplicate_file = !fragment; if(!fragment) { fragment = get_and_fill_fragment(file_buffer, dir_ent); if(duplicate_checking) add_non_dup(size, 0, NULL, 0, fragment, 0, checksum, TRUE, TRUE); } if(dont_put) free(file_buffer); else cache_block_put(file_buffer); total_bytes += size; file_count ++; inc_progress_bar(); create_inode(inode, NULL, dir_ent, SQUASHFS_FILE_TYPE, size, 0, 0, NULL, fragment, NULL, 0); if(!duplicate_checking) free_fragment(fragment); } int write_file_process(squashfs_inode *inode, struct dir_ent *dir_ent, struct file_buffer *read_buffer, int *duplicate_file) { long long read_size, file_bytes, start; struct fragment *fragment; unsigned int *block_list = NULL; int block = 0, status; long long sparse = 0; struct file_buffer *fragment_buffer = NULL; *duplicate_file = FALSE; lock_fragments(); file_bytes = 0; start = bytes; while (1) { read_size = read_buffer->file_size; if(read_buffer->fragment) fragment_buffer = read_buffer; else { block_list = realloc(block_list, (block + 1) * sizeof(unsigned int)); if(block_list == NULL) MEM_ERROR(); block_list[block ++] = read_buffer->c_byte; if(read_buffer->c_byte) { read_buffer->block = bytes; bytes += read_buffer->size; cache_hash(read_buffer, read_buffer->block); file_bytes += read_buffer->size; queue_put(to_writer, read_buffer); } else { sparse += read_buffer->size; cache_block_put(read_buffer); } } inc_progress_bar(); if(read_size != -1) break; read_buffer = get_file_buffer(); if(read_buffer->error) goto read_err; } unlock_fragments(); fragment = get_and_fill_fragment(fragment_buffer, dir_ent); if(duplicate_checking) add_non_dup(read_size, file_bytes, block_list, start, fragment, 0, fragment_buffer ? fragment_buffer->checksum : 0, FALSE, TRUE); cache_block_put(fragment_buffer); file_count ++; total_bytes += read_size; create_inode(inode, NULL, dir_ent, SQUASHFS_FILE_TYPE, read_size, start, block, block_list, fragment, NULL, sparse); if(duplicate_checking == FALSE) { free(block_list); free_fragment(fragment); } return 0; read_err: dec_progress_bar(block); status = read_buffer->error; bytes = start; if(!block_device) { int res; queue_put(to_writer, NULL); if(queue_get(from_writer) != 0) EXIT_MKSQUASHFS(); res = ftruncate(fd, bytes); if(res != 0) BAD_ERROR("Failed to truncate dest file because %s\n", strerror(errno)); } unlock_fragments(); free(block_list); cache_block_put(read_buffer); return status; } int write_file_blocks_dup(squashfs_inode *inode, struct dir_ent *dir_ent, struct file_buffer *read_buffer, int *duplicate_file) { int block, thresh; long long read_size = read_buffer->file_size; long long file_bytes, dup_start, start; struct fragment *fragment; struct file_info *dupl_ptr; int blocks = (read_size + block_size - 1) >> block_log; unsigned int *block_list, *block_listp; struct file_buffer **buffer_list; int status; long long sparse = 0; struct file_buffer *fragment_buffer = NULL; block_list = malloc(blocks * sizeof(unsigned int)); if(block_list == NULL) MEM_ERROR(); block_listp = block_list; buffer_list = malloc(blocks * sizeof(struct file_buffer *)); if(buffer_list == NULL) MEM_ERROR(); lock_fragments(); file_bytes = 0; start = dup_start = bytes; thresh = blocks > bwriter_size ? blocks - bwriter_size : 0; for(block = 0; block < blocks;) { if(read_buffer->fragment) { block_list[block] = 0; buffer_list[block] = NULL; fragment_buffer = read_buffer; blocks = read_size >> block_log; } else { block_list[block] = read_buffer->c_byte; if(read_buffer->c_byte) { read_buffer->block = bytes; bytes += read_buffer->size; file_bytes += read_buffer->size; cache_hash(read_buffer, read_buffer->block); if(block < thresh) { buffer_list[block] = NULL; queue_put(to_writer, read_buffer); } else buffer_list[block] = read_buffer; } else { buffer_list[block] = NULL; sparse += read_buffer->size; cache_block_put(read_buffer); } } inc_progress_bar(); if(++block < blocks) { read_buffer = get_file_buffer(); if(read_buffer->error) goto read_err; } } dupl_ptr = duplicate(read_size, file_bytes, &block_listp, &dup_start, &fragment, fragment_buffer, blocks, 0, FALSE); if(dupl_ptr) { *duplicate_file = FALSE; for(block = thresh; block < blocks; block ++) if(buffer_list[block]) queue_put(to_writer, buffer_list[block]); fragment = get_and_fill_fragment(fragment_buffer, dir_ent); dupl_ptr->fragment = fragment; } else { *duplicate_file = TRUE; for(block = thresh; block < blocks; block ++) cache_block_put(buffer_list[block]); bytes = start; if(thresh && !block_device) { int res; queue_put(to_writer, NULL); if(queue_get(from_writer) != 0) EXIT_MKSQUASHFS(); res = ftruncate(fd, bytes); if(res != 0) BAD_ERROR("Failed to truncate dest file because" " %s\n", strerror(errno)); } } unlock_fragments(); cache_block_put(fragment_buffer); free(buffer_list); file_count ++; total_bytes += read_size; /* * sparse count is needed to ensure squashfs correctly reports a * a smaller block count on stat calls to sparse files. This is * to ensure intelligent applications like cp correctly handle the * file as a sparse file. If the file in the original filesystem isn't * stored as a sparse file then still store it sparsely in squashfs, but * report it as non-sparse on stat calls to preserve semantics */ if(sparse && (dir_ent->inode->buf.st_blocks << 9) >= read_size) sparse = 0; create_inode(inode, NULL, dir_ent, SQUASHFS_FILE_TYPE, read_size, dup_start, blocks, block_listp, fragment, NULL, sparse); if(*duplicate_file == TRUE) free(block_list); return 0; read_err: dec_progress_bar(block); status = read_buffer->error; bytes = start; if(thresh && !block_device) { int res; queue_put(to_writer, NULL); if(queue_get(from_writer) != 0) EXIT_MKSQUASHFS(); res = ftruncate(fd, bytes); if(res != 0) BAD_ERROR("Failed to truncate dest file because %s\n", strerror(errno)); } unlock_fragments(); for(blocks = thresh; blocks < block; blocks ++) cache_block_put(buffer_list[blocks]); free(buffer_list); free(block_list); cache_block_put(read_buffer); return status; } int write_file_blocks(squashfs_inode *inode, struct dir_ent *dir_ent, struct file_buffer *read_buffer, int *dup) { long long read_size = read_buffer->file_size; long long file_bytes, start; struct fragment *fragment; unsigned int *block_list; int block, status; int blocks = (read_size + block_size - 1) >> block_log; long long sparse = 0; struct file_buffer *fragment_buffer = NULL; if(pre_duplicate(read_size)) return write_file_blocks_dup(inode, dir_ent, read_buffer, dup); *dup = FALSE; block_list = malloc(blocks * sizeof(unsigned int)); if(block_list == NULL) MEM_ERROR(); lock_fragments(); file_bytes = 0; start = bytes; for(block = 0; block < blocks;) { if(read_buffer->fragment) { block_list[block] = 0; fragment_buffer = read_buffer; blocks = read_size >> block_log; } else { block_list[block] = read_buffer->c_byte; if(read_buffer->c_byte) { read_buffer->block = bytes; bytes += read_buffer->size; cache_hash(read_buffer, read_buffer->block); file_bytes += read_buffer->size; queue_put(to_writer, read_buffer); } else { sparse += read_buffer->size; cache_block_put(read_buffer); } } inc_progress_bar(); if(++block < blocks) { read_buffer = get_file_buffer(); if(read_buffer->error) goto read_err; } } unlock_fragments(); fragment = get_and_fill_fragment(fragment_buffer, dir_ent); if(duplicate_checking) add_non_dup(read_size, file_bytes, block_list, start, fragment, 0, fragment_buffer ? fragment_buffer->checksum : 0, FALSE, TRUE); cache_block_put(fragment_buffer); file_count ++; total_bytes += read_size; /* * sparse count is needed to ensure squashfs correctly reports a * a smaller block count on stat calls to sparse files. This is * to ensure intelligent applications like cp correctly handle the * file as a sparse file. If the file in the original filesystem isn't * stored as a sparse file then still store it sparsely in squashfs, but * report it as non-sparse on stat calls to preserve semantics */ if(sparse && (dir_ent->inode->buf.st_blocks << 9) >= read_size) sparse = 0; create_inode(inode, NULL, dir_ent, SQUASHFS_FILE_TYPE, read_size, start, blocks, block_list, fragment, NULL, sparse); if(duplicate_checking == FALSE) { free(block_list); free_fragment(fragment); } return 0; read_err: dec_progress_bar(block); status = read_buffer->error; bytes = start; if(!block_device) { int res; queue_put(to_writer, NULL); if(queue_get(from_writer) != 0) EXIT_MKSQUASHFS(); res = ftruncate(fd, bytes); if(res != 0) BAD_ERROR("Failed to truncate dest file because %s\n", strerror(errno)); } unlock_fragments(); free(block_list); cache_block_put(read_buffer); return status; } void write_file(squashfs_inode *inode, struct dir_ent *dir, int *dup) { int status; struct file_buffer *read_buffer; again: read_buffer = get_file_buffer(); status = read_buffer->error; if(status) cache_block_put(read_buffer); else if(read_buffer->file_size == -1) status = write_file_process(inode, dir, read_buffer, dup); else if(read_buffer->file_size == 0) write_file_empty(inode, dir, read_buffer, dup); else if(read_buffer->fragment && read_buffer->c_byte) write_file_frag(inode, dir, read_buffer, dup); else status = write_file_blocks(inode, dir, read_buffer, dup); if(status == 2) { ERROR("File %s changed size while reading filesystem, " "attempting to re-read\n", pathname(dir)); goto again; } else if(status == 1) { ERROR_START("Failed to read file %s", pathname(dir)); ERROR_EXIT(", creating empty file\n"); write_file_empty(inode, dir, NULL, dup); } } #define BUFF_SIZE 512 char *name; char *basename_r(); char *getbase(char *pathname) { static char *b_buffer = NULL; static int b_size = BUFF_SIZE; char *result; if(b_buffer == NULL) { b_buffer = malloc(b_size); if(b_buffer == NULL) MEM_ERROR(); } while(1) { if(*pathname != '/') { result = getcwd(b_buffer, b_size); if(result == NULL && errno != ERANGE) BAD_ERROR("Getcwd failed in getbase\n"); /* enough room for pathname + "/" + '\0' terminator? */ if(result && strlen(pathname) + 2 <= b_size - strlen(b_buffer)) { strcat(strcat(b_buffer, "/"), pathname); break; } } else if(strlen(pathname) < b_size) { strcpy(b_buffer, pathname); break; } /* Buffer not large enough, realloc and try again */ b_buffer = realloc(b_buffer, b_size += BUFF_SIZE); if(b_buffer == NULL) MEM_ERROR(); } name = b_buffer; if(((result = basename_r()) == NULL) || (strcmp(result, "..") == 0)) return NULL; else return result; } char *basename_r() { char *s; char *p; int n = 1; for(;;) { s = name; if(*name == '\0') return NULL; if(*name != '/') { while(*name != '\0' && *name != '/') name++; n = name - s; } while(*name == '/') name++; if(strncmp(s, ".", n) == 0) continue; if((*name == '\0') || (strncmp(s, "..", n) == 0) || ((p = basename_r()) == NULL)) { s[n] = '\0'; return s; } if(strcmp(p, "..") == 0) continue; return p; } } struct inode_info *lookup_inode2(struct stat *buf, int pseudo, int id) { int ino_hash = INODE_HASH(buf->st_dev, buf->st_ino); struct inode_info *inode; /* * Look-up inode in hash table, if it already exists we have a * hard-link, so increment the nlink count and return it. * Don't do the look-up for directories because we don't hard-link * directories. */ if ((buf->st_mode & S_IFMT) != S_IFDIR) { for(inode = inode_info[ino_hash]; inode; inode = inode->next) { if(memcmp(buf, &inode->buf, sizeof(struct stat)) == 0) { inode->nlink ++; return inode; } } } inode = malloc(sizeof(struct inode_info)); if(inode == NULL) MEM_ERROR(); memcpy(&inode->buf, buf, sizeof(struct stat)); inode->read = FALSE; inode->root_entry = FALSE; inode->pseudo_file = pseudo; inode->pseudo_id = id; inode->inode = SQUASHFS_INVALID_BLK; inode->nlink = 1; inode->inode_number = 0; /* * Copy filesystem wide defaults into inode, these filesystem * wide defaults may be altered on an individual inode basis by * user specified actions * */ inode->no_fragments = no_fragments; inode->always_use_fragments = always_use_fragments; inode->noD = noD; inode->noF = noF; if((buf->st_mode & S_IFMT) == S_IFREG) progress_bar_size((buf->st_size + block_size - 1) >> block_log); inode->next = inode_info[ino_hash]; inode_info[ino_hash] = inode; return inode; } inline struct inode_info *lookup_inode(struct stat *buf) { return lookup_inode2(buf, 0, 0); } inline void alloc_inode_no(struct inode_info *inode, unsigned int use_this) { if (inode->inode_number == 0) inode->inode_number = use_this ? : inode_no ++; } inline struct dir_ent *create_dir_entry(char *name, char *source_name, char *nonstandard_pathname, struct dir_info *dir) { struct dir_ent *dir_ent = malloc(sizeof(struct dir_ent)); if(dir_ent == NULL) MEM_ERROR(); dir_ent->name = name; dir_ent->source_name = source_name; dir_ent->nonstandard_pathname = nonstandard_pathname; dir_ent->our_dir = dir; dir_ent->next = NULL; return dir_ent; } inline void add_dir_entry(struct dir_ent *dir_ent, struct dir_info *sub_dir, struct inode_info *inode_info) { struct dir_info *dir = dir_ent->our_dir; if(sub_dir) sub_dir->dir_ent = dir_ent; dir_ent->inode = inode_info; dir_ent->dir = sub_dir; dir_ent->next = dir->list; dir->list = dir_ent; dir->count++; } inline void add_dir_entry2(char *name, char *source_name, char *nonstandard_pathname, struct dir_info *sub_dir, struct inode_info *inode_info, struct dir_info *dir) { struct dir_ent *dir_ent = create_dir_entry(name, source_name, nonstandard_pathname, dir); add_dir_entry(dir_ent, sub_dir, inode_info); } inline void free_dir_entry(struct dir_ent *dir_ent) { if(dir_ent->name) free(dir_ent->name); if(dir_ent->source_name) free(dir_ent->source_name); free(dir_ent); } inline void add_excluded(struct dir_info *dir) { dir->excluded ++; } void dir_scan(squashfs_inode *inode, char *pathname, struct dir_ent *(_readdir)(struct dir_info *), int progress) { struct stat buf; struct dir_info *dir_info = dir_scan1(pathname, "", paths, _readdir, 1); struct dir_ent *dir_ent; if(dir_info == NULL) return; /* * Process most actions and any pseudo files */ if(actions() || get_pseudo()) dir_scan2(dir_info, get_pseudo()); /* * Process move actions */ if(move_actions()) { dir_scan3(dir_info, dir_info); do_move_actions(); } /* * Process empty actions */ if(empty_actions()) dir_scan4(dir_info); /* * Sort directories and compute the inode numbers */ dir_scan5(dir_info); dir_ent = create_dir_entry("", NULL, pathname, scan1_opendir("", "", 0)); if(pathname[0] == '\0') { /* * dummy top level directory, if multiple sources specified on * command line */ memset(&buf, 0, sizeof(buf)); buf.st_mode = S_IRWXU | S_IRWXG | S_IRWXO | S_IFDIR; buf.st_uid = getuid(); buf.st_gid = getgid(); buf.st_mtime = time(NULL); buf.st_dev = 0; buf.st_ino = 0; dir_ent->inode = lookup_inode2(&buf, PSEUDO_FILE_OTHER, 0); } else { if(lstat(pathname, &buf) == -1) /* source directory has disappeared? */ BAD_ERROR("Cannot stat source directory %s because %s\n", pathname, strerror(errno)); dir_ent->inode = lookup_inode(&buf); } alloc_inode_no(dir_ent->inode, root_inode_number); dir_ent->dir = dir_info; dir_info->dir_ent = dir_ent; eval_actions(dir_ent); if(sorted) generate_file_priorities(dir_info, 0, &dir_info->dir_ent->inode->buf); if(appending) { sigset_t sigmask; restore_thread = init_restore_thread(); sigemptyset(&sigmask); sigaddset(&sigmask, SIGINT); sigaddset(&sigmask, SIGTERM); sigaddset(&sigmask, SIGUSR1); if(pthread_sigmask(SIG_BLOCK, &sigmask, NULL) == -1) BAD_ERROR("Failed to set signal mask\n"); write_destination(fd, SQUASHFS_START, 4, "\0\0\0\0"); } queue_put(to_reader, dir_info); if(sorted) sort_files_and_write(dir_info); set_progressbar_state(progress); dir_scan6(inode, dir_info); dir_ent->inode->inode = *inode; dir_ent->inode->type = SQUASHFS_DIR_TYPE; } /* * dir_scan1 routines... * These scan the source directories into memory for processing. * Exclude actions are processed here (in contrast to the other actions) * because they affect what is scanned. */ struct dir_info *scan1_opendir(char *pathname, char *subpath, int depth) { struct dir_info *dir; dir = malloc(sizeof(struct dir_info)); if(dir == NULL) MEM_ERROR(); if(pathname[0] != '\0') { dir->linuxdir = opendir(pathname); if(dir->linuxdir == NULL) { free(dir); return NULL; } } dir->pathname = strdup(pathname); dir->subpath = strdup(subpath); dir->count = 0; dir->directory_count = 0; dir->dir_is_ldir = TRUE; dir->list = NULL; dir->depth = depth; dir->excluded = 0; return dir; } struct dir_ent *scan1_encomp_readdir(struct dir_info *dir) { static int index = 0; if(dir->count < old_root_entries) { int i; for(i = 0; i < old_root_entries; i++) { if(old_root_entry[i].inode.type == SQUASHFS_DIR_TYPE) dir->directory_count ++; add_dir_entry2(old_root_entry[i].name, NULL, NULL, NULL, &old_root_entry[i].inode, dir); } } while(index < source) { char *basename = NULL; char *dir_name = getbase(source_path[index]); int pass = 1, res; if(dir_name == NULL) { ERROR_START("Bad source directory %s", source_path[index]); ERROR_EXIT(" - skipping ...\n"); index ++; continue; } dir_name = strdup(dir_name); for(;;) { struct dir_ent *dir_ent = dir->list; for(; dir_ent && strcmp(dir_ent->name, dir_name) != 0; dir_ent = dir_ent->next); if(dir_ent == NULL) break; ERROR("Source directory entry %s already used! - trying" " ", dir_name); if(pass == 1) basename = dir_name; else free(dir_name); res = asprintf(&dir_name, "%s_%d", basename, pass++); if(res == -1) BAD_ERROR("asprintf failed in " "scan1_encomp_readdir\n"); ERROR("%s\n", dir_name); } return create_dir_entry(dir_name, basename, source_path[index ++], dir); } return NULL; } struct dir_ent *scan1_single_readdir(struct dir_info *dir) { struct dirent *d_name; int i; if(dir->count < old_root_entries) { for(i = 0; i < old_root_entries; i++) { if(old_root_entry[i].inode.type == SQUASHFS_DIR_TYPE) dir->directory_count ++; add_dir_entry2(old_root_entry[i].name, NULL, NULL, NULL, &old_root_entry[i].inode, dir); } } if((d_name = readdir(dir->linuxdir)) != NULL) { char *basename = NULL; char *dir_name = strdup(d_name->d_name); int pass = 1, res; for(;;) { struct dir_ent *dir_ent = dir->list; for(; dir_ent && strcmp(dir_ent->name, dir_name) != 0; dir_ent = dir_ent->next); if(dir_ent == NULL) break; ERROR("Source directory entry %s already used! - trying" " ", dir_name); if (pass == 1) basename = dir_name; else free(dir_name); res = asprintf(&dir_name, "%s_%d", d_name->d_name, pass++); if(res == -1) BAD_ERROR("asprintf failed in " "scan1_single_readdir\n"); ERROR("%s\n", dir_name); } return create_dir_entry(dir_name, basename, NULL, dir); } return NULL; } struct dir_ent *scan1_readdir(struct dir_info *dir) { struct dirent *d_name = readdir(dir->linuxdir); return d_name ? create_dir_entry(strdup(d_name->d_name), NULL, NULL, dir) : NULL; } void scan1_freedir(struct dir_info *dir) { if(dir->pathname[0] != '\0') closedir(dir->linuxdir); } struct dir_info *dir_scan1(char *filename, char *subpath, struct pathnames *paths, struct dir_ent *(_readdir)(struct dir_info *), int depth) { struct dir_info *dir = scan1_opendir(filename, subpath, depth); struct dir_ent *dir_ent; if(dir == NULL) { ERROR_START("Could not open %s", filename); ERROR_EXIT(", skipping...\n"); return NULL; } while((dir_ent = _readdir(dir))) { struct dir_info *sub_dir; struct stat buf; struct pathnames *new = NULL; char *filename = pathname(dir_ent); char *subpath = NULL; char *dir_name = dir_ent->name; if(strcmp(dir_name, ".") == 0 || strcmp(dir_name, "..") == 0) { free_dir_entry(dir_ent); continue; } if(lstat(filename, &buf) == -1) { ERROR_START("Cannot stat dir/file %s because %s", filename, strerror(errno)); ERROR_EXIT(", ignoring\n"); free_dir_entry(dir_ent); continue; } if((buf.st_mode & S_IFMT) != S_IFREG && (buf.st_mode & S_IFMT) != S_IFDIR && (buf.st_mode & S_IFMT) != S_IFLNK && (buf.st_mode & S_IFMT) != S_IFCHR && (buf.st_mode & S_IFMT) != S_IFBLK && (buf.st_mode & S_IFMT) != S_IFIFO && (buf.st_mode & S_IFMT) != S_IFSOCK) { ERROR_START("File %s has unrecognised filetype %d", filename, buf.st_mode & S_IFMT); ERROR_EXIT(", ignoring\n"); free_dir_entry(dir_ent); continue; } if((old_exclude && old_excluded(filename, &buf)) || (!old_exclude && excluded(dir_name, paths, &new))) { add_excluded(dir); free_dir_entry(dir_ent); continue; } if(exclude_actions()) { subpath = subpathname(dir_ent); if(eval_exclude_actions(dir_name, filename, subpath, &buf, depth)) { add_excluded(dir); free_dir_entry(dir_ent); continue; } } if((buf.st_mode & S_IFMT) == S_IFDIR) { if(subpath == NULL) subpath = subpathname(dir_ent); sub_dir = dir_scan1(filename, subpath, new, scan1_readdir, depth + 1); if(sub_dir == NULL) { free_dir_entry(dir_ent); free(new); continue; } dir->directory_count ++; } else sub_dir = NULL; add_dir_entry(dir_ent, sub_dir, lookup_inode(&buf)); free(new); } scan1_freedir(dir); return dir; } /* * dir_scan2 routines... * This processes most actions and any pseudo files */ struct dir_ent *scan2_readdir(struct dir_info *dir, struct dir_ent *dir_ent) { if (dir_ent == NULL) dir_ent = dir->list; else dir_ent = dir_ent->next; for(; dir_ent && dir_ent->inode->root_entry; dir_ent = dir_ent->next); return dir_ent; } struct dir_ent *scan2_lookup(struct dir_info *dir, char *name) { struct dir_ent *dir_ent = dir->list; for(; dir_ent && strcmp(dir_ent->name, name) != 0; dir_ent = dir_ent->next); return dir_ent; } void dir_scan2(struct dir_info *dir, struct pseudo *pseudo) { struct dir_ent *dir_ent = NULL; struct pseudo_entry *pseudo_ent; struct stat buf; static int pseudo_ino = 1; while((dir_ent = scan2_readdir(dir, dir_ent)) != NULL) { struct inode_info *inode_info = dir_ent->inode; struct stat *buf = &inode_info->buf; char *name = dir_ent->name; eval_actions(dir_ent); if((buf->st_mode & S_IFMT) == S_IFDIR) dir_scan2(dir_ent->dir, pseudo_subdir(name, pseudo)); } while((pseudo_ent = pseudo_readdir(pseudo)) != NULL) { dir_ent = scan2_lookup(dir, pseudo_ent->name); if(pseudo_ent->dev->type == 'm') { struct stat *buf; if(dir_ent == NULL) { ERROR_START("Pseudo modify file \"%s\" does " "not exist in source filesystem.", pseudo_ent->pathname); ERROR_EXIT(" Ignoring.\n"); continue; } if(dir_ent->inode->root_entry) { ERROR_START("Pseudo modify file \"%s\" is a " "pre-existing file in the filesystem " "being appended to. It cannot be "\ "modified.", pseudo_ent->pathname); ERROR_EXIT(" Ignoring.\n"); continue; } buf = &dir_ent->inode->buf; buf->st_mode = (buf->st_mode & S_IFMT) | pseudo_ent->dev->mode; buf->st_uid = pseudo_ent->dev->uid; buf->st_gid = pseudo_ent->dev->gid; continue; } if(dir_ent) { if(dir_ent->inode->root_entry) { ERROR_START("Pseudo file \"%s\" is a " "pre-existing file in the filesystem " "being appended to.", pseudo_ent->pathname); ERROR_EXIT(" Ignoring.\n"); } else { ERROR_START("Pseudo file \"%s\" exists in " "source filesystem \"%s\".", pseudo_ent->pathname, pathname(dir_ent)); ERROR_EXIT("\nIgnoring, exclude it (-e/-ef) to " "override.\n"); } continue; } memset(&buf, 0, sizeof(buf)); buf.st_mode = pseudo_ent->dev->mode; buf.st_uid = pseudo_ent->dev->uid; buf.st_gid = pseudo_ent->dev->gid; buf.st_rdev = makedev(pseudo_ent->dev->major, pseudo_ent->dev->minor); buf.st_mtime = time(NULL); buf.st_ino = pseudo_ino ++; if(pseudo_ent->dev->type == 'd') { struct dir_ent *dir_ent = create_dir_entry(pseudo_ent->name, NULL, pseudo_ent->pathname, dir); char *subpath = strdup(subpathname(dir_ent)); struct dir_info *sub_dir = scan1_opendir("", subpath, dir->depth + 1); if(sub_dir == NULL) { ERROR_START("Could not create pseudo directory " "\"%s\"", pseudo_ent->pathname); ERROR_EXIT(", skipping...\n"); free(subpath); pseudo_ino --; continue; } dir_scan2(sub_dir, pseudo_ent->pseudo); dir->directory_count ++; add_dir_entry(dir_ent, sub_dir, lookup_inode2(&buf, PSEUDO_FILE_OTHER, 0)); } else if(pseudo_ent->dev->type == 'f') { add_dir_entry2(pseudo_ent->name, NULL, pseudo_ent->pathname, NULL, lookup_inode2(&buf, PSEUDO_FILE_PROCESS, pseudo_ent->dev->pseudo_id), dir); } else { add_dir_entry2(pseudo_ent->name, NULL, pseudo_ent->pathname, NULL, lookup_inode2(&buf, PSEUDO_FILE_OTHER, 0), dir); } } } /* * dir_scan3 routines... * This processes the move action */ void dir_scan3(struct dir_info *root, struct dir_info *dir) { struct dir_ent *dir_ent = NULL; while((dir_ent = scan2_readdir(dir, dir_ent)) != NULL) { eval_move_actions(root, dir_ent); if((dir_ent->inode->buf.st_mode & S_IFMT) == S_IFDIR) dir_scan3(root, dir_ent->dir); } } /* * dir_scan4 routines... * This processes the empty action. This action has to be processed after * all other actions because the previous exclude and move actions and the * pseudo actions affect whether a directory is empty */ void dir_scan4(struct dir_info *dir) { struct dir_ent *dir_ent = dir->list, *prev = NULL; while(dir_ent) { if((dir_ent->inode->buf.st_mode & S_IFMT) == S_IFDIR) { dir_scan4(dir_ent->dir); if(eval_empty_actions(dir_ent)) { struct dir_ent *tmp = dir_ent; /* * delete sub-directory, this is by definition * empty */ free(dir_ent->dir->pathname); free(dir_ent->dir->subpath); free(dir_ent->dir); /* remove dir_ent from list */ dir_ent = dir_ent->next; if(prev) prev->next = dir_ent; else dir->list = dir_ent; /* free it */ free_dir_entry(tmp); /* update counts */ dir->directory_count --; dir->count --; add_excluded(dir); continue; } } prev = dir_ent; dir_ent = dir_ent->next; } } /* * dir_scan5 routines... * This sorts every directory and computes the inode numbers */ /* * Bottom up linked list merge sort. * * Qsort and other O(n log n) algorithms work well with arrays but not * linked lists. Merge sort another O(n log n) sort algorithm on the other hand * is not ideal for arrays (as it needs an additonal n storage locations * as sorting is not done in place), but it is ideal for linked lists because * it doesn't require any extra storage, */ void sort_directory(struct dir_info *dir) { struct dir_ent *cur, *l1, *l2, *next; int len1, len2, stride = 1; if(dir->list == NULL || dir->count < 2) return; /* * We can consider our linked-list to be made up of stride length * sublists. Eacn iteration around this loop merges adjacent * stride length sublists into larger 2*stride sublists. We stop * when stride becomes equal to the entire list. * * Initially stride = 1 (by definition a sublist of 1 is sorted), and * these 1 element sublists are merged into 2 element sublists, which * are then merged into 4 element sublists and so on. */ do { l2 = dir->list; /* head of current linked list */ cur = NULL; /* empty output list */ /* * Iterate through the linked list, merging adjacent sublists. * On each interation l2 points to the next sublist pair to be * merged (if there's only one sublist left this is simply added * to the output list) */ while(l2) { l1 = l2; for(len1 = 0; l2 && len1 < stride; len1 ++, l2 = l2->next); len2 = stride; /* * l1 points to first sublist. * l2 points to second sublist. * Merge them onto the output list */ while(len1 && l2 && len2) { if(strcmp(l1->name, l2->name) <= 0) { next = l1; l1 = l1->next; len1 --; } else { next = l2; l2 = l2->next; len2 --; } if(cur) { cur->next = next; cur = next; } else dir->list = cur = next; } /* * One sublist is now empty, copy the other one onto the * output list */ for(; len1; len1 --, l1 = l1->next) { if(cur) { cur->next = l1; cur = l1; } else dir->list = cur = l1; } for(; l2 && len2; len2 --, l2 = l2->next) { if(cur) { cur->next = l2; cur = l2; } else dir->list = cur = l2; } } cur->next = NULL; stride = stride << 1; } while(stride < dir->count); } void dir_scan5(struct dir_info *dir) { struct dir_ent *dir_ent; unsigned int byte_count = 0; sort_directory(dir); for(dir_ent = dir->list; dir_ent; dir_ent = dir_ent->next) { byte_count += strlen(dir_ent->name) + sizeof(struct squashfs_dir_entry); if(dir_ent->inode->root_entry) continue; alloc_inode_no(dir_ent->inode, 0); if((dir_ent->inode->buf.st_mode & S_IFMT) == S_IFDIR) dir_scan5(dir_ent->dir); } if((dir->count < 257 && byte_count < SQUASHFS_METADATA_SIZE)) dir->dir_is_ldir = FALSE; } /* * dir_scan6 routines... * This generates the filesystem metadata and writes it out to the destination */ void scan6_init_dir(struct directory *dir) { dir->buff = malloc(SQUASHFS_METADATA_SIZE); if(dir->buff == NULL) MEM_ERROR(); dir->size = SQUASHFS_METADATA_SIZE; dir->p = dir->index_count_p = dir->buff; dir->entry_count = 256; dir->entry_count_p = NULL; dir->index = NULL; dir->i_count = dir->i_size = 0; } struct dir_ent *scan6_readdir(struct directory *dir, struct dir_info *dir_info, struct dir_ent *dir_ent) { if (dir_ent == NULL) dir_ent = dir_info->list; else dir_ent = dir_ent->next; for(; dir_ent && dir_ent->inode->root_entry; dir_ent = dir_ent->next) add_dir(dir_ent->inode->inode, dir_ent->inode->inode_number, dir_ent->name, dir_ent->inode->type, dir); return dir_ent; } void scan6_freedir(struct directory *dir) { if(dir->index) free(dir->index); free(dir->buff); } void dir_scan6(squashfs_inode *inode, struct dir_info *dir_info) { int squashfs_type; int duplicate_file; struct directory dir; struct dir_ent *dir_ent = NULL; scan6_init_dir(&dir); while((dir_ent = scan6_readdir(&dir, dir_info, dir_ent)) != NULL) { struct stat *buf = &dir_ent->inode->buf; update_info(dir_ent); if(dir_ent->inode->inode == SQUASHFS_INVALID_BLK) { switch(buf->st_mode & S_IFMT) { case S_IFREG: squashfs_type = SQUASHFS_FILE_TYPE; write_file(inode, dir_ent, &duplicate_file); INFO("file %s, uncompressed size %lld " "bytes %s\n", subpathname(dir_ent), (long long) buf->st_size, duplicate_file ? "DUPLICATE" : ""); break; case S_IFDIR: squashfs_type = SQUASHFS_DIR_TYPE; dir_scan6(inode, dir_ent->dir); break; case S_IFLNK: squashfs_type = SQUASHFS_SYMLINK_TYPE; create_inode(inode, NULL, dir_ent, squashfs_type, 0, 0, 0, NULL, NULL, NULL, 0); INFO("symbolic link %s inode 0x%llx\n", subpathname(dir_ent), *inode); sym_count ++; break; case S_IFCHR: squashfs_type = SQUASHFS_CHRDEV_TYPE; create_inode(inode, NULL, dir_ent, squashfs_type, 0, 0, 0, NULL, NULL, NULL, 0); INFO("character device %s inode 0x%llx" "\n", subpathname(dir_ent), *inode); dev_count ++; break; case S_IFBLK: squashfs_type = SQUASHFS_BLKDEV_TYPE; create_inode(inode, NULL, dir_ent, squashfs_type, 0, 0, 0, NULL, NULL, NULL, 0); INFO("block device %s inode 0x%llx\n", subpathname(dir_ent), *inode); dev_count ++; break; case S_IFIFO: squashfs_type = SQUASHFS_FIFO_TYPE; create_inode(inode, NULL, dir_ent, squashfs_type, 0, 0, 0, NULL, NULL, NULL, 0); INFO("fifo %s inode 0x%llx\n", subpathname(dir_ent), *inode); fifo_count ++; break; case S_IFSOCK: squashfs_type = SQUASHFS_SOCKET_TYPE; create_inode(inode, NULL, dir_ent, squashfs_type, 0, 0, 0, NULL, NULL, NULL, 0); INFO("unix domain socket %s inode " "0x%llx\n", subpathname(dir_ent), *inode); sock_count ++; break; default: BAD_ERROR("%s unrecognised file type, " "mode is %x\n", subpathname(dir_ent), buf->st_mode); } dir_ent->inode->inode = *inode; dir_ent->inode->type = squashfs_type; } else { *inode = dir_ent->inode->inode; squashfs_type = dir_ent->inode->type; switch(squashfs_type) { case SQUASHFS_FILE_TYPE: if(!sorted) INFO("file %s, uncompressed " "size %lld bytes LINK" "\n", subpathname(dir_ent), (long long) buf->st_size); break; case SQUASHFS_SYMLINK_TYPE: INFO("symbolic link %s inode 0x%llx " "LINK\n", subpathname(dir_ent), *inode); break; case SQUASHFS_CHRDEV_TYPE: INFO("character device %s inode 0x%llx " "LINK\n", subpathname(dir_ent), *inode); break; case SQUASHFS_BLKDEV_TYPE: INFO("block device %s inode 0x%llx " "LINK\n", subpathname(dir_ent), *inode); break; case SQUASHFS_FIFO_TYPE: INFO("fifo %s inode 0x%llx LINK\n", subpathname(dir_ent), *inode); break; case SQUASHFS_SOCKET_TYPE: INFO("unix domain socket %s inode " "0x%llx LINK\n", subpathname(dir_ent), *inode); break; } } add_dir(*inode, get_inode_no(dir_ent->inode), dir_ent->name, squashfs_type, &dir); } write_dir(inode, dir_info, &dir); INFO("directory %s inode 0x%llx\n", subpathname(dir_info->dir_ent), *inode); scan6_freedir(&dir); } unsigned int slog(unsigned int block) { int i; for(i = 12; i <= 20; i++) if(block == (1 << i)) return i; return 0; } int old_excluded(char *filename, struct stat *buf) { int i; for(i = 0; i < exclude; i++) if((exclude_paths[i].st_dev == buf->st_dev) && (exclude_paths[i].st_ino == buf->st_ino)) return TRUE; return FALSE; } #define ADD_ENTRY(buf) \ if(exclude % EXCLUDE_SIZE == 0) { \ exclude_paths = realloc(exclude_paths, (exclude + EXCLUDE_SIZE) \ * sizeof(struct exclude_info)); \ if(exclude_paths == NULL) \ MEM_ERROR(); \ } \ exclude_paths[exclude].st_dev = buf.st_dev; \ exclude_paths[exclude++].st_ino = buf.st_ino; int old_add_exclude(char *path) { int i; char *filename; struct stat buf; if(path[0] == '/' || strncmp(path, "./", 2) == 0 || strncmp(path, "../", 3) == 0) { if(lstat(path, &buf) == -1) { ERROR_START("Cannot stat exclude dir/file %s because " "%s", path, strerror(errno)); ERROR_EXIT(", ignoring\n"); return TRUE; } ADD_ENTRY(buf); return TRUE; } for(i = 0; i < source; i++) { int res = asprintf(&filename, "%s/%s", source_path[i], path); if(res == -1) BAD_ERROR("asprintf failed in old_add_exclude\n"); if(lstat(filename, &buf) == -1) { if(!(errno == ENOENT || errno == ENOTDIR)) { ERROR_START("Cannot stat exclude dir/file %s " "because %s", filename, strerror(errno)); ERROR_EXIT(", ignoring\n"); } free(filename); continue; } free(filename); ADD_ENTRY(buf); } return TRUE; } void add_old_root_entry(char *name, squashfs_inode inode, int inode_number, int type) { old_root_entry = realloc(old_root_entry, sizeof(struct old_root_entry_info) * (old_root_entries + 1)); if(old_root_entry == NULL) MEM_ERROR(); old_root_entry[old_root_entries].name = strdup(name); old_root_entry[old_root_entries].inode.inode = inode; old_root_entry[old_root_entries].inode.inode_number = inode_number; old_root_entry[old_root_entries].inode.type = type; old_root_entry[old_root_entries++].inode.root_entry = TRUE; } void initialise_threads(int readq, int fragq, int bwriteq, int fwriteq, int freelst, char *destination_file) { int i; sigset_t sigmask, old_mask; int total_mem = readq; int reader_size; int fragment_size; int fwriter_size; /* * bwriter_size is global because it is needed in * write_file_blocks_dup() */ /* * Never allow the total size of the queues to be larger than * physical memory * * When adding together the possibly user supplied values, make * sure they've not been deliberately contrived to overflow an int */ if(add_overflow(total_mem, fragq)) BAD_ERROR("Queue sizes rediculously too large\n"); total_mem += fragq; if(add_overflow(total_mem, bwriteq)) BAD_ERROR("Queue sizes rediculously too large\n"); total_mem += bwriteq; if(add_overflow(total_mem, fwriteq)) BAD_ERROR("Queue sizes rediculously too large\n"); total_mem += fwriteq; if(total_mem > get_physical_memory()) { ERROR("Total queue sizes larger than physical memory.\n"); ERROR("Mksquashfs will exhaust physical memory and thrash.\n"); BAD_ERROR("Queues too large\n"); } /* * convert from queue size in Mbytes to queue size in * blocks. * * This isn't going to overflow an int unless there exists * systems with more than 8 Petabytes of RAM! */ reader_size = readq << (20 - block_log); fragment_size = fragq << (20 - block_log); bwriter_size = bwriteq << (20 - block_log); fwriter_size = fwriteq << (20 - block_log); /* * setup signal handlers for the main thread, these cleanup * deleting the destination file, if appending the * handlers for SIGTERM and SIGINT will be replaced with handlers * allowing the user to press ^C twice to restore the existing * filesystem. * * SIGUSR1 is an internal signal, which is used by the sub-threads * to tell the main thread to terminate, deleting the destination file, * or if necessary restoring the filesystem on appending */ signal(SIGTERM, sighandler); signal(SIGINT, sighandler); signal(SIGUSR1, sighandler); /* block SIGQUIT and SIGHUP, these are handled by the info thread */ sigemptyset(&sigmask); sigaddset(&sigmask, SIGQUIT); sigaddset(&sigmask, SIGHUP); if(pthread_sigmask(SIG_BLOCK, &sigmask, NULL) == -1) BAD_ERROR("Failed to set signal mask in intialise_threads\n"); /* * temporarily block these signals, so the created sub-threads * will ignore them, ensuring the main thread handles them */ sigemptyset(&sigmask); sigaddset(&sigmask, SIGINT); sigaddset(&sigmask, SIGTERM); sigaddset(&sigmask, SIGUSR1); if(pthread_sigmask(SIG_BLOCK, &sigmask, &old_mask) == -1) BAD_ERROR("Failed to set signal mask in intialise_threads\n"); if(processors == -1) { #ifndef linux int mib[2]; size_t len = sizeof(processors); mib[0] = CTL_HW; #ifdef HW_AVAILCPU mib[1] = HW_AVAILCPU; #else mib[1] = HW_NCPU; #endif if(sysctl(mib, 2, &processors, &len, NULL, 0) == -1) { ERROR_START("Failed to get number of available " "processors."); ERROR_EXIT(" Defaulting to 1\n"); processors = 1; } #else processors = sysconf(_SC_NPROCESSORS_ONLN); #endif } if(multiply_overflow(processors, 3) || multiply_overflow(processors * 3, sizeof(pthread_t))) BAD_ERROR("Processors too large\n"); deflator_thread = malloc(processors * 3 * sizeof(pthread_t)); if(deflator_thread == NULL) MEM_ERROR(); frag_deflator_thread = &deflator_thread[processors]; frag_thread = &frag_deflator_thread[processors]; to_reader = queue_init(1); to_deflate = queue_init(reader_size); to_process_frag = queue_init(reader_size); to_writer = queue_init(bwriter_size + fwriter_size); from_writer = queue_init(1); to_frag = queue_init(fragment_size); locked_fragment = queue_init(fragment_size); to_main = seq_queue_init(); reader_buffer = cache_init(block_size, reader_size, 0, 0); bwriter_buffer = cache_init(block_size, bwriter_size, 1, freelst); fwriter_buffer = cache_init(block_size, fwriter_size, 1, freelst); fragment_buffer = cache_init(block_size, fragment_size, 1, 0); reserve_cache = cache_init(block_size, processors + 1, 1, 0); pthread_create(&reader_thread, NULL, reader, NULL); pthread_create(&writer_thread, NULL, writer, NULL); init_progress_bar(); init_info(); for(i = 0; i < processors; i++) { if(pthread_create(&deflator_thread[i], NULL, deflator, NULL)) BAD_ERROR("Failed to create thread\n"); if(pthread_create(&frag_deflator_thread[i], NULL, frag_deflator, NULL) != 0) BAD_ERROR("Failed to create thread\n"); if(pthread_create(&frag_thread[i], NULL, frag_thrd, (void *) destination_file) != 0) BAD_ERROR("Failed to create thread\n"); } main_thread = pthread_self(); printf("Parallel mksquashfs: Using %d processor%s\n", processors, processors == 1 ? "" : "s"); /* Restore the signal mask for the main thread */ if(pthread_sigmask(SIG_SETMASK, &old_mask, NULL) == -1) BAD_ERROR("Failed to set signal mask in intialise_threads\n"); } long long write_inode_lookup_table() { int i, inode_number, lookup_bytes = SQUASHFS_LOOKUP_BYTES(inode_count); void *it; if(inode_count == sinode_count) goto skip_inode_hash_table; it = realloc(inode_lookup_table, lookup_bytes); if(it == NULL) MEM_ERROR(); inode_lookup_table = it; for(i = 0; i < INODE_HASH_SIZE; i ++) { struct inode_info *inode; for(inode = inode_info[i]; inode; inode = inode->next) { inode_number = get_inode_no(inode); SQUASHFS_SWAP_LONG_LONGS(&inode->inode, &inode_lookup_table[inode_number - 1], 1); } } skip_inode_hash_table: return generic_write_table(lookup_bytes, inode_lookup_table, 0, NULL, noI); } char *get_component(char *target, char **targname) { char *start; while(*target == '/') target ++; start = target; while(*target != '/' && *target != '\0') target ++; *targname = strndup(start, target - start); while(*target == '/') target ++; return target; } void free_path(struct pathname *paths) { int i; for(i = 0; i < paths->names; i++) { if(paths->name[i].paths) free_path(paths->name[i].paths); free(paths->name[i].name); if(paths->name[i].preg) { regfree(paths->name[i].preg); free(paths->name[i].preg); } } free(paths); } struct pathname *add_path(struct pathname *paths, char *target, char *alltarget) { char *targname; int i, error; target = get_component(target, &targname); if(paths == NULL) { paths = malloc(sizeof(struct pathname)); if(paths == NULL) MEM_ERROR(); paths->names = 0; paths->name = NULL; } for(i = 0; i < paths->names; i++) if(strcmp(paths->name[i].name, targname) == 0) break; if(i == paths->names) { /* allocate new name entry */ paths->names ++; paths->name = realloc(paths->name, (i + 1) * sizeof(struct path_entry)); if(paths->name == NULL) MEM_ERROR(); paths->name[i].name = targname; paths->name[i].paths = NULL; if(use_regex) { paths->name[i].preg = malloc(sizeof(regex_t)); if(paths->name[i].preg == NULL) MEM_ERROR(); error = regcomp(paths->name[i].preg, targname, REG_EXTENDED|REG_NOSUB); if(error) { char str[1024]; /* overflow safe */ regerror(error, paths->name[i].preg, str, 1024); BAD_ERROR("invalid regex %s in export %s, " "because %s\n", targname, alltarget, str); } } else paths->name[i].preg = NULL; if(target[0] == '\0') /* at leaf pathname component */ paths->name[i].paths = NULL; else /* recurse adding child components */ paths->name[i].paths = add_path(NULL, target, alltarget); } else { /* existing matching entry */ free(targname); if(paths->name[i].paths == NULL) { /* No sub-directory which means this is the leaf * component of a pre-existing exclude which subsumes * the exclude currently being added, in which case stop * adding components */ } else if(target[0] == '\0') { /* at leaf pathname component and child components exist * from more specific excludes, delete as they're * subsumed by this exclude */ free_path(paths->name[i].paths); paths->name[i].paths = NULL; } else /* recurse adding child components */ add_path(paths->name[i].paths, target, alltarget); } return paths; } void add_exclude(char *target) { if(target[0] == '/' || strncmp(target, "./", 2) == 0 || strncmp(target, "../", 3) == 0) BAD_ERROR("/, ./ and ../ prefixed excludes not supported with " "-wildcards or -regex options\n"); else if(strncmp(target, "... ", 4) == 0) stickypath = add_path(stickypath, target + 4, target + 4); else path = add_path(path, target, target); } void display_path(int depth, struct pathname *paths) { int i, n; if(paths == NULL) return; for(i = 0; i < paths->names; i++) { for(n = 0; n < depth; n++) printf("\t"); printf("%d: %s\n", depth, paths->name[i].name); display_path(depth + 1, paths->name[i].paths); } } void display_path2(struct pathname *paths, char *string) { int i; char *path; if(paths == NULL) { printf("%s\n", string); return; } for(i = 0; i < paths->names; i++) { int res = asprintf(&path, "%s/%s", string, paths->name[i].name); if(res == -1) BAD_ERROR("asprintf failed in display_path2\n"); display_path2(paths->name[i].paths, path); free(path); } } struct pathnames *add_subdir(struct pathnames *paths, struct pathname *path) { int count = paths == NULL ? 0 : paths->count; if(count % PATHS_ALLOC_SIZE == 0) { paths = realloc(paths, sizeof(struct pathnames) + (count + PATHS_ALLOC_SIZE) * sizeof(struct pathname *)); if(paths == NULL) MEM_ERROR(); } paths->path[count] = path; paths->count = count + 1; return paths; } int excluded_match(char *name, struct pathname *path, struct pathnames **new) { int i; for(i = 0; i < path->names; i++) { int match = use_regex ? regexec(path->name[i].preg, name, (size_t) 0, NULL, 0) == 0 : fnmatch(path->name[i].name, name, FNM_PATHNAME|FNM_PERIOD|FNM_EXTMATCH) == 0; if(match) { if(path->name[i].paths == NULL || new == NULL) /* match on a leaf component, any subdirectories * in the filesystem should be excluded */ return TRUE; else /* match on a non-leaf component, add any * subdirectories to the new set of * subdirectories to scan for this name */ *new = add_subdir(*new, path->name[i].paths); } } return FALSE; } int excluded(char *name, struct pathnames *paths, struct pathnames **new) { int n; if(stickypath && excluded_match(name, stickypath, NULL)) return TRUE; for(n = 0; paths && n < paths->count; n++) { int res = excluded_match(name, paths->path[n], new); if(res) { free(*new); *new = NULL; return TRUE; } } /* * Either: * - no matching names found, return empty new search set, or * - one or more matches with sub-directories found (no leaf matches), * in which case return new search set. * * In either case return FALSE as we don't want to exclude this entry */ return FALSE; } void process_exclude_file(char *argv) { FILE *fd; char buffer[MAX_LINE + 1]; /* overflow safe */ char *filename; fd = fopen(argv, "r"); if(fd == NULL) BAD_ERROR("Failed to open exclude file \"%s\" because %s\n", argv, strerror(errno)); while(fgets(filename = buffer, MAX_LINE + 1, fd) != NULL) { int len = strlen(filename); if(len == MAX_LINE && filename[len - 1] != '\n') /* line too large */ BAD_ERROR("Line too long when reading " "exclude file \"%s\", larger than %d " "bytes\n", argv, MAX_LINE); /* * Remove '\n' terminator if it exists (the last line * in the file may not be '\n' terminated) */ if(len && filename[len - 1] == '\n') filename[len - 1] = '\0'; /* Skip any leading whitespace */ while(isspace(*filename)) filename ++; /* if comment line, skip */ if(*filename == '#') continue; /* * check for initial backslash, to accommodate * filenames with leading space or leading # character */ if(*filename == '\\') filename ++; /* if line is now empty after skipping characters, skip it */ if(*filename == '\0') continue; if(old_exclude) old_add_exclude(filename); else add_exclude(filename); } if(ferror(fd)) BAD_ERROR("Reading exclude file \"%s\" failed because %s\n", argv, strerror(errno)); fclose(fd); } #define RECOVER_ID "Squashfs recovery file v1.0\n" #define RECOVER_ID_SIZE 28 void write_recovery_data(struct squashfs_super_block *sBlk) { int res, recoverfd, bytes = sBlk->bytes_used - sBlk->inode_table_start; pid_t pid = getpid(); char *metadata; char header[] = RECOVER_ID; if(recover == FALSE) { printf("No recovery data option specified.\n"); printf("Skipping saving recovery file.\n\n"); return; } metadata = malloc(bytes); if(metadata == NULL) MEM_ERROR(); res = read_fs_bytes(fd, sBlk->inode_table_start, bytes, metadata); if(res == 0) { ERROR("Failed to read append filesystem metadata\n"); BAD_ERROR("Filesystem corrupted?\n"); } res = asprintf(&recovery_file, "squashfs_recovery_%s_%d", getbase(destination_file), pid); if(res == -1) MEM_ERROR(); recoverfd = open(recovery_file, O_CREAT | O_TRUNC | O_RDWR, S_IRWXU); if(recoverfd == -1) BAD_ERROR("Failed to create recovery file, because %s. " "Aborting\n", strerror(errno)); if(write_bytes(recoverfd, header, RECOVER_ID_SIZE) == -1) BAD_ERROR("Failed to write recovery file, because %s\n", strerror(errno)); if(write_bytes(recoverfd, sBlk, sizeof(struct squashfs_super_block)) == -1) BAD_ERROR("Failed to write recovery file, because %s\n", strerror(errno)); if(write_bytes(recoverfd, metadata, bytes) == -1) BAD_ERROR("Failed to write recovery file, because %s\n", strerror(errno)); close(recoverfd); free(metadata); printf("Recovery file \"%s\" written\n", recovery_file); printf("If Mksquashfs aborts abnormally (i.e. power failure), run\n"); printf("mksquashfs dummy %s -recover %s\n", destination_file, recovery_file); printf("to restore filesystem\n\n"); } void read_recovery_data(char *recovery_file, char *destination_file) { int fd, recoverfd, bytes; struct squashfs_super_block orig_sBlk, sBlk; char *metadata; int res; struct stat buf; char header[] = RECOVER_ID; char header2[RECOVER_ID_SIZE]; recoverfd = open(recovery_file, O_RDONLY); if(recoverfd == -1) BAD_ERROR("Failed to open recovery file because %s\n", strerror(errno)); if(stat(destination_file, &buf) == -1) BAD_ERROR("Failed to stat destination file, because %s\n", strerror(errno)); fd = open(destination_file, O_RDWR); if(fd == -1) BAD_ERROR("Failed to open destination file because %s\n", strerror(errno)); res = read_bytes(recoverfd, header2, RECOVER_ID_SIZE); if(res == -1) BAD_ERROR("Failed to read recovery file, because %s\n", strerror(errno)); if(res < RECOVER_ID_SIZE) BAD_ERROR("Recovery file appears to be truncated\n"); if(strncmp(header, header2, RECOVER_ID_SIZE) !=0 ) BAD_ERROR("Not a recovery file\n"); res = read_bytes(recoverfd, &sBlk, sizeof(struct squashfs_super_block)); if(res == -1) BAD_ERROR("Failed to read recovery file, because %s\n", strerror(errno)); if(res < sizeof(struct squashfs_super_block)) BAD_ERROR("Recovery file appears to be truncated\n"); res = read_fs_bytes(fd, 0, sizeof(struct squashfs_super_block), &orig_sBlk); if(res == 0) { ERROR("Failed to read superblock from output filesystem\n"); BAD_ERROR("Output filesystem is empty!\n"); } if(memcmp(((char *) &sBlk) + 4, ((char *) &orig_sBlk) + 4, sizeof(struct squashfs_super_block) - 4) != 0) BAD_ERROR("Recovery file and destination file do not seem to " "match\n"); bytes = sBlk.bytes_used - sBlk.inode_table_start; metadata = malloc(bytes); if(metadata == NULL) MEM_ERROR(); res = read_bytes(recoverfd, metadata, bytes); if(res == -1) BAD_ERROR("Failed to read recovery file, because %s\n", strerror(errno)); if(res < bytes) BAD_ERROR("Recovery file appears to be truncated\n"); write_destination(fd, 0, sizeof(struct squashfs_super_block), &sBlk); write_destination(fd, sBlk.inode_table_start, bytes, metadata); close(recoverfd); close(fd); printf("Successfully wrote recovery file \"%s\". Exiting\n", recovery_file); exit(0); } void write_filesystem_tables(struct squashfs_super_block *sBlk, int nopad) { int i; sBlk->fragments = fragments; sBlk->no_ids = id_count; sBlk->inode_table_start = write_inodes(); sBlk->directory_table_start = write_directories(); sBlk->fragment_table_start = write_fragment_table(); sBlk->lookup_table_start = exportable ? write_inode_lookup_table() : SQUASHFS_INVALID_BLK; sBlk->id_table_start = write_id_table(); sBlk->xattr_id_table_start = write_xattrs(); TRACE("sBlk->inode_table_start 0x%llx\n", sBlk->inode_table_start); TRACE("sBlk->directory_table_start 0x%llx\n", sBlk->directory_table_start); TRACE("sBlk->fragment_table_start 0x%llx\n", sBlk->fragment_table_start); if(exportable) TRACE("sBlk->lookup_table_start 0x%llx\n", sBlk->lookup_table_start); sBlk->bytes_used = bytes; sBlk->compression = comp->id; SQUASHFS_INSWAP_SUPER_BLOCK(sBlk); write_destination(fd, SQUASHFS_START, sizeof(*sBlk), sBlk); if(!nopad && (i = bytes & (4096 - 1))) { char temp[4096] = {0}; write_destination(fd, bytes, 4096 - i, temp); } close(fd); if(recovery_file) unlink(recovery_file); total_bytes += total_inode_bytes + total_directory_bytes + sizeof(struct squashfs_super_block) + total_xattr_bytes; printf("\n%sSquashfs %d.%d filesystem, %s compressed, data block size" " %d\n", exportable ? "Exportable " : "", SQUASHFS_MAJOR, SQUASHFS_MINOR, comp->name, block_size); printf("\t%s data, %s metadata, %s fragments, %s xattrs\n", noD ? "uncompressed" : "compressed", noI ? "uncompressed" : "compressed", no_fragments ? "no" : noF ? "uncompressed" : "compressed", no_xattrs ? "no" : noX ? "uncompressed" : "compressed"); printf("\tduplicates are %sremoved\n", duplicate_checking ? "" : "not "); printf("Filesystem size %.2f Kbytes (%.2f Mbytes)\n", bytes / 1024.0, bytes / (1024.0 * 1024.0)); printf("\t%.2f%% of uncompressed filesystem size (%.2f Kbytes)\n", ((float) bytes / total_bytes) * 100.0, total_bytes / 1024.0); printf("Inode table size %d bytes (%.2f Kbytes)\n", inode_bytes, inode_bytes / 1024.0); printf("\t%.2f%% of uncompressed inode table size (%d bytes)\n", ((float) inode_bytes / total_inode_bytes) * 100.0, total_inode_bytes); printf("Directory table size %d bytes (%.2f Kbytes)\n", directory_bytes, directory_bytes / 1024.0); printf("\t%.2f%% of uncompressed directory table size (%d bytes)\n", ((float) directory_bytes / total_directory_bytes) * 100.0, total_directory_bytes); if(total_xattr_bytes) { printf("Xattr table size %d bytes (%.2f Kbytes)\n", xattr_bytes, xattr_bytes / 1024.0); printf("\t%.2f%% of uncompressed xattr table size (%d bytes)\n", ((float) xattr_bytes / total_xattr_bytes) * 100.0, total_xattr_bytes); } if(duplicate_checking) printf("Number of duplicate files found %d\n", file_count - dup_files); else printf("No duplicate files removed\n"); printf("Number of inodes %d\n", inode_count); printf("Number of files %d\n", file_count); if(!no_fragments) printf("Number of fragments %d\n", fragments); printf("Number of symbolic links %d\n", sym_count); printf("Number of device nodes %d\n", dev_count); printf("Number of fifo nodes %d\n", fifo_count); printf("Number of socket nodes %d\n", sock_count); printf("Number of directories %d\n", dir_count); printf("Number of ids (unique uids + gids) %d\n", id_count); printf("Number of uids %d\n", uid_count); for(i = 0; i < id_count; i++) { if(id_table[i]->flags & ISA_UID) { struct passwd *user = getpwuid(id_table[i]->id); printf("\t%s (%d)\n", user == NULL ? "unknown" : user->pw_name, id_table[i]->id); } } printf("Number of gids %d\n", guid_count); for(i = 0; i < id_count; i++) { if(id_table[i]->flags & ISA_GID) { struct group *group = getgrgid(id_table[i]->id); printf("\t%s (%d)\n", group == NULL ? "unknown" : group->gr_name, id_table[i]->id); } } } int parse_numberll(char *start, long long *res, int size) { char *end; long long number; errno = 0; /* To distinguish success/failure after call */ number = strtoll(start, &end, 10); /* * check for strtoll underflow or overflow in conversion, and other * errors. */ if((errno == ERANGE && (number == LLONG_MIN || number == LLONG_MAX)) || (errno != 0 && number == 0)) return 0; /* reject negative numbers as invalid */ if(number < 0) return 0; if(size) { /* * Check for multiplier and trailing junk. * But first check that a number exists before the * multiplier */ if(end == start) return 0; switch(end[0]) { case 'g': case 'G': if(multiply_overflowll(number, 1073741824)) return 0; number *= 1073741824; if(end[1] != '\0') /* trailing junk after multiplier, but * allow it to be "bytes" */ if(strcmp(end + 1, "bytes")) return 0; break; case 'm': case 'M': if(multiply_overflowll(number, 1048576)) return 0; number *= 1048576; if(end[1] != '\0') /* trailing junk after multiplier, but * allow it to be "bytes" */ if(strcmp(end + 1, "bytes")) return 0; break; case 'k': case 'K': if(multiply_overflowll(number, 1024)) return 0; number *= 1024; if(end[1] != '\0') /* trailing junk after multiplier, but * allow it to be "bytes" */ if(strcmp(end + 1, "bytes")) return 0; break; case '\0': break; default: /* trailing junk after number */ return 0; } } else if(end[0] != '\0') /* trailing junk after number */ return 0; *res = number; return 1; } int parse_number(char *start, int *res, int size) { long long number; if(!parse_numberll(start, &number, size)) return 0; /* check if long result will overflow signed int */ if(number > INT_MAX) return 0; *res = (int) number; return 1; } int parse_num(char *arg, int *res) { return parse_number(arg, res, 0); } int get_physical_memory() { /* Long longs are used here because with PAE, a 32-bit machine can have more than 4GB of physical memory */ long long num_pages = sysconf(_SC_PHYS_PAGES); long long page_size = sysconf(_SC_PAGESIZE); int phys_mem = num_pages * page_size >> 20; if(phys_mem < SQUASHFS_LOWMEM) BAD_ERROR("Mksquashfs requires more physical memory than is " "available!\n"); return phys_mem; } void calculate_queue_sizes(int mem, int *readq, int *fragq, int *bwriteq, int *fwriteq) { *readq = mem / SQUASHFS_READQ_MEM; *bwriteq = mem / SQUASHFS_BWRITEQ_MEM; *fwriteq = mem / SQUASHFS_FWRITEQ_MEM; *fragq = mem - *readq - *bwriteq - *fwriteq; } #define VERSION() \ printf("mksquashfs version 4.3 (2014/05/12)\n");\ printf("copyright (C) 2014 Phillip Lougher "\ "\n\n"); \ printf("This program is free software; you can redistribute it and/or"\ "\n");\ printf("modify it under the terms of the GNU General Public License"\ "\n");\ printf("as published by the Free Software Foundation; either version "\ "2,\n");\ printf("or (at your option) any later version.\n\n");\ printf("This program is distributed in the hope that it will be "\ "useful,\n");\ printf("but WITHOUT ANY WARRANTY; without even the implied warranty "\ "of\n");\ printf("MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the"\ "\n");\ printf("GNU General Public License for more details.\n"); int main(int argc, char *argv[]) { struct stat buf, source_buf; int res, i; char *b, *root_name = NULL; int keep_as_directory = FALSE; squashfs_inode inode; int readq; int fragq; int bwriteq; int fwriteq; int total_mem = get_physical_memory() / SQUASHFS_TAKE; int progress = TRUE; int force_progress = FALSE; struct file_buffer **fragment = NULL; if(argc > 1 && strcmp(argv[1], "-version") == 0) { VERSION(); exit(0); } block_log = slog(block_size); calculate_queue_sizes(total_mem, &readq, &fragq, &bwriteq, &fwriteq); for(i = 1; i < argc && argv[i][0] != '-'; i++); if(i < 3) goto printOptions; source_path = argv + 1; source = i - 2; /* * Scan the command line for -comp xxx option, this is to ensure * any -X compressor specific options are passed to the * correct compressor */ for(; i < argc; i++) { struct compressor *prev_comp = comp; if(strcmp(argv[i], "-comp") == 0) { if(++i == argc) { ERROR("%s: -comp missing compression type\n", argv[0]); exit(1); } comp = lookup_compressor(argv[i]); if(!comp->supported) { ERROR("%s: Compressor \"%s\" is not supported!" "\n", argv[0], argv[i]); ERROR("%s: Compressors available:\n", argv[0]); display_compressors("", COMP_DEFAULT); exit(1); } if(prev_comp != NULL && prev_comp != comp) { ERROR("%s: -comp multiple conflicting -comp" " options specified on command line" ", previously %s, now %s\n", argv[0], prev_comp->name, comp->name); exit(1); } compressor_opt_parsed = 1; } else if(strcmp(argv[i], "-e") == 0) break; else if(strcmp(argv[i], "-root-becomes") == 0 || strcmp(argv[i], "-ef") == 0 || strcmp(argv[i], "-pf") == 0 || strcmp(argv[i], "-af") == 0 || strcmp(argv[i], "-comp") == 0) i++; } /* * if no -comp option specified lookup default compressor. Note the * Makefile ensures the default compressor has been built, and so we * don't need to to check for failure here */ if(comp == NULL) comp = lookup_compressor(COMP_DEFAULT); for(i = source + 2; i < argc; i++) { if(strcmp(argv[i], "-action") == 0 || strcmp(argv[i], "-a") ==0) { if(++i == argc) { ERROR("%s: %s missing action\n", argv[0], argv[i - 1]); exit(1); } res = parse_action(argv[i]); if(res == 0) exit(1); } else if(strcmp(argv[i], "-af") == 0) { if(++i == argc) { ERROR("%s: -af missing filename\n", argv[0]); exit(1); } if(read_action_file(argv[i]) == FALSE) exit(1); } else if(strcmp(argv[i], "-comp") == 0) /* parsed previously */ i++; else if(strncmp(argv[i], "-X", 2) == 0) { int args; if(strcmp(argv[i] + 2, "help") == 0) goto print_compressor_options; args = compressor_options(comp, argv + i, argc - i); if(args < 0) { if(args == -1) { ERROR("%s: Unrecognised compressor" " option %s\n", argv[0], argv[i]); if(!compressor_opt_parsed) ERROR("%s: Did you forget to" " specify -comp?\n", argv[0]); print_compressor_options: ERROR("%s: selected compressor \"%s\"" ". Options supported: %s\n", argv[0], comp->name, comp->usage ? "" : "none"); if(comp->usage) comp->usage(); } exit(1); } i += args; } else if(strcmp(argv[i], "-pf") == 0) { if(++i == argc) { ERROR("%s: -pf missing filename\n", argv[0]); exit(1); } if(read_pseudo_file(argv[i]) == FALSE) exit(1); } else if(strcmp(argv[i], "-p") == 0) { if(++i == argc) { ERROR("%s: -p missing pseudo file definition\n", argv[0]); exit(1); } if(read_pseudo_def(argv[i]) == FALSE) exit(1); } else if(strcmp(argv[i], "-recover") == 0) { if(++i == argc) { ERROR("%s: -recover missing recovery file\n", argv[0]); exit(1); } read_recovery_data(argv[i], argv[source + 1]); } else if(strcmp(argv[i], "-no-recovery") == 0) recover = FALSE; else if(strcmp(argv[i], "-wildcards") == 0) { old_exclude = FALSE; use_regex = FALSE; } else if(strcmp(argv[i], "-regex") == 0) { old_exclude = FALSE; use_regex = TRUE; } else if(strcmp(argv[i], "-no-sparse") == 0) sparse_files = FALSE; else if(strcmp(argv[i], "-no-progress") == 0) progress = FALSE; else if(strcmp(argv[i], "-progress") == 0) force_progress = TRUE; else if(strcmp(argv[i], "-no-exports") == 0) exportable = FALSE; else if(strcmp(argv[i], "-processors") == 0) { if((++i == argc) || !parse_num(argv[i], &processors)) { ERROR("%s: -processors missing or invalid " "processor number\n", argv[0]); exit(1); } if(processors < 1) { ERROR("%s: -processors should be 1 or larger\n", argv[0]); exit(1); } } else if(strcmp(argv[i], "-read-queue") == 0) { if((++i == argc) || !parse_num(argv[i], &readq)) { ERROR("%s: -read-queue missing or invalid " "queue size\n", argv[0]); exit(1); } if(readq < 1) { ERROR("%s: -read-queue should be 1 megabyte or " "larger\n", argv[0]); exit(1); } } else if(strcmp(argv[i], "-write-queue") == 0) { if((++i == argc) || !parse_num(argv[i], &bwriteq)) { ERROR("%s: -write-queue missing or invalid " "queue size\n", argv[0]); exit(1); } if(bwriteq < 2) { ERROR("%s: -write-queue should be 2 megabytes " "or larger\n", argv[0]); exit(1); } fwriteq = bwriteq >> 1; bwriteq -= fwriteq; } else if(strcmp(argv[i], "-fragment-queue") == 0) { if((++i == argc) || !parse_num(argv[i], &fragq)) { ERROR("%s: -fragment-queue missing or invalid " "queue size\n", argv[0]); exit(1); } if(fragq < 1) { ERROR("%s: -fragment-queue should be 1 " "megabyte or larger\n", argv[0]); exit(1); } } else if(strcmp(argv[i], "-mem") == 0) { long long number; if((++i == argc) || !parse_numberll(argv[i], &number, 1)) { ERROR("%s: -mem missing or invalid mem size\n", argv[0]); exit(1); } /* convert from bytes to Mbytes */ total_mem = number / 1048576; if(total_mem < (SQUASHFS_LOWMEM / SQUASHFS_TAKE)) { ERROR("%s: -mem should be %d Mbytes or " "larger\n", argv[0], SQUASHFS_LOWMEM / SQUASHFS_TAKE); exit(1); } calculate_queue_sizes(total_mem, &readq, &fragq, &bwriteq, &fwriteq); } else if(strcmp(argv[i], "-b") == 0) { if(++i == argc) { ERROR("%s: -b missing block size\n", argv[0]); exit(1); } if(!parse_number(argv[i], &block_size, 1)) { ERROR("%s: -b invalid block size\n", argv[0]); exit(1); } if((block_log = slog(block_size)) == 0) { ERROR("%s: -b block size not power of two or " "not between 4096 and 1Mbyte\n", argv[0]); exit(1); } } else if(strcmp(argv[i], "-ef") == 0) { if(++i == argc) { ERROR("%s: -ef missing filename\n", argv[0]); exit(1); } } else if(strcmp(argv[i], "-no-duplicates") == 0) duplicate_checking = FALSE; else if(strcmp(argv[i], "-no-fragments") == 0) no_fragments = TRUE; else if(strcmp(argv[i], "-always-use-fragments") == 0) always_use_fragments = TRUE; else if(strcmp(argv[i], "-sort") == 0) { if(++i == argc) { ERROR("%s: -sort missing filename\n", argv[0]); exit(1); } } else if(strcmp(argv[i], "-all-root") == 0 || strcmp(argv[i], "-root-owned") == 0) global_uid = global_gid = 0; else if(strcmp(argv[i], "-force-uid") == 0) { if(++i == argc) { ERROR("%s: -force-uid missing uid or user\n", argv[0]); exit(1); } if((global_uid = strtoll(argv[i], &b, 10)), *b =='\0') { if(global_uid < 0 || global_uid > (((long long) 1 << 32) - 1)) { ERROR("%s: -force-uid uid out of range" "\n", argv[0]); exit(1); } } else { struct passwd *uid = getpwnam(argv[i]); if(uid) global_uid = uid->pw_uid; else { ERROR("%s: -force-uid invalid uid or " "unknown user\n", argv[0]); exit(1); } } } else if(strcmp(argv[i], "-force-gid") == 0) { if(++i == argc) { ERROR("%s: -force-gid missing gid or group\n", argv[0]); exit(1); } if((global_gid = strtoll(argv[i], &b, 10)), *b =='\0') { if(global_gid < 0 || global_gid > (((long long) 1 << 32) - 1)) { ERROR("%s: -force-gid gid out of range" "\n", argv[0]); exit(1); } } else { struct group *gid = getgrnam(argv[i]); if(gid) global_gid = gid->gr_gid; else { ERROR("%s: -force-gid invalid gid or " "unknown group\n", argv[0]); exit(1); } } } else if(strcmp(argv[i], "-noI") == 0 || strcmp(argv[i], "-noInodeCompression") == 0) noI = TRUE; else if(strcmp(argv[i], "-noD") == 0 || strcmp(argv[i], "-noDataCompression") == 0) noD = TRUE; else if(strcmp(argv[i], "-noF") == 0 || strcmp(argv[i], "-noFragmentCompression") == 0) noF = TRUE; else if(strcmp(argv[i], "-noX") == 0 || strcmp(argv[i], "-noXattrCompression") == 0) noX = TRUE; else if(strcmp(argv[i], "-no-xattrs") == 0) no_xattrs = TRUE; else if(strcmp(argv[i], "-xattrs") == 0) no_xattrs = FALSE; else if(strcmp(argv[i], "-nopad") == 0) nopad = TRUE; else if(strcmp(argv[i], "-info") == 0) silent = FALSE; else if(strcmp(argv[i], "-e") == 0) break; else if(strcmp(argv[i], "-noappend") == 0) delete = TRUE; else if(strcmp(argv[i], "-keep-as-directory") == 0) keep_as_directory = TRUE; else if(strcmp(argv[i], "-exit-on-error") == 0) exit_on_error = TRUE; else if(strcmp(argv[i], "-root-becomes") == 0) { if(++i == argc) { ERROR("%s: -root-becomes: missing name\n", argv[0]); exit(1); } root_name = argv[i]; } else if(strcmp(argv[i], "-version") == 0) { VERSION(); } else { ERROR("%s: invalid option\n\n", argv[0]); printOptions: ERROR("SYNTAX:%s source1 source2 ... dest [options] " "[-e list of exclude\ndirs/files]\n", argv[0]); ERROR("\nFilesystem build options:\n"); ERROR("-comp \t\tselect compression\n"); ERROR("\t\t\tCompressors available:\n"); display_compressors("\t\t\t", COMP_DEFAULT); ERROR("-b \t\tset data block to " ". Default 128 Kbytes\n"); ERROR("\t\t\tOptionally a suffix of K or M can be" " given to specify\n\t\t\tKbytes or Mbytes" " respectively\n"); ERROR("-no-exports\t\tdon't make the filesystem " "exportable via NFS\n"); ERROR("-no-sparse\t\tdon't detect sparse files\n"); ERROR("-no-xattrs\t\tdon't store extended attributes" NOXOPT_STR "\n"); ERROR("-xattrs\t\t\tstore extended attributes" XOPT_STR "\n"); ERROR("-noI\t\t\tdo not compress inode table\n"); ERROR("-noD\t\t\tdo not compress data blocks\n"); ERROR("-noF\t\t\tdo not compress fragment blocks\n"); ERROR("-noX\t\t\tdo not compress extended " "attributes\n"); ERROR("-no-fragments\t\tdo not use fragments\n"); ERROR("-always-use-fragments\tuse fragment blocks for " "files larger than block size\n"); ERROR("-no-duplicates\t\tdo not perform duplicate " "checking\n"); ERROR("-all-root\t\tmake all files owned by root\n"); ERROR("-force-uid uid\t\tset all file uids to uid\n"); ERROR("-force-gid gid\t\tset all file gids to gid\n"); ERROR("-nopad\t\t\tdo not pad filesystem to a multiple " "of 4K\n"); ERROR("-keep-as-directory\tif one source directory is " "specified, create a root\n"); ERROR("\t\t\tdirectory containing that directory, " "rather than the\n"); ERROR("\t\t\tcontents of the directory\n"); ERROR("\nFilesystem filter options:\n"); ERROR("-p \tAdd pseudo file " "definition\n"); ERROR("-pf \tAdd list of pseudo file " "definitions\n"); ERROR("-sort \tsort files according to " "priorities in . One\n"); ERROR("\t\t\tfile or dir with priority per line. " "Priority -32768 to\n"); ERROR("\t\t\t32767, default priority 0\n"); ERROR("-ef \tlist of exclude dirs/files." " One per line\n"); ERROR("-wildcards\t\tAllow extended shell wildcards " "(globbing) to be used in\n\t\t\texclude " "dirs/files\n"); ERROR("-regex\t\t\tAllow POSIX regular expressions to " "be used in exclude\n\t\t\tdirs/files\n"); ERROR("\nFilesystem append options:\n"); ERROR("-noappend\t\tdo not append to existing " "filesystem\n"); ERROR("-root-becomes \twhen appending source " "files/directories, make the\n"); ERROR("\t\t\toriginal root become a subdirectory in " "the new root\n"); ERROR("\t\t\tcalled , rather than adding the new " "source items\n"); ERROR("\t\t\tto the original root\n"); ERROR("\nMksquashfs runtime options:\n"); ERROR("-version\t\tprint version, licence and " "copyright message\n"); ERROR("-exit-on-error\t\ttreat normally ignored errors " "as fatal\n"); ERROR("-recover \t\trecover filesystem data " "using recovery file \n"); ERROR("-no-recovery\t\tdon't generate a recovery " "file\n"); ERROR("-info\t\t\tprint files written to filesystem\n"); ERROR("-no-progress\t\tdon't display the progress " "bar\n"); ERROR("-progress\t\tdisplay progress bar when using " "the -info option\n"); ERROR("-processors \tUse processors." " By default will use number of\n"); ERROR("\t\t\tprocessors available\n"); ERROR("-mem \t\tUse physical memory. " "Currently set to %dM\n", total_mem); ERROR("\t\t\tOptionally a suffix of K, M or G can be" " given to specify\n\t\t\tKbytes, Mbytes or" " Gbytes respectively\n"); ERROR("\nMiscellaneous options:\n"); ERROR("-root-owned\t\talternative name for -all-root" "\n"); ERROR("-noInodeCompression\talternative name for -noI" "\n"); ERROR("-noDataCompression\talternative name for -noD" "\n"); ERROR("-noFragmentCompression\talternative name for " "-noF\n"); ERROR("-noXattrCompression\talternative name for " "-noX\n"); ERROR("\n-Xhelp\t\t\tprint compressor options for" " selected compressor\n"); ERROR("\nCompressors available and compressor specific " "options:\n"); display_compressor_usage(COMP_DEFAULT); exit(1); } } /* * Some compressors may need the options to be checked for validity * once all the options have been processed */ res = compressor_options_post(comp, block_size); if(res) EXIT_MKSQUASHFS(); /* * If the -info option has been selected then disable the * progress bar unless it has been explicitly enabled with * the -progress option */ if(!silent) progress = force_progress; #ifdef SQUASHFS_TRACE /* * Disable progress bar if full debug tracing is enabled. * The progress bar in this case just gets in the way of the * debug trace output */ progress = FALSE; #endif for(i = 0; i < source; i++) if(lstat(source_path[i], &source_buf) == -1) { fprintf(stderr, "Cannot stat source directory \"%s\" " "because %s\n", source_path[i], strerror(errno)); EXIT_MKSQUASHFS(); } destination_file = argv[source + 1]; if(stat(argv[source + 1], &buf) == -1) { if(errno == ENOENT) { /* Does not exist */ fd = open(argv[source + 1], O_CREAT | O_TRUNC | O_RDWR, S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH); if(fd == -1) { perror("Could not create destination file"); exit(1); } delete = TRUE; } else { perror("Could not stat destination file"); exit(1); } } else { if(S_ISBLK(buf.st_mode)) { if((fd = open(argv[source + 1], O_RDWR)) == -1) { perror("Could not open block device as " "destination"); exit(1); } block_device = 1; } else if(S_ISREG(buf.st_mode)) { fd = open(argv[source + 1], (delete ? O_TRUNC : 0) | O_RDWR); if(fd == -1) { perror("Could not open regular file for " "writing as destination"); exit(1); } } else { ERROR("Destination not block device or regular file\n"); exit(1); } } /* * process the exclude files - must be done afer destination file has * been possibly created */ for(i = source + 2; i < argc; i++) if(strcmp(argv[i], "-ef") == 0) /* * Note presence of filename arg has already * been checked */ process_exclude_file(argv[++i]); else if(strcmp(argv[i], "-e") == 0) break; else if(strcmp(argv[i], "-root-becomes") == 0 || strcmp(argv[i], "-sort") == 0 || strcmp(argv[i], "-pf") == 0 || strcmp(argv[i], "-af") == 0 || strcmp(argv[i], "-comp") == 0) i++; if(i != argc) { if(++i == argc) { ERROR("%s: -e missing arguments\n", argv[0]); EXIT_MKSQUASHFS(); } while(i < argc) if(old_exclude) old_add_exclude(argv[i++]); else add_exclude(argv[i++]); } /* process the sort files - must be done afer the exclude files */ for(i = source + 2; i < argc; i++) if(strcmp(argv[i], "-sort") == 0) { int res = read_sort_file(argv[++i], source, source_path); if(res == FALSE) BAD_ERROR("Failed to read sort file\n"); sorted ++; } else if(strcmp(argv[i], "-e") == 0) break; else if(strcmp(argv[i], "-root-becomes") == 0 || strcmp(argv[i], "-ef") == 0 || strcmp(argv[i], "-pf") == 0 || strcmp(argv[i], "-af") == 0 || strcmp(argv[i], "-comp") == 0) i++; if(!delete) { comp = read_super(fd, &sBlk, argv[source + 1]); if(comp == NULL) { ERROR("Failed to read existing filesystem - will not " "overwrite - ABORTING!\n"); ERROR("To force Mksquashfs to write to this block " "device or file use -noappend\n"); EXIT_MKSQUASHFS(); } block_log = slog(block_size = sBlk.block_size); noI = SQUASHFS_UNCOMPRESSED_INODES(sBlk.flags); noD = SQUASHFS_UNCOMPRESSED_DATA(sBlk.flags); noF = SQUASHFS_UNCOMPRESSED_FRAGMENTS(sBlk.flags); noX = SQUASHFS_UNCOMPRESSED_XATTRS(sBlk.flags); no_fragments = SQUASHFS_NO_FRAGMENTS(sBlk.flags); always_use_fragments = SQUASHFS_ALWAYS_FRAGMENTS(sBlk.flags); duplicate_checking = SQUASHFS_DUPLICATES(sBlk.flags); exportable = SQUASHFS_EXPORTABLE(sBlk.flags); no_xattrs = SQUASHFS_NO_XATTRS(sBlk.flags); comp_opts = SQUASHFS_COMP_OPTS(sBlk.flags); } initialise_threads(readq, fragq, bwriteq, fwriteq, delete, destination_file); res = compressor_init(comp, &stream, SQUASHFS_METADATA_SIZE, 0); if(res) BAD_ERROR("compressor_init failed\n"); if(delete) { int size; void *comp_data = compressor_dump_options(comp, block_size, &size); printf("Creating %d.%d filesystem on %s, block size %d.\n", SQUASHFS_MAJOR, SQUASHFS_MINOR, argv[source + 1], block_size); /* * store any compressor specific options after the superblock, * and set the COMP_OPT flag to show that the filesystem has * compressor specfic options */ if(comp_data) { unsigned short c_byte = size | SQUASHFS_COMPRESSED_BIT; SQUASHFS_INSWAP_SHORTS(&c_byte, 1); write_destination(fd, sizeof(struct squashfs_super_block), sizeof(c_byte), &c_byte); write_destination(fd, sizeof(struct squashfs_super_block) + sizeof(c_byte), size, comp_data); bytes = sizeof(struct squashfs_super_block) + sizeof(c_byte) + size; comp_opts = TRUE; } else bytes = sizeof(struct squashfs_super_block); } else { unsigned int last_directory_block, inode_dir_offset, inode_dir_file_size, root_inode_size, inode_dir_start_block, uncompressed_data, compressed_data, inode_dir_inode_number, inode_dir_parent_inode; unsigned int root_inode_start = SQUASHFS_INODE_BLK(sBlk.root_inode), root_inode_offset = SQUASHFS_INODE_OFFSET(sBlk.root_inode); if((bytes = read_filesystem(root_name, fd, &sBlk, &inode_table, &data_cache, &directory_table, &directory_data_cache, &last_directory_block, &inode_dir_offset, &inode_dir_file_size, &root_inode_size, &inode_dir_start_block, &file_count, &sym_count, &dev_count, &dir_count, &fifo_count, &sock_count, &total_bytes, &total_inode_bytes, &total_directory_bytes, &inode_dir_inode_number, &inode_dir_parent_inode, add_old_root_entry, &fragment_table, &inode_lookup_table)) == 0) { ERROR("Failed to read existing filesystem - will not " "overwrite - ABORTING!\n"); ERROR("To force Mksquashfs to write to this block " "device or file use -noappend\n"); EXIT_MKSQUASHFS(); } if((append_fragments = fragments = sBlk.fragments)) { fragment_table = realloc((char *) fragment_table, ((fragments + FRAG_SIZE - 1) & ~(FRAG_SIZE - 1)) * sizeof(struct squashfs_fragment_entry)); if(fragment_table == NULL) BAD_ERROR("Out of memory in save filesystem state\n"); } printf("Appending to existing %d.%d filesystem on %s, block " "size %d\n", SQUASHFS_MAJOR, SQUASHFS_MINOR, argv[source + 1], block_size); printf("All -b, -noI, -noD, -noF, -noX, no-duplicates, no-fragments, " "-always-use-fragments,\n-exportable and -comp options " "ignored\n"); printf("\nIf appending is not wanted, please re-run with " "-noappend specified!\n\n"); compressed_data = (inode_dir_offset + inode_dir_file_size) & ~(SQUASHFS_METADATA_SIZE - 1); uncompressed_data = (inode_dir_offset + inode_dir_file_size) & (SQUASHFS_METADATA_SIZE - 1); /* save original filesystem state for restoring ... */ sfragments = fragments; sbytes = bytes; sinode_count = sBlk.inodes; scache_bytes = root_inode_offset + root_inode_size; sdirectory_cache_bytes = uncompressed_data; sdata_cache = malloc(scache_bytes); if(sdata_cache == NULL) BAD_ERROR("Out of memory in save filesystem state\n"); sdirectory_data_cache = malloc(sdirectory_cache_bytes); if(sdirectory_data_cache == NULL) BAD_ERROR("Out of memory in save filesystem state\n"); memcpy(sdata_cache, data_cache, scache_bytes); memcpy(sdirectory_data_cache, directory_data_cache + compressed_data, sdirectory_cache_bytes); sinode_bytes = root_inode_start; stotal_bytes = total_bytes; stotal_inode_bytes = total_inode_bytes; stotal_directory_bytes = total_directory_bytes + compressed_data; sfile_count = file_count; ssym_count = sym_count; sdev_count = dev_count; sdir_count = dir_count + 1; sfifo_count = fifo_count; ssock_count = sock_count; sdup_files = dup_files; sid_count = id_count; write_recovery_data(&sBlk); save_xattrs(); appending = TRUE; /* * set the filesystem state up to be able to append to the * original filesystem. The filesystem state differs depending * on whether we're appending to the original root directory, or * if the original root directory becomes a sub-directory * (root-becomes specified on command line, here root_name != * NULL) */ inode_bytes = inode_size = root_inode_start; directory_size = last_directory_block; cache_size = root_inode_offset + root_inode_size; directory_cache_size = inode_dir_offset + inode_dir_file_size; if(root_name) { sdirectory_bytes = last_directory_block; sdirectory_compressed_bytes = 0; root_inode_number = inode_dir_parent_inode; inode_no = sBlk.inodes + 2; directory_bytes = last_directory_block; directory_cache_bytes = uncompressed_data; memmove(directory_data_cache, directory_data_cache + compressed_data, uncompressed_data); cache_bytes = root_inode_offset + root_inode_size; add_old_root_entry(root_name, sBlk.root_inode, inode_dir_inode_number, SQUASHFS_DIR_TYPE); total_directory_bytes += compressed_data; dir_count ++; } else { sdirectory_compressed_bytes = last_directory_block - inode_dir_start_block; sdirectory_compressed = malloc(sdirectory_compressed_bytes); if(sdirectory_compressed == NULL) BAD_ERROR("Out of memory in save filesystem " "state\n"); memcpy(sdirectory_compressed, directory_table + inode_dir_start_block, sdirectory_compressed_bytes); sdirectory_bytes = inode_dir_start_block; root_inode_number = inode_dir_inode_number; inode_no = sBlk.inodes + 1; directory_bytes = inode_dir_start_block; directory_cache_bytes = inode_dir_offset; cache_bytes = root_inode_offset; } inode_count = file_count + dir_count + sym_count + dev_count + fifo_count + sock_count; } if(path) paths = add_subdir(paths, path); dump_actions(); dump_pseudos(); if(delete && !keep_as_directory && source == 1 && S_ISDIR(source_buf.st_mode)) dir_scan(&inode, source_path[0], scan1_readdir, progress); else if(!keep_as_directory && source == 1 && S_ISDIR(source_buf.st_mode)) dir_scan(&inode, source_path[0], scan1_single_readdir, progress); else dir_scan(&inode, "", scan1_encomp_readdir, progress); sBlk.root_inode = inode; sBlk.inodes = inode_count; sBlk.s_magic = SQUASHFS_MAGIC; sBlk.s_major = SQUASHFS_MAJOR; sBlk.s_minor = SQUASHFS_MINOR; sBlk.block_size = block_size; sBlk.block_log = block_log; sBlk.flags = SQUASHFS_MKFLAGS(noI, noD, noF, noX, no_fragments, always_use_fragments, duplicate_checking, exportable, no_xattrs, comp_opts); sBlk.mkfs_time = time(NULL); disable_info(); while((fragment = get_frag_action(fragment))) write_fragment(*fragment); unlock_fragments(); pthread_cleanup_push((void *) pthread_mutex_unlock, &fragment_mutex); pthread_mutex_lock(&fragment_mutex); while(fragments_outstanding) { pthread_mutex_unlock(&fragment_mutex); sched_yield(); pthread_mutex_lock(&fragment_mutex); } pthread_cleanup_pop(1); queue_put(to_writer, NULL); if(queue_get(from_writer) != 0) EXIT_MKSQUASHFS(); set_progressbar_state(FALSE); write_filesystem_tables(&sBlk, nopad); return 0; } squashfs4.3/squashfs-tools/unsquashfs_info.c0000644000175000017500000000613312306776317021415 0ustar phillipphillip/* * Create a squashfs filesystem. This is a highly compressed read only * filesystem. * * Copyright (c) 2013 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * unsquashfs_info.c */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "squashfs_fs.h" #include "unsquashfs.h" #include "error.h" static int silent = 0; char *pathname = NULL; pthread_t info_thread; void disable_info() { if(pathname) free(pathname); pathname = NULL; } void update_info(char *name) { if(pathname) free(pathname); pathname = name; } void dump_state() { disable_progress_bar(); printf("Queue and cache status dump\n"); printf("===========================\n"); printf("file buffer read queue (main thread -> reader thread)\n"); dump_queue(to_reader); printf("file buffer decompress queue (reader thread -> inflate" " thread(s))\n"); dump_queue(to_inflate); printf("file buffer write queue (main thread -> writer thread)\n"); dump_queue(to_writer); printf("\nbuffer cache (uncompressed blocks and compressed blocks " "'in flight')\n"); dump_cache(data_cache); printf("fragment buffer cache (uncompressed frags and compressed" " frags 'in flight')\n"); dump_cache(fragment_cache); enable_progress_bar(); } void *info_thrd(void *arg) { sigset_t sigmask; struct timespec timespec = { .tv_sec = 1, .tv_nsec = 0 }; int sig, waiting = 0; sigemptyset(&sigmask); sigaddset(&sigmask, SIGQUIT); sigaddset(&sigmask, SIGHUP); while(1) { if(waiting) sig = sigtimedwait(&sigmask, NULL, ×pec); else sig = sigwaitinfo(&sigmask, NULL); if(sig == -1) { switch(errno) { case EAGAIN: /* interval timed out */ waiting = 0; /* FALLTHROUGH */ case EINTR: /* if waiting, the wait will be longer, but that's OK */ continue; default: BAD_ERROR("sigtimedwait/sigwaitinfo failed " "because %s\n", strerror(errno)); } } if(sig == SIGQUIT && !waiting) { if(pathname) INFO("%s\n", pathname); /* set one second interval period, if ^\ received within then, dump queue and cache status */ waiting = 1; } else dump_state(); } } void init_info() { pthread_create(&info_thread, NULL, info_thrd, NULL); } squashfs4.3/squashfs-tools/unsquashfs.h0000644000175000017500000001502412333330365020373 0ustar phillipphillip#ifndef UNSQUASHFS_H #define UNSQUASHFS_H /* * Unsquash a squashfs filesystem. This is a highly compressed read only * filesystem. * * Copyright (c) 2009, 2010, 2013, 2014 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * unsquashfs.h */ #define TRUE 1 #define FALSE 0 #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #ifndef linux #define __BYTE_ORDER BYTE_ORDER #define __BIG_ENDIAN BIG_ENDIAN #define __LITTLE_ENDIAN LITTLE_ENDIAN #else #include #endif #include "squashfs_fs.h" #include "error.h" #define CALCULATE_HASH(start) (start & 0xffff) /* * Unified superblock containing fields for all superblocks */ struct super_block { struct squashfs_super_block s; /* fields only used by squashfs 3 and earlier layouts */ unsigned int no_uids; unsigned int no_guids; long long uid_start; long long guid_start; }; struct hash_table_entry { long long start; int bytes; struct hash_table_entry *next; }; struct inode { int blocks; char *block_ptr; long long data; int fragment; int frag_bytes; gid_t gid; int inode_number; int mode; int offset; long long start; char *symlink; time_t time; int type; uid_t uid; char sparse; unsigned int xattr; }; typedef struct squashfs_operations { struct dir *(*squashfs_opendir)(unsigned int block_start, unsigned int offset, struct inode **i); void (*read_fragment)(unsigned int fragment, long long *start_block, int *size); int (*read_fragment_table)(long long *); void (*read_block_list)(unsigned int *block_list, char *block_ptr, int blocks); struct inode *(*read_inode)(unsigned int start_block, unsigned int offset); int (*read_uids_guids)(); } squashfs_operations; struct test { int mask; int value; int position; char mode; }; /* Cache status struct. Caches are used to keep track of memory buffers passed between different threads */ struct cache { int max_buffers; int count; int used; int buffer_size; int wait_free; int wait_pending; pthread_mutex_t mutex; pthread_cond_t wait_for_free; pthread_cond_t wait_for_pending; struct cache_entry *free_list; struct cache_entry *hash_table[65536]; }; /* struct describing a cache entry passed between threads */ struct cache_entry { struct cache *cache; long long block; int size; int used; int error; int pending; struct cache_entry *hash_next; struct cache_entry *hash_prev; struct cache_entry *free_next; struct cache_entry *free_prev; char *data; }; /* struct describing queues used to pass data between threads */ struct queue { int size; int readp; int writep; pthread_mutex_t mutex; pthread_cond_t empty; pthread_cond_t full; void **data; }; /* default size of fragment buffer in Mbytes */ #define FRAGMENT_BUFFER_DEFAULT 256 /* default size of data buffer in Mbytes */ #define DATA_BUFFER_DEFAULT 256 #define DIR_ENT_SIZE 16 struct dir_ent { char name[SQUASHFS_NAME_LEN + 1]; unsigned int start_block; unsigned int offset; unsigned int type; }; struct dir { int dir_count; int cur_entry; unsigned int mode; uid_t uid; gid_t guid; unsigned int mtime; unsigned int xattr; struct dir_ent *dirs; }; struct file_entry { int offset; int size; struct cache_entry *buffer; }; struct squashfs_file { int fd; int blocks; long long file_size; int mode; uid_t uid; gid_t gid; time_t time; char *pathname; char sparse; unsigned int xattr; }; struct path_entry { char *name; regex_t *preg; struct pathname *paths; }; struct pathname { int names; struct path_entry *name; }; struct pathnames { int count; struct pathname *path[0]; }; #define PATHS_ALLOC_SIZE 10 /* globals */ extern struct super_block sBlk; extern squashfs_operations s_ops; extern int swap; extern char *inode_table, *directory_table; extern struct hash_table_entry *inode_table_hash[65536], *directory_table_hash[65536]; extern unsigned int *uid_table, *guid_table; extern pthread_mutex_t screen_mutex; extern int progress_enabled; extern int inode_number; extern int lookup_type[]; extern int fd; extern struct queue *to_reader, *to_inflate, *to_writer; extern struct cache *fragment_cache, *data_cache; /* unsquashfs.c */ extern int lookup_entry(struct hash_table_entry **, long long); extern int read_fs_bytes(int fd, long long, int, void *); extern int read_block(int, long long, long long *, int, void *); extern void enable_progress_bar(); extern void disable_progress_bar(); extern void dump_queue(struct queue *); extern void dump_cache(struct cache *); /* unsquash-1.c */ extern void read_block_list_1(unsigned int *, char *, int); extern int read_fragment_table_1(long long *); extern struct inode *read_inode_1(unsigned int, unsigned int); extern struct dir *squashfs_opendir_1(unsigned int, unsigned int, struct inode **); extern int read_uids_guids_1(); /* unsquash-2.c */ extern void read_block_list_2(unsigned int *, char *, int); extern int read_fragment_table_2(long long *); extern void read_fragment_2(unsigned int, long long *, int *); extern struct inode *read_inode_2(unsigned int, unsigned int); /* unsquash-3.c */ extern int read_fragment_table_3(long long *); extern void read_fragment_3(unsigned int, long long *, int *); extern struct inode *read_inode_3(unsigned int, unsigned int); extern struct dir *squashfs_opendir_3(unsigned int, unsigned int, struct inode **); /* unsquash-4.c */ extern int read_fragment_table_4(long long *); extern void read_fragment_4(unsigned int, long long *, int *); extern struct inode *read_inode_4(unsigned int, unsigned int); extern struct dir *squashfs_opendir_4(unsigned int, unsigned int, struct inode **); extern int read_uids_guids_4(); #endif squashfs4.3/squashfs-tools/Makefile0000644000175000017500000002061712333743700017467 0ustar phillipphillip############################################### # Compression build options # ############################################### # # ############# Building gzip support ########### # # Gzip support is by default enabled, and the compression type default # (COMP_DEFAULT) is gzip. # # If you don't want/need gzip support then comment out the GZIP SUPPORT line # below, and change COMP_DEFAULT to one of the compression types you have # selected. # # Obviously, you must select at least one of the available gzip, lzma, lzo # compression types. # GZIP_SUPPORT = 1 ########### Building XZ support ############# # # LZMA2 compression. # # XZ Utils liblzma (http://tukaani.org/xz/) is supported # # To build using XZ Utils liblzma - install the library and uncomment # the XZ_SUPPORT line below. # #XZ_SUPPORT = 1 ############ Building LZO support ############## # # The LZO library (http://www.oberhumer.com/opensource/lzo/) is supported. # # To build using the LZO library - install the library and uncomment the # LZO_SUPPORT line below. If needed, uncomment and set LZO_DIR to the # installation prefix. # #LZO_SUPPORT = 1 #LZO_DIR = /usr/local ########### Building LZ4 support ############# # # Yann Collet's LZ4 tools are supported # LZ4 homepage: http://fastcompression.blogspot.com/p/lz4.html # LZ4 source repository: http://code.google.com/p/lz4 # # To build configure the tools using cmake to build shared libraries, # install and uncomment # the LZ4_SUPPORT line below. # #LZ4_SUPPORT = 1 ########### Building LZMA support ############# # # LZMA1 compression. # # LZMA1 compression is deprecated, and the newer and better XZ (LZMA2) # compression should be used in preference. # # Both XZ Utils liblzma (http://tukaani.org/xz/) and LZMA SDK # (http://www.7-zip.org/sdk.html) are supported # # To build using XZ Utils liblzma - install the library and uncomment # the LZMA_XZ_SUPPORT line below. # # To build using the LZMA SDK (4.65 used in development, other versions may # work) - download and unpack it, uncomment and set LZMA_DIR to unpacked source, # and uncomment the LZMA_SUPPORT line below. # #LZMA_XZ_SUPPORT = 1 #LZMA_SUPPORT = 1 #LZMA_DIR = ../../../../LZMA/lzma465 ######## Specifying default compression ######## # # The next line specifies which compression algorithm is used by default # in Mksquashfs. Obviously the compression algorithm must have been # selected to be built # COMP_DEFAULT = gzip ############################################### # Extended attribute (XATTRs) build options # ############################################### # # Building XATTR support for Mksquashfs and Unsquashfs # # If your C library or build/target environment doesn't support XATTRs then # comment out the next line to build Mksquashfs and Unsquashfs without XATTR # support XATTR_SUPPORT = 1 # Select whether you wish xattrs to be stored by Mksquashfs and extracted # by Unsquashfs by default. If selected users can disable xattr support by # using the -no-xattrs option # # If unselected, Mksquashfs/Unsquashfs won't store and extract xattrs by # default. Users can enable xattrs by using the -xattrs option. XATTR_DEFAULT = 1 ############################################### # End of BUILD options section # ############################################### INCLUDEDIR = -I. INSTALL_DIR = /usr/local/bin MKSQUASHFS_OBJS = mksquashfs.o read_fs.o action.o swap.o pseudo.o compressor.o \ sort.o progressbar.o read_file.o info.o restore.o process_fragments.o \ caches-queues-lists.o UNSQUASHFS_OBJS = unsquashfs.o unsquash-1.o unsquash-2.o unsquash-3.o \ unsquash-4.o swap.o compressor.o unsquashfs_info.o CFLAGS ?= -O2 CFLAGS += $(EXTRA_CFLAGS) $(INCLUDEDIR) -D_FILE_OFFSET_BITS=64 \ -D_LARGEFILE_SOURCE -D_GNU_SOURCE -DCOMP_DEFAULT=\"$(COMP_DEFAULT)\" \ -Wall LIBS = -lpthread -lm ifeq ($(GZIP_SUPPORT),1) CFLAGS += -DGZIP_SUPPORT MKSQUASHFS_OBJS += gzip_wrapper.o UNSQUASHFS_OBJS += gzip_wrapper.o LIBS += -lz COMPRESSORS += gzip endif ifeq ($(LZMA_SUPPORT),1) LZMA_OBJS = $(LZMA_DIR)/C/Alloc.o $(LZMA_DIR)/C/LzFind.o \ $(LZMA_DIR)/C/LzmaDec.o $(LZMA_DIR)/C/LzmaEnc.o $(LZMA_DIR)/C/LzmaLib.o INCLUDEDIR += -I$(LZMA_DIR)/C CFLAGS += -DLZMA_SUPPORT MKSQUASHFS_OBJS += lzma_wrapper.o $(LZMA_OBJS) UNSQUASHFS_OBJS += lzma_wrapper.o $(LZMA_OBJS) COMPRESSORS += lzma endif ifeq ($(LZMA_XZ_SUPPORT),1) CFLAGS += -DLZMA_SUPPORT MKSQUASHFS_OBJS += lzma_xz_wrapper.o UNSQUASHFS_OBJS += lzma_xz_wrapper.o LIBS += -llzma COMPRESSORS += lzma endif ifeq ($(XZ_SUPPORT),1) CFLAGS += -DXZ_SUPPORT MKSQUASHFS_OBJS += xz_wrapper.o UNSQUASHFS_OBJS += xz_wrapper.o LIBS += -llzma COMPRESSORS += xz endif ifeq ($(LZO_SUPPORT),1) CFLAGS += -DLZO_SUPPORT ifdef LZO_DIR INCLUDEDIR += -I$(LZO_DIR)/include LZO_LIBDIR = -L$(LZO_DIR)/lib endif MKSQUASHFS_OBJS += lzo_wrapper.o UNSQUASHFS_OBJS += lzo_wrapper.o LIBS += $(LZO_LIBDIR) -llzo2 COMPRESSORS += lzo endif ifeq ($(LZ4_SUPPORT),1) CFLAGS += -DLZ4_SUPPORT MKSQUASHFS_OBJS += lz4_wrapper.o UNSQUASHFS_OBJS += lz4_wrapper.o LIBS += -llz4 COMPRESSORS += lz4 endif ifeq ($(XATTR_SUPPORT),1) ifeq ($(XATTR_DEFAULT),1) CFLAGS += -DXATTR_SUPPORT -DXATTR_DEFAULT else CFLAGS += -DXATTR_SUPPORT endif MKSQUASHFS_OBJS += xattr.o read_xattrs.o UNSQUASHFS_OBJS += read_xattrs.o unsquashfs_xattr.o endif # # If LZMA_SUPPORT is specified then LZMA_DIR must be specified too # ifeq ($(LZMA_SUPPORT),1) ifndef LZMA_DIR $(error "LZMA_SUPPORT requires LZMA_DIR to be also defined") endif endif # # Both LZMA_XZ_SUPPORT and LZMA_SUPPORT cannot be specified # ifeq ($(LZMA_XZ_SUPPORT),1) ifeq ($(LZMA_SUPPORT),1) $(error "Both LZMA_XZ_SUPPORT and LZMA_SUPPORT cannot be specified") endif endif # # At least one compressor must have been selected # ifndef COMPRESSORS $(error "No compressor selected! Select one or more of GZIP, LZMA, XZ, LZO or \ LZ4!") endif # # COMP_DEFAULT must be a selected compressor # ifeq (, $(findstring $(COMP_DEFAULT), $(COMPRESSORS))) $(error "COMP_DEFAULT is set to ${COMP_DEFAULT}, which isn't selected to be \ built!") endif .PHONY: all all: mksquashfs unsquashfs mksquashfs: $(MKSQUASHFS_OBJS) $(CC) $(LDFLAGS) $(EXTRA_LDFLAGS) $(MKSQUASHFS_OBJS) $(LIBS) -o $@ mksquashfs.o: Makefile mksquashfs.c squashfs_fs.h squashfs_swap.h mksquashfs.h \ sort.h pseudo.h compressor.h xattr.h action.h error.h progressbar.h \ info.h caches-queues-lists.h read_fs.h restore.h process_fragments.h read_fs.o: read_fs.c squashfs_fs.h squashfs_swap.h compressor.h xattr.h \ error.h mksquashfs.h sort.o: sort.c squashfs_fs.h mksquashfs.h sort.h error.h progressbar.h swap.o: swap.c pseudo.o: pseudo.c pseudo.h error.h progressbar.h compressor.o: Makefile compressor.c compressor.h squashfs_fs.h xattr.o: xattr.c squashfs_fs.h squashfs_swap.h mksquashfs.h xattr.h error.h \ progressbar.h read_xattrs.o: read_xattrs.c squashfs_fs.h squashfs_swap.h xattr.h error.h action.o: action.c squashfs_fs.h mksquashfs.h action.h error.h progressbar.o: progressbar.c error.h read_file.o: read_file.c error.h info.o: info.c squashfs_fs.h mksquashfs.h error.h progressbar.h \ caches-queues-lists.h restore.o: restore.c caches-queues-lists.h squashfs_fs.h mksquashfs.h error.h \ progressbar.h info.h process_fragments.o: process_fragments.c process_fragments.h caches-queues-lists.o: caches-queues-lists.c error.h caches-queues-lists.h gzip_wrapper.o: gzip_wrapper.c squashfs_fs.h gzip_wrapper.h compressor.h lzma_wrapper.o: lzma_wrapper.c compressor.h squashfs_fs.h lzma_xz_wrapper.o: lzma_xz_wrapper.c compressor.h squashfs_fs.h lzo_wrapper.o: lzo_wrapper.c squashfs_fs.h lzo_wrapper.h compressor.h lz4_wrapper.o: lz4_wrapper.c squashfs_fs.h lz4_wrapper.h compressor.h xz_wrapper.o: xz_wrapper.c squashfs_fs.h xz_wrapper.h compressor.h unsquashfs: $(UNSQUASHFS_OBJS) $(CC) $(LDFLAGS) $(EXTRA_LDFLAGS) $(UNSQUASHFS_OBJS) $(LIBS) -o $@ unsquashfs.o: unsquashfs.h unsquashfs.c squashfs_fs.h squashfs_swap.h \ squashfs_compat.h xattr.h read_fs.h compressor.h unsquash-1.o: unsquashfs.h unsquash-1.c squashfs_fs.h squashfs_compat.h unsquash-2.o: unsquashfs.h unsquash-2.c squashfs_fs.h squashfs_compat.h unsquash-3.o: unsquashfs.h unsquash-3.c squashfs_fs.h squashfs_compat.h unsquash-4.o: unsquashfs.h unsquash-4.c squashfs_fs.h squashfs_swap.h \ read_fs.h unsquashfs_xattr.o: unsquashfs_xattr.c unsquashfs.h squashfs_fs.h xattr.h unsquashfs_info.o: unsquashfs.h squashfs_fs.h .PHONY: clean clean: -rm -f *.o mksquashfs unsquashfs .PHONY: install install: mksquashfs unsquashfs mkdir -p $(INSTALL_DIR) cp mksquashfs $(INSTALL_DIR) cp unsquashfs $(INSTALL_DIR) squashfs4.3/squashfs-tools/progressbar.c0000644000175000017500000001304012333330365020513 0ustar phillipphillip/* * Create a squashfs filesystem. This is a highly compressed read only * filesystem. * * Copyright (c) 2012, 2013, 2014 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * progressbar.c */ #include #include #include #include #include #include #include #include #include #include #include "error.h" #define FALSE 0 #define TRUE 1 /* flag whether progressbar display is enabled or not */ int display_progress_bar = FALSE; /* flag whether the progress bar is temporarily disbled */ int temp_disabled = FALSE; int rotate = 0; int cur_uncompressed = 0, estimated_uncompressed = 0; int columns; pthread_t progress_thread; pthread_mutex_t progress_mutex = PTHREAD_MUTEX_INITIALIZER; static void sigwinch_handler() { struct winsize winsize; if(ioctl(1, TIOCGWINSZ, &winsize) == -1) { if(isatty(STDOUT_FILENO)) ERROR("TIOCGWINSZ ioctl failed, defaulting to 80 " "columns\n"); columns = 80; } else columns = winsize.ws_col; } static void sigalrm_handler() { rotate = (rotate + 1) % 4; } void inc_progress_bar() { cur_uncompressed ++; } void dec_progress_bar(int count) { cur_uncompressed -= count; } void progress_bar_size(int count) { estimated_uncompressed += count; } static void progress_bar(long long current, long long max, int columns) { char rotate_list[] = { '|', '/', '-', '\\' }; int max_digits, used, hashes, spaces; static int tty = -1; if(max == 0) return; max_digits = floor(log10(max)) + 1; used = max_digits * 2 + 11; hashes = (current * (columns - used)) / max; spaces = columns - used - hashes; if((current > max) || (columns - used < 0)) return; if(tty == -1) tty = isatty(STDOUT_FILENO); if(!tty) { static long long previous = -1; /* Updating much more frequently than this results in huge * log files. */ if((current % 100) != 0 && current != max) return; /* Don't update just to rotate the spinner. */ if(current == previous) return; previous = current; } printf("\r["); while (hashes --) putchar('='); putchar(rotate_list[rotate]); while(spaces --) putchar(' '); printf("] %*lld/%*lld", max_digits, current, max_digits, max); printf(" %3lld%%", current * 100 / max); fflush(stdout); } void enable_progress_bar() { pthread_cleanup_push((void *) pthread_mutex_unlock, &progress_mutex); pthread_mutex_lock(&progress_mutex); if(display_progress_bar) progress_bar(cur_uncompressed, estimated_uncompressed, columns); temp_disabled = FALSE; pthread_cleanup_pop(1); } void disable_progress_bar() { pthread_cleanup_push((void *) pthread_mutex_unlock, &progress_mutex); pthread_mutex_lock(&progress_mutex); if(display_progress_bar) printf("\n"); temp_disabled = TRUE; pthread_cleanup_pop(1); } void set_progressbar_state(int state) { pthread_cleanup_push((void *) pthread_mutex_unlock, &progress_mutex); pthread_mutex_lock(&progress_mutex); if(display_progress_bar != state) { if(display_progress_bar && !temp_disabled) { progress_bar(cur_uncompressed, estimated_uncompressed, columns); printf("\n"); } display_progress_bar = state; } pthread_cleanup_pop(1); } void *progress_thrd(void *arg) { struct timespec requested_time, remaining; struct itimerval itimerval; struct winsize winsize; if(ioctl(1, TIOCGWINSZ, &winsize) == -1) { if(isatty(STDOUT_FILENO)) ERROR("TIOCGWINSZ ioctl failed, defaulting to 80 " "columns\n"); columns = 80; } else columns = winsize.ws_col; signal(SIGWINCH, sigwinch_handler); signal(SIGALRM, sigalrm_handler); itimerval.it_value.tv_sec = 0; itimerval.it_value.tv_usec = 250000; itimerval.it_interval.tv_sec = 0; itimerval.it_interval.tv_usec = 250000; setitimer(ITIMER_REAL, &itimerval, NULL); requested_time.tv_sec = 0; requested_time.tv_nsec = 250000000; while(1) { int res = nanosleep(&requested_time, &remaining); if(res == -1 && errno != EINTR) BAD_ERROR("nanosleep failed in progress thread\n"); pthread_mutex_lock(&progress_mutex); if(display_progress_bar && !temp_disabled) progress_bar(cur_uncompressed, estimated_uncompressed, columns); pthread_mutex_unlock(&progress_mutex); } } void init_progress_bar() { pthread_create(&progress_thread, NULL, progress_thrd, NULL); } void progressbar_error(char *fmt, ...) { va_list ap; pthread_cleanup_push((void *) pthread_mutex_unlock, &progress_mutex); pthread_mutex_lock(&progress_mutex); if(display_progress_bar && !temp_disabled) fprintf(stderr, "\n"); va_start(ap, fmt); vfprintf(stderr, fmt, ap); va_end(ap); pthread_cleanup_pop(1); } void progressbar_info(char *fmt, ...) { va_list ap; pthread_cleanup_push((void *) pthread_mutex_unlock, &progress_mutex); pthread_mutex_lock(&progress_mutex); if(display_progress_bar && !temp_disabled) printf("\n"); va_start(ap, fmt); vprintf(fmt, ap); va_end(ap); pthread_cleanup_pop(1); } squashfs4.3/squashfs-tools/swap.c0000644000175000017500000000467712306776317017167 0ustar phillipphillip/* * Copyright (c) 2009, 2010 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * swap.c */ #ifndef linux #define __BYTE_ORDER BYTE_ORDER #define __BIG_ENDIAN BIG_ENDIAN #define __LITTLE_ENDIAN LITTLE_ENDIAN #else #include #endif #if __BYTE_ORDER == __BIG_ENDIAN void swap_le16(void *src, void *dest) { unsigned char *s = src; unsigned char *d = dest; d[0] = s[1]; d[1] = s[0]; } void swap_le32(void *src, void *dest) { unsigned char *s = src; unsigned char *d = dest; d[0] = s[3]; d[1] = s[2]; d[2] = s[1]; d[3] = s[0]; } void swap_le64(void *src, void *dest) { unsigned char *s = src; unsigned char *d = dest; d[0] = s[7]; d[1] = s[6]; d[2] = s[5]; d[3] = s[4]; d[4] = s[3]; d[5] = s[2]; d[6] = s[1]; d[7] = s[0]; } unsigned short inswap_le16(unsigned short num) { return (num >> 8) | ((num & 0xff) << 8); } unsigned int inswap_le32(unsigned int num) { return (num >> 24) | ((num & 0xff0000) >> 8) | ((num & 0xff00) << 8) | ((num & 0xff) << 24); } long long inswap_le64(long long n) { unsigned long long num = n; return (num >> 56) | ((num & 0xff000000000000LL) >> 40) | ((num & 0xff0000000000LL) >> 24) | ((num & 0xff00000000LL) >> 8) | ((num & 0xff000000) << 8) | ((num & 0xff0000) << 24) | ((num & 0xff00) << 40) | ((num & 0xff) << 56); } #define SWAP_LE_NUM(BITS) \ void swap_le##BITS##_num(void *s, void *d, int n) \ {\ int i;\ for(i = 0; i < n; i++, s += BITS / 8, d += BITS / 8)\ swap_le##BITS(s, d);\ } SWAP_LE_NUM(16) SWAP_LE_NUM(32) SWAP_LE_NUM(64) #define INSWAP_LE_NUM(BITS, TYPE) \ void inswap_le##BITS##_num(TYPE *s, int n) \ {\ int i;\ for(i = 0; i < n; i++)\ s[i] = inswap_le##BITS(s[i]);\ } INSWAP_LE_NUM(16, unsigned short) INSWAP_LE_NUM(32, unsigned int) INSWAP_LE_NUM(64, long long) #endif squashfs4.3/squashfs-tools/action.c0000644000175000017500000014272712333330365017456 0ustar phillipphillip/* * Create a squashfs filesystem. This is a highly compressed read only * filesystem. * * Copyright (c) 2011, 2012, 2013, 2014 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * action.c */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "squashfs_fs.h" #include "mksquashfs.h" #include "action.h" #include "error.h" /* * code to parse actions */ static char *cur_ptr, *source; static struct action *fragment_spec = NULL; static struct action *exclude_spec = NULL; static struct action *empty_spec = NULL; static struct action *move_spec = NULL; static struct action *other_spec = NULL; static int fragment_count = 0; static int exclude_count = 0; static int empty_count = 0; static int move_count = 0; static int other_count = 0; static struct file_buffer *def_fragment = NULL; static struct token_entry token_table[] = { { "(", TOK_OPEN_BRACKET, 1, }, { ")", TOK_CLOSE_BRACKET, 1 }, { "&&", TOK_AND, 2 }, { "||", TOK_OR, 2 }, { "!", TOK_NOT, 1 }, { ",", TOK_COMMA, 1 }, { "@", TOK_AT, 1}, { " ", TOK_WHITE_SPACE, 1 }, { "\t ", TOK_WHITE_SPACE, 1 }, { "", -1, 0 } }; static struct test_entry test_table[]; static struct action_entry action_table[]; static struct expr *parse_expr(int subexp); extern char *pathname(struct dir_ent *); extern char *subpathname(struct dir_ent *); extern int read_file(char *filename, char *type, int (parse_line)(char *)); /* * Lexical analyser */ #define STR_SIZE 256 static int get_token(char **string) { /* string buffer */ static char *str = NULL; static int size = 0; char *str_ptr; int cur_size, i, quoted; while (1) { if (*cur_ptr == '\0') return TOK_EOF; for (i = 0; token_table[i].token != -1; i++) if (strncmp(cur_ptr, token_table[i].string, token_table[i].size) == 0) break; if (token_table[i].token != TOK_WHITE_SPACE) break; cur_ptr ++; } if (token_table[i].token != -1) { cur_ptr += token_table[i].size; return token_table[i].token; } /* string */ if(str == NULL) { str = malloc(STR_SIZE); if(str == NULL) MEM_ERROR(); size = STR_SIZE; } /* Initialise string being read */ str_ptr = str; cur_size = 0; quoted = 0; while(1) { while(*cur_ptr == '"') { cur_ptr ++; quoted = !quoted; } if(*cur_ptr == '\0') { /* inside quoted string EOF, otherwise end of string */ if(quoted) return TOK_EOF; else break; } if(!quoted) { for(i = 0; token_table[i].token != -1; i++) if (strncmp(cur_ptr, token_table[i].string, token_table[i].size) == 0) break; if (token_table[i].token != -1) break; } if(*cur_ptr == '\\') { cur_ptr ++; if(*cur_ptr == '\0') return TOK_EOF; } if(cur_size + 2 > size) { char *tmp; size = (cur_size + 1 + STR_SIZE) & ~(STR_SIZE - 1); tmp = realloc(str, size); if(tmp == NULL) MEM_ERROR(); str_ptr = str_ptr - str + tmp; str = tmp; } *str_ptr ++ = *cur_ptr ++; cur_size ++; } *str_ptr = '\0'; *string = str; return TOK_STRING; } static int peek_token(char **string) { char *saved = cur_ptr; int token = get_token(string); cur_ptr = saved; return token; } /* * Expression parser */ static void free_parse_tree(struct expr *expr) { if(expr->type == ATOM_TYPE) { int i; for(i = 0; i < expr->atom.test->args; i++) free(expr->atom.argv[i]); free(expr->atom.argv); } else if (expr->type == UNARY_TYPE) free_parse_tree(expr->unary_op.expr); else { free_parse_tree(expr->expr_op.lhs); free_parse_tree(expr->expr_op.rhs); } free(expr); } static struct expr *create_expr(struct expr *lhs, int op, struct expr *rhs) { struct expr *expr; if (rhs == NULL) { free_parse_tree(lhs); return NULL; } expr = malloc(sizeof(*expr)); if (expr == NULL) MEM_ERROR(); expr->type = OP_TYPE; expr->expr_op.lhs = lhs; expr->expr_op.rhs = rhs; expr->expr_op.op = op; return expr; } static struct expr *create_unary_op(struct expr *lhs, int op) { struct expr *expr; if (lhs == NULL) return NULL; expr = malloc(sizeof(*expr)); if (expr == NULL) MEM_ERROR(); expr->type = UNARY_TYPE; expr->unary_op.expr = lhs; expr->unary_op.op = op; return expr; } static struct expr *parse_test(char *name) { char *string; int token; int i; struct test_entry *test; struct expr *expr; for (i = 0; test_table[i].args != -1; i++) if (strcmp(name, test_table[i].name) == 0) break; if (test_table[i].args == -1) { SYNTAX_ERROR("Non-existent test \"%s\"\n", name); return NULL; } test = &test_table[i]; expr = malloc(sizeof(*expr)); if (expr == NULL) MEM_ERROR(); expr->type = ATOM_TYPE; expr->atom.argv = malloc(test->args * sizeof(char *)); if (expr->atom.argv == NULL) MEM_ERROR(); expr->atom.test = test; expr->atom.data = NULL; /* * If the test has no arguments, allow it to be typed * without brackets */ if (test->args == 0) { token = peek_token(&string); if (token != TOK_OPEN_BRACKET) goto skip_args; } token = get_token(&string); if (token != TOK_OPEN_BRACKET) { SYNTAX_ERROR("Unexpected token \"%s\", expected \"(\"\n", TOK_TO_STR(token, string)); goto failed; } for (i = 0; i < test->args; i++) { token = get_token(&string); if (token != TOK_STRING) { SYNTAX_ERROR("Unexpected token \"%s\", expected " "argument\n", TOK_TO_STR(token, string)); goto failed; } expr->atom.argv[i] = strdup(string); if (i + 1 < test->args) { token = get_token(&string); if (token != TOK_COMMA) { SYNTAX_ERROR("Unexpected token \"%s\", " "expected \",\"\n", TOK_TO_STR(token, string)); goto failed; } } } if (test->parse_args) { int res = test->parse_args(test, &expr->atom); if (res == 0) goto failed; } token = get_token(&string); if (token != TOK_CLOSE_BRACKET) { SYNTAX_ERROR("Unexpected token \"%s\", expected \")\"\n", TOK_TO_STR(token, string)); goto failed; } skip_args: return expr; failed: free(expr->atom.argv); free(expr); return NULL; } static struct expr *get_atom() { char *string; int token = get_token(&string); switch(token) { case TOK_NOT: return create_unary_op(get_atom(), token); case TOK_OPEN_BRACKET: return parse_expr(1); case TOK_STRING: return parse_test(string); default: SYNTAX_ERROR("Unexpected token \"%s\", expected test " "operation, \"!\", or \"(\"\n", TOK_TO_STR(token, string)); return NULL; } } static struct expr *parse_expr(int subexp) { struct expr *expr = get_atom(); while (expr) { char *string; int op = get_token(&string); if (op == TOK_EOF) { if (subexp) { free_parse_tree(expr); SYNTAX_ERROR("Expected \"&&\", \"||\" or " "\")\", got EOF\n"); return NULL; } break; } if (op == TOK_CLOSE_BRACKET) { if (!subexp) { free_parse_tree(expr); SYNTAX_ERROR("Unexpected \")\", expected " "\"&&\", \"!!\" or EOF\n"); return NULL; } break; } if (op != TOK_AND && op != TOK_OR) { free_parse_tree(expr); SYNTAX_ERROR("Unexpected token \"%s\", expected " "\"&&\" or \"||\"\n", TOK_TO_STR(op, string)); return NULL; } expr = create_expr(expr, op, get_atom()); } return expr; } /* * Action parser */ int parse_action(char *s) { char *string, **argv = NULL; int i, token, args = 0; struct expr *expr; struct action_entry *action; void *data = NULL; struct action **spec_list; int spec_count; cur_ptr = source = s; token = get_token(&string); if (token != TOK_STRING) { SYNTAX_ERROR("Unexpected token \"%s\", expected name\n", TOK_TO_STR(token, string)); return 0; } for (i = 0; action_table[i].args != -1; i++) if (strcmp(string, action_table[i].name) == 0) break; if (action_table[i].args == -1) { SYNTAX_ERROR("Non-existent action \"%s\"\n", string); return 0; } action = &action_table[i]; token = get_token(&string); if (token == TOK_AT) goto skip_args; if (token != TOK_OPEN_BRACKET) { SYNTAX_ERROR("Unexpected token \"%s\", expected \"(\"\n", TOK_TO_STR(token, string)); goto failed; } /* * speculatively read all the arguments, and then see if the * number of arguments read is the number expected, this handles * actions with a variable number of arguments */ token = get_token(&string); if (token == TOK_CLOSE_BRACKET) goto skip_args; while (1) { if (token != TOK_STRING) { SYNTAX_ERROR("Unexpected token \"%s\", expected " "argument\n", TOK_TO_STR(token, string)); goto failed; } argv = realloc(argv, (args + 1) * sizeof(char *)); if (argv == NULL) MEM_ERROR(); argv[args ++] = strdup(string); token = get_token(&string); if (token == TOK_CLOSE_BRACKET) break; if (token != TOK_COMMA) { SYNTAX_ERROR("Unexpected token \"%s\", expected " "\",\" or \")\"\n", TOK_TO_STR(token, string)); goto failed; } token = get_token(&string); } skip_args: /* * expected number of arguments? */ if(action->args != -2 && args != action->args) { SYNTAX_ERROR("Unexpected number of arguments, expected %d, " "got %d\n", action->args, args); goto failed; } if (action->parse_args) { int res = action->parse_args(action, args, argv, &data); if (res == 0) goto failed; } if (token == TOK_CLOSE_BRACKET) token = get_token(&string); if (token != TOK_AT) { SYNTAX_ERROR("Unexpected token \"%s\", expected \"@\"\n", TOK_TO_STR(token, string)); goto failed; } expr = parse_expr(0); if (expr == NULL) goto failed; /* * choose action list and increment action counter */ switch(action->type) { case FRAGMENT_ACTION: spec_count = fragment_count ++; spec_list = &fragment_spec; break; case EXCLUDE_ACTION: spec_count = exclude_count ++; spec_list = &exclude_spec; break; case EMPTY_ACTION: spec_count = empty_count ++; spec_list = &empty_spec; break; case MOVE_ACTION: spec_count = move_count ++; spec_list = &move_spec; break; default: spec_count = other_count ++; spec_list = &other_spec; } *spec_list = realloc(*spec_list, (spec_count + 1) * sizeof(struct action)); if (*spec_list == NULL) MEM_ERROR(); (*spec_list)[spec_count].type = action->type; (*spec_list)[spec_count].action = action; (*spec_list)[spec_count].args = args; (*spec_list)[spec_count].argv = argv; (*spec_list)[spec_count].expr = expr; (*spec_list)[spec_count].data = data; return 1; failed: free(argv); return 0; } /* * Evaluate expressions */ static int eval_expr(struct expr *expr, struct action_data *action_data) { int match; switch (expr->type) { case ATOM_TYPE: match = expr->atom.test->fn(&expr->atom, action_data); break; case UNARY_TYPE: match = !eval_expr(expr->unary_op.expr, action_data); break; default: match = eval_expr(expr->expr_op.lhs, action_data); if ((expr->expr_op.op == TOK_AND && match) || (expr->expr_op.op == TOK_OR && !match)) match = eval_expr(expr->expr_op.rhs, action_data); break; } return match; } /* * Read action file, passing each line to parse_action() for * parsing. * * One action per line, of the form * action(arg1,arg2)@expr(arg1,arg2).... * * Actions can be split across multiple lines using "\". * * Blank lines and comment lines indicated by # are supported. */ int read_action_file(char *filename) { return read_file(filename, "action", parse_action); } /* * General action evaluation code */ int actions() { return other_count; } void eval_actions(struct dir_ent *dir_ent) { int i, match; struct action_data action_data; int file_type = dir_ent->inode->buf.st_mode & S_IFMT; action_data.name = dir_ent->name; action_data.pathname = pathname(dir_ent); action_data.subpath = subpathname(dir_ent); action_data.buf = &dir_ent->inode->buf; action_data.depth = dir_ent->our_dir->depth; for (i = 0; i < other_count; i++) { struct action *action = &other_spec[i]; if ((action->action->file_types & file_type) == 0) /* action does not operate on this file type */ continue; match = eval_expr(action->expr, &action_data); if (match) action->action->run_action(action, dir_ent); } } /* * Fragment specific action code */ void *eval_frag_actions(struct dir_ent *dir_ent) { int i, match; struct action_data action_data; action_data.name = dir_ent->name; action_data.pathname = pathname(dir_ent); action_data.subpath = subpathname(dir_ent); action_data.buf = &dir_ent->inode->buf; action_data.depth = dir_ent->our_dir->depth; for (i = 0; i < fragment_count; i++) { match = eval_expr(fragment_spec[i].expr, &action_data); if (match) return &fragment_spec[i].data; } return &def_fragment; } void *get_frag_action(void *fragment) { struct action *spec_list_end = &fragment_spec[fragment_count]; struct action *action; if (fragment == NULL) return &def_fragment; if (fragment_count == 0) return NULL; if (fragment == &def_fragment) action = &fragment_spec[0] - 1; else action = fragment - offsetof(struct action, data); if (++action == spec_list_end) return NULL; return &action->data; } /* * Exclude specific action code */ int exclude_actions() { return exclude_count; } int eval_exclude_actions(char *name, char *pathname, char *subpath, struct stat *buf, int depth) { int i, match = 0; struct action_data action_data; action_data.name = name; action_data.pathname = pathname; action_data.subpath = subpath; action_data.buf = buf; action_data.depth = depth; for (i = 0; i < exclude_count && !match; i++) match = eval_expr(exclude_spec[i].expr, &action_data); return match; } /* * Fragment specific action code */ static void frag_action(struct action *action, struct dir_ent *dir_ent) { struct inode_info *inode = dir_ent->inode; inode->no_fragments = 0; } static void no_frag_action(struct action *action, struct dir_ent *dir_ent) { struct inode_info *inode = dir_ent->inode; inode->no_fragments = 1; } static void always_frag_action(struct action *action, struct dir_ent *dir_ent) { struct inode_info *inode = dir_ent->inode; inode->always_use_fragments = 1; } static void no_always_frag_action(struct action *action, struct dir_ent *dir_ent) { struct inode_info *inode = dir_ent->inode; inode->always_use_fragments = 0; } /* * Compression specific action code */ static void comp_action(struct action *action, struct dir_ent *dir_ent) { struct inode_info *inode = dir_ent->inode; inode->noD = inode->noF = 0; } static void uncomp_action(struct action *action, struct dir_ent *dir_ent) { struct inode_info *inode = dir_ent->inode; inode->noD = inode->noF = 1; } /* * Uid/gid specific action code */ static long long parse_uid(char *arg) { char *b; long long uid = strtoll(arg, &b, 10); if (*b == '\0') { if (uid < 0 || uid >= (1LL << 32)) { SYNTAX_ERROR("action: uid out of range\n"); return -1; } } else { struct passwd *passwd = getpwnam(arg); if (passwd) uid = passwd->pw_uid; else { SYNTAX_ERROR("action: invalid uid or unknown user\n"); return -1; } } return uid; } static long long parse_gid(char *arg) { char *b; long long gid = strtoll(arg, &b, 10); if (*b == '\0') { if (gid < 0 || gid >= (1LL << 32)) { SYNTAX_ERROR("action: gid out of range\n"); return -1; } } else { struct group *group = getgrnam(arg); if (group) gid = group->gr_gid; else { SYNTAX_ERROR("action: invalid gid or unknown user\n"); return -1; } } return gid; } static int parse_uid_args(struct action_entry *action, int args, char **argv, void **data) { long long uid; struct uid_info *uid_info; uid = parse_uid(argv[0]); if (uid == -1) return 0; uid_info = malloc(sizeof(struct uid_info)); if (uid_info == NULL) MEM_ERROR(); uid_info->uid = uid; *data = uid_info; return 1; } static int parse_gid_args(struct action_entry *action, int args, char **argv, void **data) { long long gid; struct gid_info *gid_info; gid = parse_gid(argv[0]); if (gid == -1) return 0; gid_info = malloc(sizeof(struct gid_info)); if (gid_info == NULL) MEM_ERROR(); gid_info->gid = gid; *data = gid_info; return 1; } static int parse_guid_args(struct action_entry *action, int args, char **argv, void **data) { long long uid, gid; struct guid_info *guid_info; uid = parse_uid(argv[0]); if (uid == -1) return 0; gid = parse_gid(argv[1]); if (gid == -1) return 0; guid_info = malloc(sizeof(struct guid_info)); if (guid_info == NULL) MEM_ERROR(); guid_info->uid = uid; guid_info->gid = gid; *data = guid_info; return 1; } static void uid_action(struct action *action, struct dir_ent *dir_ent) { struct inode_info *inode = dir_ent->inode; struct uid_info *uid_info = action->data; inode->buf.st_uid = uid_info->uid; } static void gid_action(struct action *action, struct dir_ent *dir_ent) { struct inode_info *inode = dir_ent->inode; struct gid_info *gid_info = action->data; inode->buf.st_gid = gid_info->gid; } static void guid_action(struct action *action, struct dir_ent *dir_ent) { struct inode_info *inode = dir_ent->inode; struct guid_info *guid_info = action->data; inode->buf.st_uid = guid_info->uid; inode->buf.st_gid = guid_info->gid; } /* * Mode specific action code */ static int parse_octal_mode_args(unsigned int mode, int bytes, int args, char **argv, void **data) { struct mode_data *mode_data; /* check there's no trailing junk */ if (argv[0][bytes] != '\0') { SYNTAX_ERROR("Unexpected trailing bytes after octal " "mode number\n"); return 0; } /* check there's only one argument */ if (args > 1) { SYNTAX_ERROR("Octal mode number is first argument, " "expected one argument, got %d\n", args); return 0; } /* check mode is within range */ if (mode > 07777) { SYNTAX_ERROR("Octal mode %o is out of range\n", mode); return 0; } mode_data = malloc(sizeof(struct mode_data)); if (mode_data == NULL) MEM_ERROR(); mode_data->operation = ACTION_MODE_OCT; mode_data->mode = mode; mode_data->next = NULL; *data = mode_data; return 1; } /* * Parse symbolic mode of format [ugoa]+[+-=]PERMS * PERMS = [rwxXst]+ or [ugo] */ static struct mode_data *parse_sym_mode_arg(char *arg) { struct mode_data *mode_data = malloc(sizeof(*mode_data)); int mode = 0; int mask = 0; int op; char X = 0; if (mode_data == NULL) MEM_ERROR(); if (arg[0] != 'u' && arg[0] != 'g' && arg[0] != 'o' && arg[0] != 'a') { /* no ownership specifiers, default to a */ mask = 0777; goto parse_operation; } /* parse ownership specifiers */ while(1) { switch(*arg) { case 'u': mask |= 04700; break; case 'g': mask |= 02070; break; case 'o': mask |= 01007; break; case 'a': mask = 07777; break; default: goto parse_operation; } arg ++; } parse_operation: switch(*arg) { case '+': op = ACTION_MODE_ADD; break; case '-': op = ACTION_MODE_REM; break; case '=': op = ACTION_MODE_SET; break; default: SYNTAX_ERROR("Action mode: Expected one of '+', '-' or '=', " "got '%c'\n", *arg); goto failed; } arg ++; /* Parse PERMS */ if (*arg == 'u' || *arg == 'g' || *arg == 'o') { /* PERMS = [ugo] */ mode = - *arg; if (*++arg != '\0') { SYNTAX_ERROR("Action mode: permission 'u', 'g' or 'o' " "has trailing characters\n"); goto failed; } } else { /* PERMS = [rwxXst]+ */ while(*arg != '\0') { switch(*arg) { case 'r': mode |= 0444; break; case 'w': mode |= 0222; break; case 'x': mode |= 0111; break; case 's': mode |= 06000; break; case 't': mode |= 01000; break; case 'X': X = 1; break; default: SYNTAX_ERROR("Action mode: unrecognised " "permission '%c'\n", *arg); goto failed; } arg ++; } mode &= mask; } mode_data->operation = op; mode_data->mode = mode; mode_data->mask = mask; mode_data->X = X; mode_data->next = NULL; return mode_data; failed: free(mode_data); return NULL; } static int parse_sym_mode_args(struct action_entry *action, int args, char **argv, void **data) { int i; struct mode_data *head = NULL, *cur = NULL; for (i = 0; i < args; i++) { struct mode_data *entry = parse_sym_mode_arg(argv[i]); if (entry == NULL) return 0; if (cur) { cur->next = entry; cur = entry; } else head = cur = entry; } *data = head; return 1; } static int parse_mode_args(struct action_entry *action, int args, char **argv, void **data) { int n, bytes; unsigned int mode; if (args == 0) { SYNTAX_ERROR("Mode action expects one or more arguments\n"); return 0; } /* octal mode number? */ n = sscanf(argv[0], "%o%n", &mode, &bytes); if(n >= 1) return parse_octal_mode_args(mode, bytes, args, argv, data); else return parse_sym_mode_args(action, args, argv, data); } static void mode_action(struct action *action, struct dir_ent *dir_ent) { struct stat *buf = &dir_ent->inode->buf; struct mode_data *mode_data = action->data; int mode = 0; for (;mode_data; mode_data = mode_data->next) { if (mode_data->mode < 0) { /* 'u', 'g' or 'o' */ switch(-mode_data->mode) { case 'u': mode = (buf->st_mode >> 6) & 07; break; case 'g': mode = (buf->st_mode >> 3) & 07; break; case 'o': mode = buf->st_mode & 07; break; } mode = ((mode << 6) | (mode << 3) | mode) & mode_data->mask; } else if (mode_data->X && ((buf->st_mode & S_IFMT) == S_IFDIR || (buf->st_mode & 0111))) /* X permission, only takes effect if inode is a * directory or x is set for some owner */ mode = mode_data->mode | (0111 & mode_data->mask); else mode = mode_data->mode; switch(mode_data->operation) { case ACTION_MODE_OCT: buf->st_mode = (buf->st_mode & ~S_IFMT) | mode; break; case ACTION_MODE_SET: buf->st_mode = (buf->st_mode & ~mode_data->mask) | mode; break; case ACTION_MODE_ADD: buf->st_mode |= mode; break; case ACTION_MODE_REM: buf->st_mode &= ~mode; } } } /* * Empty specific action code */ int empty_actions() { return empty_count; } static int parse_empty_args(struct action_entry *action, int args, char **argv, void **data) { struct empty_data *empty_data; int val; if (args >= 2) { SYNTAX_ERROR("Empty action expects zero or one argument\n"); return 0; } if (args == 0 || strcmp(argv[0], "all") == 0) val = EMPTY_ALL; else if (strcmp(argv[0], "source") == 0) val = EMPTY_SOURCE; else if (strcmp(argv[0], "excluded") == 0) val = EMPTY_EXCLUDED; else { SYNTAX_ERROR("Empty action expects zero arguments, or one" "argument containing \"all\", \"source\", or \"excluded\"" "\n"); return 0; } empty_data = malloc(sizeof(*empty_data)); if (empty_data == NULL) MEM_ERROR(); empty_data->val = val; *data = empty_data; return 1; } int eval_empty_actions(struct dir_ent *dir_ent) { int i, match = 0; struct action_data action_data; struct empty_data *data; struct dir_info *dir = dir_ent->dir; /* * Empty action only works on empty directories */ if (dir->count != 0) return 0; action_data.name = dir_ent->name; action_data.pathname = pathname(dir_ent); action_data.subpath = subpathname(dir_ent); action_data.buf = &dir_ent->inode->buf; action_data.depth = dir_ent->our_dir->depth; for (i = 0; i < empty_count && !match; i++) { data = empty_spec[i].data; /* * determine the cause of the empty directory and evaluate * the empty action specified. Three empty actions: * - EMPTY_SOURCE: empty action triggers only if the directory * was originally empty, i.e directories that are empty * only due to excluding are ignored. * - EMPTY_EXCLUDED: empty action triggers only if the directory * is empty because of excluding, i.e. directories that * were originally empty are ignored. * - EMPTY_ALL (the default): empty action triggers if the * directory is empty, irrespective of the reason, i.e. * the directory could have been originally empty or could * be empty due to excluding. */ if ((data->val == EMPTY_EXCLUDED && !dir->excluded) || (data->val == EMPTY_SOURCE && dir->excluded)) continue; match = eval_expr(empty_spec[i].expr, &action_data); } return match; } /* * Move specific action code */ static struct move_ent *move_list = NULL; int move_actions() { return move_count; } static char *move_pathname(struct move_ent *move) { struct dir_info *dest; char *name, *pathname; int res; dest = (move->ops & ACTION_MOVE_MOVE) ? move->dest : move->dir_ent->our_dir; name = (move->ops & ACTION_MOVE_RENAME) ? move->name : move->dir_ent->name; if(dest->subpath[0] != '\0') res = asprintf(&pathname, "%s/%s", dest->subpath, name); else res = asprintf(&pathname, "/%s", name); if(res == -1) BAD_ERROR("asprintf failed in move_pathname\n"); return pathname; } static char *get_comp(char **pathname) { char *path = *pathname, *start; while(*path == '/') path ++; if(*path == '\0') return NULL; start = path; while(*path != '/' && *path != '\0') path ++; *pathname = path; return strndup(start, path - start); } static struct dir_ent *lookup_comp(char *comp, struct dir_info *dest) { struct dir_ent *dir_ent; for(dir_ent = dest->list; dir_ent; dir_ent = dir_ent->next) if(strcmp(comp, dir_ent->name) == 0) break; return dir_ent; } void eval_move(struct action_data *action_data, struct move_ent *move, struct dir_info *root, struct dir_ent *dir_ent, char *pathname) { struct dir_info *dest, *source = dir_ent->our_dir; struct dir_ent *comp_ent; char *comp, *path = pathname; /* * Walk pathname to get the destination directory * * Like the mv command, if the last component exists and it * is a directory, then move the file into that directory, * otherwise, move the file into parent directory of the last * component and rename to the last component. */ if (pathname[0] == '/') /* absolute pathname, walk from root directory */ dest = root; else /* relative pathname, walk from current directory */ dest = source; for(comp = get_comp(&pathname); comp; free(comp), comp = get_comp(&pathname)) { if (strcmp(comp, ".") == 0) continue; if (strcmp(comp, "..") == 0) { /* if we're in the root directory then ignore */ if(dest->depth > 1) dest = dest->dir_ent->our_dir; continue; } /* * Look up comp in current directory, if it exists and it is a * directory continue walking the pathname, otherwise exit, * we've walked as far as we can go, normally this is because * we've arrived at the leaf component which we are going to * rename source to */ comp_ent = lookup_comp(comp, dest); if (comp_ent == NULL || (comp_ent->inode->buf.st_mode & S_IFMT) != S_IFDIR) break; dest = comp_ent->dir; } if(comp) { /* Leaf component? If so we're renaming to this */ char *remainder = get_comp(&pathname); free(remainder); if(remainder) { /* * trying to move source to a subdirectory of * comp, but comp either doesn't exist, or it isn't * a directory, which is impossible */ if (comp_ent == NULL) ERROR("Move action: cannot move %s to %s, no " "such directory %s\n", action_data->subpath, path, comp); else ERROR("Move action: cannot move %s to %s, %s " "is not a directory\n", action_data->subpath, path, comp); free(comp); return; } /* * Multiple move actions triggering on one file can be merged * if one is a RENAME and the other is a MOVE. Multiple RENAMEs * can only merge if they're doing the same thing */ if(move->ops & ACTION_MOVE_RENAME) { if(strcmp(comp, move->name) != 0) { char *conf_path = move_pathname(move); ERROR("Move action: Cannot move %s to %s, " "conflicting move, already moving " "to %s via another move action!\n", action_data->subpath, path, conf_path); free(conf_path); free(comp); return; } free(comp); } else { move->name = comp; move->ops |= ACTION_MOVE_RENAME; } } if(dest != source) { /* * Multiple move actions triggering on one file can be merged * if one is a RENAME and the other is a MOVE. Multiple MOVEs * can only merge if they're doing the same thing */ if(move->ops & ACTION_MOVE_MOVE) { if(dest != move->dest) { char *conf_path = move_pathname(move); ERROR("Move action: Cannot move %s to %s, " "conflicting move, already moving " "to %s via another move action!\n", action_data->subpath, path, conf_path); free(conf_path); return; } } else { move->dest = dest; move->ops |= ACTION_MOVE_MOVE; } } } static int subdirectory(struct dir_info *source, struct dir_info *dest) { if(source == NULL) return 0; return strlen(source->subpath) <= strlen(dest->subpath) && (dest->subpath[strlen(source->subpath)] == '/' || dest->subpath[strlen(source->subpath)] == '\0') && strncmp(source->subpath, dest->subpath, strlen(source->subpath)) == 0; } void eval_move_actions(struct dir_info *root, struct dir_ent *dir_ent) { int i; struct action_data action_data; struct move_ent *move = NULL; action_data.name = dir_ent->name; action_data.pathname = pathname(dir_ent); action_data.subpath = subpathname(dir_ent); action_data.buf = &dir_ent->inode->buf; action_data.depth = dir_ent->our_dir->depth; /* * Evaluate each move action against the current file. For any * move actions that match don't actually perform the move now, but, * store it, and execute all the stored move actions together once the * directory scan is complete. This is done to ensure each separate * move action does not nondeterministically interfere with other move * actions. Each move action is considered to act independently, and * each move action sees the directory tree in the same state. */ for (i = 0; i < move_count; i++) { struct action *action = &move_spec[i]; int match = eval_expr(action->expr, &action_data); if(match) { if(move == NULL) { move = malloc(sizeof(*move)); if(move == NULL) MEM_ERROR(); move->ops = 0; move->dir_ent = dir_ent; } eval_move(&action_data, move, root, dir_ent, action->argv[0]); } } if(move) { struct dir_ent *comp_ent; struct dir_info *dest; char *name; /* * Move contains the result of all triggered move actions. * Check the destination doesn't already exist */ if(move->ops == 0) { free(move); return; } dest = (move->ops & ACTION_MOVE_MOVE) ? move->dest : dir_ent->our_dir; name = (move->ops & ACTION_MOVE_RENAME) ? move->name : dir_ent->name; comp_ent = lookup_comp(name, dest); if(comp_ent) { char *conf_path = move_pathname(move); ERROR("Move action: Cannot move %s to %s, " "destination already exists\n", action_data.subpath, conf_path); free(conf_path); free(move); return; } /* * If we're moving a directory, check we're not moving it to a * subdirectory of itself */ if(subdirectory(dir_ent->dir, dest)) { char *conf_path = move_pathname(move); ERROR("Move action: Cannot move %s to %s, this is a " "subdirectory of itself\n", action_data.subpath, conf_path); free(conf_path); free(move); return; } move->next = move_list; move_list = move; } } static void move_dir(struct dir_ent *dir_ent) { struct dir_info *dir = dir_ent->dir; struct dir_ent *comp_ent; /* update our directory's subpath name */ free(dir->subpath); dir->subpath = strdup(subpathname(dir_ent)); /* recursively update the subpaths of any sub-directories */ for(comp_ent = dir->list; comp_ent; comp_ent = comp_ent->next) if(comp_ent->dir) move_dir(comp_ent); } static void move_file(struct move_ent *move_ent) { struct dir_ent *dir_ent = move_ent->dir_ent; if(move_ent->ops & ACTION_MOVE_MOVE) { struct dir_ent *comp_ent, *prev = NULL; struct dir_info *source = dir_ent->our_dir, *dest = move_ent->dest; char *filename = pathname(dir_ent); /* * If we're moving a directory, check we're not moving it to a * subdirectory of itself */ if(subdirectory(dir_ent->dir, dest)) { char *conf_path = move_pathname(move_ent); ERROR("Move action: Cannot move %s to %s, this is a " "subdirectory of itself\n", subpathname(dir_ent), conf_path); free(conf_path); return; } /* Remove the file from source directory */ for(comp_ent = source->list; comp_ent != dir_ent; prev = comp_ent, comp_ent = comp_ent->next); if(prev) prev->next = comp_ent->next; else source->list = comp_ent->next; source->count --; if((comp_ent->inode->buf.st_mode & S_IFMT) == S_IFDIR) source->directory_count --; /* Add the file to dest directory */ comp_ent->next = dest->list; dest->list = comp_ent; comp_ent->our_dir = dest; dest->count ++; if((comp_ent->inode->buf.st_mode & S_IFMT) == S_IFDIR) dest->directory_count ++; /* * We've moved the file, and so we can't now use the * parent directory's pathname to calculate the pathname */ if(dir_ent->nonstandard_pathname == NULL) { dir_ent->nonstandard_pathname = strdup(filename); if(dir_ent->source_name) { free(dir_ent->source_name); dir_ent->source_name = NULL; } } } if(move_ent->ops & ACTION_MOVE_RENAME) { /* * If we're using name in conjunction with the parent * directory's pathname to calculate the pathname, we need * to use source_name to override. Otherwise it's already being * over-ridden */ if(dir_ent->nonstandard_pathname == NULL && dir_ent->source_name == NULL) dir_ent->source_name = dir_ent->name; else free(dir_ent->name); dir_ent->name = move_ent->name; } if(dir_ent->dir) /* * dir_ent is a directory, and we have to recursively fix-up * its subpath, and the subpaths of all of its sub-directories */ move_dir(dir_ent); } void do_move_actions() { while(move_list) { struct move_ent *temp = move_list; struct dir_info *dest = (move_list->ops & ACTION_MOVE_MOVE) ? move_list->dest : move_list->dir_ent->our_dir; char *name = (move_list->ops & ACTION_MOVE_RENAME) ? move_list->name : move_list->dir_ent->name; struct dir_ent *comp_ent = lookup_comp(name, dest); if(comp_ent) { char *conf_path = move_pathname(move_list); ERROR("Move action: Cannot move %s to %s, " "destination already exists\n", subpathname(move_list->dir_ent), conf_path); free(conf_path); } else move_file(move_list); move_list = move_list->next; free(temp); } } /* * General test evaluation code */ /* * A number can be of the form [range]number[size] * [range] is either: * '<' or '-', match on less than number * '>' or '+', match on greater than number * '' (nothing), match on exactly number * [size] is either: * '' (nothing), number * 'k' or 'K', number * 2^10 * 'm' or 'M', number * 2^20 * 'g' or 'G', number * 2^30 */ static int parse_number(char *start, long long *size, int *range, char **error) { char *end; long long number; if (*start == '>' || *start == '+') { *range = NUM_GREATER; start ++; } else if (*start == '<' || *start == '-') { *range = NUM_LESS; start ++; } else *range = NUM_EQ; errno = 0; /* To enable failure after call to be determined */ number = strtoll(start, &end, 10); if((errno == ERANGE && (number == LLONG_MAX || number == LLONG_MIN)) || (errno != 0 && number == 0)) { /* long long underflow or overflow in conversion, or other * conversion error. * Note: we don't check for LLONG_MIN and LLONG_MAX only * because strtoll can validly return that if the * user used these values */ *error = "Long long underflow, overflow or other conversion " "error"; return 0; } if (end == start) { /* Couldn't read any number */ *error = "Number expected"; return 0; } switch (end[0]) { case 'g': case 'G': number *= 1024; case 'm': case 'M': number *= 1024; case 'k': case 'K': number *= 1024; if (end[1] != '\0') { *error = "Trailing junk after size specifier"; return 0; } break; case '\0': break; default: *error = "Trailing junk after number"; return 0; } *size = number; return 1; } static int parse_number_arg(struct test_entry *test, struct atom *atom) { struct test_number_arg *number; long long size; int range; char *error; int res = parse_number(atom->argv[0], &size, &range, &error); if (res == 0) { TEST_SYNTAX_ERROR(test, 0, "%s\n", error); return 0; } number = malloc(sizeof(*number)); if (number == NULL) MEM_ERROR(); number->range = range; number->size = size; atom->data = number; return 1; } static int parse_range_args(struct test_entry *test, struct atom *atom) { struct test_range_args *range; long long start, end; int type; int res; char *error; res = parse_number(atom->argv[0], &start, &type, &error); if (res == 0) { TEST_SYNTAX_ERROR(test, 0, "%s\n", error); return 0; } if (type != NUM_EQ) { TEST_SYNTAX_ERROR(test, 0, "Range specifier (<, >, -, +) not " "expected\n"); return 0; } res = parse_number(atom->argv[1], &end, &type, &error); if (res == 0) { TEST_SYNTAX_ERROR(test, 1, "%s\n", error); return 0; } if (type != NUM_EQ) { TEST_SYNTAX_ERROR(test, 1, "Range specifier (<, >, -, +) not " "expected\n"); return 0; } range = malloc(sizeof(*range)); if (range == NULL) MEM_ERROR(); range->start = start; range->end = end; atom->data = range; return 1; } /* * Generic test code macro */ #define TEST_FN(NAME, MATCH, CODE) \ static int NAME##_fn(struct atom *atom, struct action_data *action_data) \ { \ /* test operates on MATCH file types only */ \ if (!(action_data->buf->st_mode & MATCH)) \ return 0; \ \ CODE \ } /* * Generic test code macro testing VAR for size (eq, less than, greater than) */ #define TEST_VAR_FN(NAME, MATCH, VAR) TEST_FN(NAME, MATCH, \ { \ int match = 0; \ struct test_number_arg *number = atom->data; \ \ switch (number->range) { \ case NUM_EQ: \ match = VAR == number->size; \ break; \ case NUM_LESS: \ match = VAR < number->size; \ break; \ case NUM_GREATER: \ match = VAR > number->size; \ break; \ } \ \ return match; \ }) /* * Generic test code macro testing VAR for range [x, y] (value between x and y * inclusive). */ #define TEST_VAR_RANGE_FN(NAME, MATCH, VAR) TEST_FN(NAME##_range, MATCH, \ { \ struct test_range_args *range = atom->data; \ \ return range->start <= VAR && VAR <= range->end; \ }) /* * Name, Pathname and Subpathname test specific code */ /* * Add a leading "/" if subpathname and pathname lacks it */ static int check_pathname(struct test_entry *test, struct atom *atom) { int res; char *name; if(atom->argv[0][0] != '/') { res = asprintf(&name, "/%s", atom->argv[0]); if(res == -1) BAD_ERROR("asprintf failed in check_pathname\n"); free(atom->argv[0]); atom->argv[0] = name; } return 1; } TEST_FN(name, ACTION_ALL_LNK, \ return fnmatch(atom->argv[0], action_data->name, FNM_PATHNAME|FNM_PERIOD|FNM_EXTMATCH) == 0;) TEST_FN(pathname, ACTION_ALL_LNK, \ return fnmatch(atom->argv[0], action_data->subpath, FNM_PATHNAME|FNM_PERIOD|FNM_EXTMATCH) == 0;) static int count_components(char *path) { int count; for (count = 0; *path != '\0'; count ++) { while (*path == '/') path ++; while (*path != '\0' && *path != '/') path ++; } return count; } static char *get_start(char *s, int n) { int count; char *path = s; for (count = 0; *path != '\0' && count < n; count ++) { while (*path == '/') path ++; while (*path != '\0' && *path != '/') path ++; } if (count == n) *path = '\0'; return s; } static int subpathname_fn(struct atom *atom, struct action_data *action_data) { return fnmatch(atom->argv[0], get_start(strdupa(action_data->subpath), count_components(atom->argv[0])), FNM_PATHNAME|FNM_PERIOD|FNM_EXTMATCH) == 0; } TEST_VAR_FN(filesize, ACTION_REG, action_data->buf->st_size) TEST_VAR_FN(dirsize, ACTION_DIR, action_data->buf->st_size) TEST_VAR_FN(size, ACTION_ALL_LNK, action_data->buf->st_size) TEST_VAR_FN(inode, ACTION_ALL_LNK, action_data->buf->st_ino) TEST_VAR_FN(nlink, ACTION_ALL_LNK, action_data->buf->st_nlink) TEST_VAR_FN(fileblocks, ACTION_REG, action_data->buf->st_blocks) TEST_VAR_FN(dirblocks, ACTION_DIR, action_data->buf->st_blocks) TEST_VAR_FN(blocks, ACTION_ALL_LNK, action_data->buf->st_blocks) TEST_VAR_FN(gid, ACTION_ALL_LNK, action_data->buf->st_gid) TEST_VAR_FN(uid, ACTION_ALL_LNK, action_data->buf->st_uid) TEST_VAR_FN(depth, ACTION_ALL_LNK, action_data->depth) TEST_VAR_RANGE_FN(filesize, ACTION_REG, action_data->buf->st_size) TEST_VAR_RANGE_FN(dirsize, ACTION_DIR, action_data->buf->st_size) TEST_VAR_RANGE_FN(size, ACTION_ALL_LNK, action_data->buf->st_size) TEST_VAR_RANGE_FN(inode, ACTION_ALL_LNK, action_data->buf->st_ino) TEST_VAR_RANGE_FN(nlink, ACTION_ALL_LNK, action_data->buf->st_nlink) TEST_VAR_RANGE_FN(fileblocks, ACTION_REG, action_data->buf->st_blocks) TEST_VAR_RANGE_FN(dirblocks, ACTION_DIR, action_data->buf->st_blocks) TEST_VAR_RANGE_FN(blocks, ACTION_ALL_LNK, action_data->buf->st_blocks) TEST_VAR_RANGE_FN(gid, ACTION_ALL_LNK, action_data->buf->st_gid) TEST_VAR_RANGE_FN(uid, ACTION_ALL_LNK, action_data->buf->st_uid) TEST_VAR_RANGE_FN(depth, ACTION_ALL_LNK, action_data->depth) /* * Type test specific code */ struct type_entry type_table[] = { { S_IFSOCK, 's' }, { S_IFLNK, 'l' }, { S_IFREG, 'f' }, { S_IFBLK, 'b' }, { S_IFDIR, 'd' }, { S_IFCHR, 'c' }, { S_IFIFO, 'p' }, { 0, 0 }, }; static int parse_type_arg(struct test_entry *test, struct atom *atom) { int i; if (strlen(atom->argv[0]) != 1) goto failed; for(i = 0; type_table[i].type != 0; i++) if (type_table[i].type == atom->argv[0][0]) break; atom->data = &type_table[i]; if(type_table[i].type != 0) return 1; failed: TEST_SYNTAX_ERROR(test, 0, "Unexpected file type, expected 'f', 'd', " "'c', 'b', 'l', 's' or 'p'\n"); return 0; } static int type_fn(struct atom *atom, struct action_data *action_data) { struct type_entry *type = atom->data; return (action_data->buf->st_mode & S_IFMT) == type->value; } /* * True test specific code */ static int true_fn(struct atom *atom, struct action_data *action_data) { return 1; } /* * False test specific code */ static int false_fn(struct atom *atom, struct action_data *action_data) { return 0; } /* * File test specific code */ static int parse_file_arg(struct test_entry *test, struct atom *atom) { int res; regex_t *preg = malloc(sizeof(regex_t)); if (preg == NULL) MEM_ERROR(); res = regcomp(preg, atom->argv[0], REG_EXTENDED); if (res) { char str[1024]; /* overflow safe */ regerror(res, preg, str, 1024); free(preg); TEST_SYNTAX_ERROR(test, 0, "invalid regex \"%s\" because " "\"%s\"\n", atom->argv[0], str); return 0; } atom->data = preg; return 1; } static int file_fn(struct atom *atom, struct action_data *action_data) { int child, res, size = 0, status; int pipefd[2]; char *buffer = NULL; regex_t *preg = atom->data; res = pipe(pipefd); if (res == -1) BAD_ERROR("file_fn pipe failed\n"); child = fork(); if (child == -1) BAD_ERROR("file_fn fork_failed\n"); if (child == 0) { /* * Child process * Connect stdout to pipefd[1] and execute file command */ close(STDOUT_FILENO); res = dup(pipefd[1]); if (res == -1) exit(EXIT_FAILURE); execlp("file", "file", "-b", action_data->pathname, (char *) NULL); exit(EXIT_FAILURE); } /* * Parent process. Read stdout from file command */ close(pipefd[1]); do { buffer = realloc(buffer, size + 512); if (buffer == NULL) MEM_ERROR(); res = read_bytes(pipefd[0], buffer + size, 512); if (res == -1) BAD_ERROR("file_fn pipe read error\n"); size += 512; } while (res == 512); size = size + res - 512; buffer[size] = '\0'; res = waitpid(child, &status, 0); if (res == -1) BAD_ERROR("file_fn waitpid failed\n"); if (!WIFEXITED(status) || WEXITSTATUS(status) != 0) BAD_ERROR("file_fn file returned error\n"); close(pipefd[0]); res = regexec(preg, buffer, (size_t) 0, NULL, 0); free(buffer); return res == 0; } /* * Exec test specific code */ static int exec_fn(struct atom *atom, struct action_data *action_data) { int child, i, res, status; child = fork(); if (child == -1) BAD_ERROR("exec_fn fork_failed\n"); if (child == 0) { /* * Child process * redirect stdin, stdout & stderr to /dev/null and * execute atom->argv[0] */ int fd = open("/dev/null", O_RDWR); if(fd == -1) exit(EXIT_FAILURE); close(STDIN_FILENO); close(STDOUT_FILENO); close(STDERR_FILENO); for(i = 0; i < 3; i++) { res = dup(fd); if (res == -1) exit(EXIT_FAILURE); } close(fd); /* * Create environment variables * NAME: name of file * PATHNAME: pathname of file relative to squashfs root * SOURCE_PATHNAME: the pathname of the file in the source * directory */ res = setenv("NAME", action_data->name, 1); if(res == -1) exit(EXIT_FAILURE); res = setenv("PATHNAME", action_data->subpath, 1); if(res == -1) exit(EXIT_FAILURE); res = setenv("SOURCE_PATHNAME", action_data->pathname, 1); if(res == -1) exit(EXIT_FAILURE); execl("/bin/sh", "sh", "-c", atom->argv[0], (char *) NULL); exit(EXIT_FAILURE); } /* * Parent process. */ res = waitpid(child, &status, 0); if (res == -1) BAD_ERROR("exec_fn waitpid failed\n"); return WIFEXITED(status) ? WEXITSTATUS(status) == 0 : 0; } #ifdef SQUASHFS_TRACE static void dump_parse_tree(struct expr *expr) { if(expr->type == ATOM_TYPE) { int i; printf("%s(", expr->atom.test->name); for(i = 0; i < expr->atom.test->args; i++) { printf("%s", expr->atom.argv[i]); if (i + 1 < expr->atom.test->args) printf(","); } printf(")"); } else if (expr->type == UNARY_TYPE) { printf("%s", token_table[expr->unary_op.op].string); dump_parse_tree(expr->unary_op.expr); } else { printf("("); dump_parse_tree(expr->expr_op.lhs); printf("%s", token_table[expr->expr_op.op].string); dump_parse_tree(expr->expr_op.rhs); printf(")"); } } void dump_action_list(struct action *spec_list, int spec_count) { int i; for (i = 0; i < spec_count; i++) { printf("%s", spec_list[i].action->name); if (spec_list[i].action->args) { int n; printf("("); for (n = 0; n < spec_list[i].action->args; n++) { printf("%s", spec_list[i].argv[n]); if (n + 1 < spec_list[i].action->args) printf(","); } printf(")"); } printf("="); dump_parse_tree(spec_list[i].expr); printf("\n"); } } void dump_actions() { dump_action_list(exclude_spec, exclude_count); dump_action_list(fragment_spec, fragment_count); dump_action_list(other_spec, other_count); dump_action_list(move_spec, move_count); dump_action_list(empty_spec, empty_count); } #else void dump_actions() { } #endif static struct test_entry test_table[] = { { "name", 1, name_fn}, { "pathname", 1, pathname_fn, check_pathname}, { "subpathname", 1, subpathname_fn, check_pathname}, { "filesize", 1, filesize_fn, parse_number_arg}, { "dirsize", 1, dirsize_fn, parse_number_arg}, { "size", 1, size_fn, parse_number_arg}, { "inode", 1, inode_fn, parse_number_arg}, { "nlink", 1, nlink_fn, parse_number_arg}, { "fileblocks", 1, fileblocks_fn, parse_number_arg}, { "dirblocks", 1, dirblocks_fn, parse_number_arg}, { "blocks", 1, blocks_fn, parse_number_arg}, { "gid", 1, gid_fn, parse_number_arg}, { "uid", 1, uid_fn, parse_number_arg}, { "depth", 1, depth_fn, parse_number_arg}, { "filesize_range", 2, filesize_range_fn, parse_range_args}, { "dirsize_range", 2, dirsize_range_fn, parse_range_args}, { "size_range", 2, size_range_fn, parse_range_args}, { "inode_range", 2, inode_range_fn, parse_range_args}, { "nlink_range", 2, nlink_range_fn, parse_range_args}, { "fileblocks_range", 2, fileblocks_range_fn, parse_range_args}, { "dirblocks_range", 2, dirblocks_range_fn, parse_range_args}, { "blocks_range", 2, blocks_range_fn, parse_range_args}, { "gid_range", 2, gid_range_fn, parse_range_args}, { "uid_range", 2, uid_range_fn, parse_range_args}, { "depth_range", 2, depth_range_fn, parse_range_args}, { "type", 1, type_fn, parse_type_arg}, { "true", 0, true_fn, NULL}, { "false", 0, false_fn, NULL}, { "file", 1, file_fn, parse_file_arg}, { "exec", 1, exec_fn, NULL}, { "", -1 } }; static struct action_entry action_table[] = { { "fragment", FRAGMENT_ACTION, 1, ACTION_REG, NULL, NULL}, { "exclude", EXCLUDE_ACTION, 0, ACTION_ALL_LNK, NULL, NULL}, { "fragments", FRAGMENTS_ACTION, 0, ACTION_REG, NULL, frag_action}, { "no-fragments", NO_FRAGMENTS_ACTION, 0, ACTION_REG, NULL, no_frag_action}, { "always-use-fragments", ALWAYS_FRAGS_ACTION, 0, ACTION_REG, NULL, always_frag_action}, { "dont-always-use-fragments", NO_ALWAYS_FRAGS_ACTION, 0, ACTION_REG, NULL, no_always_frag_action}, { "compressed", COMPRESSED_ACTION, 0, ACTION_REG, NULL, comp_action}, { "uncompressed", UNCOMPRESSED_ACTION, 0, ACTION_REG, NULL, uncomp_action}, { "uid", UID_ACTION, 1, ACTION_ALL_LNK, parse_uid_args, uid_action}, { "gid", GID_ACTION, 1, ACTION_ALL_LNK, parse_gid_args, gid_action}, { "guid", GUID_ACTION, 2, ACTION_ALL_LNK, parse_guid_args, guid_action}, { "mode", MODE_ACTION, -2, ACTION_ALL, parse_mode_args, mode_action }, { "empty", EMPTY_ACTION, -2, ACTION_DIR, parse_empty_args, NULL}, { "move", MOVE_ACTION, -2, ACTION_ALL_LNK, NULL, NULL}, { "", 0, -1, 0, NULL, NULL} }; squashfs4.3/squashfs-tools/read_fs.c0000644000175000017500000007030012333330365017567 0ustar phillipphillip/* * Read a squashfs filesystem. This is a highly compressed read only * filesystem. * * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, * 2012, 2013, 2014 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * read_fs.c */ #define TRUE 1 #define FALSE 0 #include #include #include #include #include #include #include #include #include #ifndef linux #define __BYTE_ORDER BYTE_ORDER #define __BIG_ENDIAN BIG_ENDIAN #define __LITTLE_ENDIAN LITTLE_ENDIAN #else #include #endif #include #include "squashfs_fs.h" #include "squashfs_swap.h" #include "compressor.h" #include "xattr.h" #include "error.h" #include "mksquashfs.h" int read_block(int fd, long long start, long long *next, int expected, void *block) { unsigned short c_byte; int res, compressed; int outlen = expected ? expected : SQUASHFS_METADATA_SIZE; /* Read block size */ res = read_fs_bytes(fd, start, 2, &c_byte); if(res == 0) return 0; SQUASHFS_INSWAP_SHORTS(&c_byte, 1); compressed = SQUASHFS_COMPRESSED(c_byte); c_byte = SQUASHFS_COMPRESSED_SIZE(c_byte); /* * The block size should not be larger than * the uncompressed size (or max uncompressed size if * expected is 0) */ if (c_byte > outlen) return 0; if(compressed) { char buffer[c_byte]; int error; res = read_fs_bytes(fd, start + 2, c_byte, buffer); if(res == 0) return 0; res = compressor_uncompress(comp, block, buffer, c_byte, outlen, &error); if(res == -1) { ERROR("%s uncompress failed with error code %d\n", comp->name, error); return 0; } } else { res = read_fs_bytes(fd, start + 2, c_byte, block); if(res == 0) return 0; res = c_byte; } if(next) *next = start + 2 + c_byte; /* * if expected, then check the (uncompressed) return data * is of the expected size */ if(expected && expected != res) return 0; else return res; } #define NO_BYTES(SIZE) \ (bytes - (cur_ptr - *inode_table) < (SIZE)) #define NO_INODE_BYTES(INODE) NO_BYTES(sizeof(struct INODE)) int scan_inode_table(int fd, long long start, long long end, long long root_inode_start, int root_inode_offset, struct squashfs_super_block *sBlk, union squashfs_inode_header *dir_inode, unsigned char **inode_table, unsigned int *root_inode_block, unsigned int *root_inode_size, long long *uncompressed_file, unsigned int *uncompressed_directory, int *file_count, int *sym_count, int *dev_count, int *dir_count, int *fifo_count, int *sock_count, unsigned int *id_table) { unsigned char *cur_ptr; int byte, files = 0; unsigned int directory_start_block, bytes = 0, size = 0; struct squashfs_base_inode_header base; TRACE("scan_inode_table: start 0x%llx, end 0x%llx, root_inode_start " "0x%llx\n", start, end, root_inode_start); *root_inode_block = UINT_MAX; while(start < end) { if(start == root_inode_start) { TRACE("scan_inode_table: read compressed block 0x%llx " "containing root inode\n", start); *root_inode_block = bytes; } if(size - bytes < SQUASHFS_METADATA_SIZE) { *inode_table = realloc(*inode_table, size += SQUASHFS_METADATA_SIZE); if(*inode_table == NULL) MEM_ERROR(); } TRACE("scan_inode_table: reading block 0x%llx\n", start); byte = read_block(fd, start, &start, 0, *inode_table + bytes); if(byte == 0) goto corrupted; bytes += byte; /* If this is not the last metadata block in the inode table * then it should be SQUASHFS_METADATA_SIZE in size. * Note, we can't use expected in read_block() above for this * because we don't know if this is the last block until * after reading. */ if(start != end && byte != SQUASHFS_METADATA_SIZE) goto corrupted; } /* * We expect to have found the metadata block containing the * root inode in the above inode_table metadata block scan. If it * hasn't been found then the filesystem is corrupted */ if(*root_inode_block == UINT_MAX) goto corrupted; /* * The number of bytes available after the root inode medata block * should be at least the root inode offset + the size of a * regular directory inode, if not the filesystem is corrupted * * +-----------------------+-----------------------+ * | | directory | * | | inode | * +-----------------------+-----------------------+ * ^ ^ ^ * *root_inode_block root_inode_offset bytes */ if((bytes - *root_inode_block) < (root_inode_offset + sizeof(struct squashfs_dir_inode_header))) goto corrupted; /* * Read last inode entry which is the root directory inode, and obtain * the last directory start block index. This is used when calculating * the total uncompressed directory size. The directory bytes in the * last * block will be counted as normal. * * Note, the previous check ensures the following calculation won't * underflow, and we won't access beyond the buffer */ *root_inode_size = bytes - (*root_inode_block + root_inode_offset); bytes = *root_inode_block + root_inode_offset; SQUASHFS_SWAP_DIR_INODE_HEADER(*inode_table + bytes, &dir_inode->dir); if(dir_inode->base.inode_type == SQUASHFS_DIR_TYPE) directory_start_block = dir_inode->dir.start_block; else if(dir_inode->base.inode_type == SQUASHFS_LDIR_TYPE) { if(*root_inode_size < sizeof(struct squashfs_ldir_inode_header)) /* corrupted filesystem */ goto corrupted; SQUASHFS_SWAP_LDIR_INODE_HEADER(*inode_table + bytes, &dir_inode->ldir); directory_start_block = dir_inode->ldir.start_block; } else /* bad type, corrupted filesystem */ goto corrupted; get_uid(id_table[dir_inode->base.uid]); get_guid(id_table[dir_inode->base.guid]); /* allocate fragment to file mapping table */ file_mapping = calloc(sBlk->fragments, sizeof(struct append_file *)); if(file_mapping == NULL) MEM_ERROR(); for(cur_ptr = *inode_table; cur_ptr < *inode_table + bytes; files ++) { if(NO_INODE_BYTES(squashfs_base_inode_header)) /* corrupted filesystem */ goto corrupted; SQUASHFS_SWAP_BASE_INODE_HEADER(cur_ptr, &base); TRACE("scan_inode_table: processing inode @ byte position " "0x%x, type 0x%x\n", (unsigned int) (cur_ptr - *inode_table), base.inode_type); get_uid(id_table[base.uid]); get_guid(id_table[base.guid]); switch(base.inode_type) { case SQUASHFS_FILE_TYPE: { struct squashfs_reg_inode_header inode; int frag_bytes, blocks, i; long long start, file_bytes = 0; unsigned int *block_list; if(NO_INODE_BYTES(squashfs_reg_inode_header)) /* corrupted filesystem */ goto corrupted; SQUASHFS_SWAP_REG_INODE_HEADER(cur_ptr, &inode); frag_bytes = inode.fragment == SQUASHFS_INVALID_FRAG ? 0 : inode.file_size % sBlk->block_size; blocks = inode.fragment == SQUASHFS_INVALID_FRAG ? (inode.file_size + sBlk->block_size - 1) >> sBlk->block_log : inode.file_size >> sBlk->block_log; start = inode.start_block; TRACE("scan_inode_table: regular file, file_size %d, " "blocks %d\n", inode.file_size, blocks); if(NO_BYTES(blocks * sizeof(unsigned int))) /* corrupted filesystem */ goto corrupted; block_list = malloc(blocks * sizeof(unsigned int)); if(block_list == NULL) MEM_ERROR(); cur_ptr += sizeof(inode); SQUASHFS_SWAP_INTS(cur_ptr, block_list, blocks); *uncompressed_file += inode.file_size; (*file_count) ++; for(i = 0; i < blocks; i++) file_bytes += SQUASHFS_COMPRESSED_SIZE_BLOCK (block_list[i]); if(inode.fragment != SQUASHFS_INVALID_FRAG && inode.fragment >= sBlk->fragments) { free(block_list); goto corrupted; } add_file(start, inode.file_size, file_bytes, block_list, blocks, inode.fragment, inode.offset, frag_bytes); cur_ptr += blocks * sizeof(unsigned int); break; } case SQUASHFS_LREG_TYPE: { struct squashfs_lreg_inode_header inode; int frag_bytes, blocks, i; long long start, file_bytes = 0; unsigned int *block_list; if(NO_INODE_BYTES(squashfs_lreg_inode_header)) /* corrupted filesystem */ goto corrupted; SQUASHFS_SWAP_LREG_INODE_HEADER(cur_ptr, &inode); frag_bytes = inode.fragment == SQUASHFS_INVALID_FRAG ? 0 : inode.file_size % sBlk->block_size; blocks = inode.fragment == SQUASHFS_INVALID_FRAG ? (inode.file_size + sBlk->block_size - 1) >> sBlk->block_log : inode.file_size >> sBlk->block_log; start = inode.start_block; TRACE("scan_inode_table: extended regular " "file, file_size %lld, blocks %d\n", inode.file_size, blocks); if(NO_BYTES(blocks * sizeof(unsigned int))) /* corrupted filesystem */ goto corrupted; block_list = malloc(blocks * sizeof(unsigned int)); if(block_list == NULL) MEM_ERROR(); cur_ptr += sizeof(inode); SQUASHFS_SWAP_INTS(cur_ptr, block_list, blocks); *uncompressed_file += inode.file_size; (*file_count) ++; for(i = 0; i < blocks; i++) file_bytes += SQUASHFS_COMPRESSED_SIZE_BLOCK (block_list[i]); if(inode.fragment != SQUASHFS_INVALID_FRAG && inode.fragment >= sBlk->fragments) { free(block_list); goto corrupted; } add_file(start, inode.file_size, file_bytes, block_list, blocks, inode.fragment, inode.offset, frag_bytes); cur_ptr += blocks * sizeof(unsigned int); break; } case SQUASHFS_SYMLINK_TYPE: case SQUASHFS_LSYMLINK_TYPE: { struct squashfs_symlink_inode_header inode; if(NO_INODE_BYTES(squashfs_symlink_inode_header)) /* corrupted filesystem */ goto corrupted; SQUASHFS_SWAP_SYMLINK_INODE_HEADER(cur_ptr, &inode); (*sym_count) ++; if (inode.inode_type == SQUASHFS_LSYMLINK_TYPE) { if(NO_BYTES(inode.symlink_size + sizeof(unsigned int))) /* corrupted filesystem */ goto corrupted; cur_ptr += sizeof(inode) + inode.symlink_size + sizeof(unsigned int); } else { if(NO_BYTES(inode.symlink_size)) /* corrupted filesystem */ goto corrupted; cur_ptr += sizeof(inode) + inode.symlink_size; } break; } case SQUASHFS_DIR_TYPE: { struct squashfs_dir_inode_header dir_inode; if(NO_INODE_BYTES(squashfs_dir_inode_header)) /* corrupted filesystem */ goto corrupted; SQUASHFS_SWAP_DIR_INODE_HEADER(cur_ptr, &dir_inode); if(dir_inode.start_block < directory_start_block) *uncompressed_directory += dir_inode.file_size; (*dir_count) ++; cur_ptr += sizeof(struct squashfs_dir_inode_header); break; } case SQUASHFS_LDIR_TYPE: { struct squashfs_ldir_inode_header dir_inode; int i; if(NO_INODE_BYTES(squashfs_ldir_inode_header)) /* corrupted filesystem */ goto corrupted; SQUASHFS_SWAP_LDIR_INODE_HEADER(cur_ptr, &dir_inode); if(dir_inode.start_block < directory_start_block) *uncompressed_directory += dir_inode.file_size; (*dir_count) ++; cur_ptr += sizeof(struct squashfs_ldir_inode_header); for(i = 0; i < dir_inode.i_count; i++) { struct squashfs_dir_index index; if(NO_BYTES(sizeof(index))) /* corrupted filesystem */ goto corrupted; SQUASHFS_SWAP_DIR_INDEX(cur_ptr, &index); if(NO_BYTES(index.size + 1)) /* corrupted filesystem */ goto corrupted; cur_ptr += sizeof(index) + index.size + 1; } break; } case SQUASHFS_BLKDEV_TYPE: case SQUASHFS_CHRDEV_TYPE: if(NO_INODE_BYTES(squashfs_dev_inode_header)) /* corrupted filesystem */ goto corrupted; (*dev_count) ++; cur_ptr += sizeof(struct squashfs_dev_inode_header); break; case SQUASHFS_LBLKDEV_TYPE: case SQUASHFS_LCHRDEV_TYPE: if(NO_INODE_BYTES(squashfs_ldev_inode_header)) /* corrupted filesystem */ goto corrupted; (*dev_count) ++; cur_ptr += sizeof(struct squashfs_ldev_inode_header); break; case SQUASHFS_FIFO_TYPE: if(NO_INODE_BYTES(squashfs_ipc_inode_header)) /* corrupted filesystem */ goto corrupted; (*fifo_count) ++; cur_ptr += sizeof(struct squashfs_ipc_inode_header); break; case SQUASHFS_LFIFO_TYPE: if(NO_INODE_BYTES(squashfs_lipc_inode_header)) /* corrupted filesystem */ goto corrupted; (*fifo_count) ++; cur_ptr += sizeof(struct squashfs_lipc_inode_header); break; case SQUASHFS_SOCKET_TYPE: if(NO_INODE_BYTES(squashfs_ipc_inode_header)) /* corrupted filesystem */ goto corrupted; (*sock_count) ++; cur_ptr += sizeof(struct squashfs_ipc_inode_header); break; case SQUASHFS_LSOCKET_TYPE: if(NO_INODE_BYTES(squashfs_lipc_inode_header)) /* corrupted filesystem */ goto corrupted; (*sock_count) ++; cur_ptr += sizeof(struct squashfs_lipc_inode_header); break; default: ERROR("Unknown inode type %d in scan_inode_table!\n", base.inode_type); goto corrupted; } } printf("Read existing filesystem, %d inodes scanned\n", files); return TRUE; corrupted: ERROR("scan_inode_table: filesystem corruption detected in " "scanning metadata\n"); free(*inode_table); return FALSE; } struct compressor *read_super(int fd, struct squashfs_super_block *sBlk, char *source) { int res, bytes = 0; char buffer[SQUASHFS_METADATA_SIZE] __attribute__ ((aligned)); res = read_fs_bytes(fd, SQUASHFS_START, sizeof(struct squashfs_super_block), sBlk); if(res == 0) { ERROR("Can't find a SQUASHFS superblock on %s\n", source); ERROR("Wrong filesystem or filesystem is corrupted!\n"); goto failed_mount; } SQUASHFS_INSWAP_SUPER_BLOCK(sBlk); if(sBlk->s_magic != SQUASHFS_MAGIC) { if(sBlk->s_magic == SQUASHFS_MAGIC_SWAP) ERROR("Pre 4.0 big-endian filesystem on %s, appending" " to this is unsupported\n", source); else { ERROR("Can't find a SQUASHFS superblock on %s\n", source); ERROR("Wrong filesystem or filesystem is corrupted!\n"); } goto failed_mount; } /* Check the MAJOR & MINOR versions */ if(sBlk->s_major != SQUASHFS_MAJOR || sBlk->s_minor > SQUASHFS_MINOR) { if(sBlk->s_major < 4) ERROR("Filesystem on %s is a SQUASHFS %d.%d filesystem." " Appending\nto SQUASHFS %d.%d filesystems is " "not supported. Please convert it to a " "SQUASHFS 4 filesystem\n", source, sBlk->s_major, sBlk->s_minor, sBlk->s_major, sBlk->s_minor); else ERROR("Filesystem on %s is %d.%d, which is a later " "filesystem version than I support\n", source, sBlk->s_major, sBlk->s_minor); goto failed_mount; } /* Check the compression type */ comp = lookup_compressor_id(sBlk->compression); if(!comp->supported) { ERROR("Filesystem on %s uses %s compression, this is " "unsupported by this version\n", source, comp->name); ERROR("Compressors available:\n"); display_compressors("", ""); goto failed_mount; } /* * Read extended superblock information from disk. * * Read compressor specific options from disk if present, and pass * to compressor to set compressor options. * * Note, if there's no compressor options present, the compressor * is still called to set the default options (the defaults may have * been changed by the user specifying options on the command * line which need to be over-ridden). * * Compressor_extract_options is also used to ensure that * we know how decompress a filesystem compressed with these * compression options. */ if(SQUASHFS_COMP_OPTS(sBlk->flags)) { bytes = read_block(fd, sizeof(*sBlk), NULL, 0, buffer); if(bytes == 0) { ERROR("Failed to read compressor options from append " "filesystem\n"); ERROR("Filesystem corrupted?\n"); goto failed_mount; } } res = compressor_extract_options(comp, sBlk->block_size, buffer, bytes); if(res == -1) { ERROR("Compressor failed to set compressor options\n"); goto failed_mount; } printf("Found a valid %sSQUASHFS superblock on %s.\n", SQUASHFS_EXPORTABLE(sBlk->flags) ? "exportable " : "", source); printf("\tCompression used %s\n", comp->name); printf("\tInodes are %scompressed\n", SQUASHFS_UNCOMPRESSED_INODES(sBlk->flags) ? "un" : ""); printf("\tData is %scompressed\n", SQUASHFS_UNCOMPRESSED_DATA(sBlk->flags) ? "un" : ""); printf("\tFragments are %scompressed\n", SQUASHFS_UNCOMPRESSED_FRAGMENTS(sBlk->flags) ? "un" : ""); printf("\tXattrs are %scompressed\n", SQUASHFS_UNCOMPRESSED_XATTRS(sBlk->flags) ? "un" : ""); printf("\tFragments are %spresent in the filesystem\n", SQUASHFS_NO_FRAGMENTS(sBlk->flags) ? "not " : ""); printf("\tAlways-use-fragments option is %sspecified\n", SQUASHFS_ALWAYS_FRAGMENTS(sBlk->flags) ? "" : "not "); printf("\tDuplicates are %sremoved\n", SQUASHFS_DUPLICATES(sBlk->flags) ? "" : "not "); printf("\tXattrs are %sstored\n", SQUASHFS_NO_XATTRS(sBlk->flags) ? "not " : ""); printf("\tFilesystem size %.2f Kbytes (%.2f Mbytes)\n", sBlk->bytes_used / 1024.0, sBlk->bytes_used / (1024.0 * 1024.0)); printf("\tBlock size %d\n", sBlk->block_size); printf("\tNumber of fragments %d\n", sBlk->fragments); printf("\tNumber of inodes %d\n", sBlk->inodes); printf("\tNumber of ids %d\n", sBlk->no_ids); TRACE("sBlk->inode_table_start %llx\n", sBlk->inode_table_start); TRACE("sBlk->directory_table_start %llx\n", sBlk->directory_table_start); TRACE("sBlk->id_table_start %llx\n", sBlk->id_table_start); TRACE("sBlk->fragment_table_start %llx\n", sBlk->fragment_table_start); TRACE("sBlk->lookup_table_start %llx\n", sBlk->lookup_table_start); TRACE("sBlk->xattr_id_table_start %llx\n", sBlk->xattr_id_table_start); printf("\n"); return comp; failed_mount: return NULL; } unsigned char *squashfs_readdir(int fd, int root_entries, unsigned int directory_start_block, int offset, int size, unsigned int *last_directory_block, struct squashfs_super_block *sBlk, void (push_directory_entry)(char *, squashfs_inode, int, int)) { struct squashfs_dir_header dirh; char buffer[sizeof(struct squashfs_dir_entry) + SQUASHFS_NAME_LEN + 1] __attribute__ ((aligned)); struct squashfs_dir_entry *dire = (struct squashfs_dir_entry *) buffer; unsigned char *directory_table = NULL; int byte, bytes = 0, dir_count; long long start = sBlk->directory_table_start + directory_start_block, last_start_block = start; size += offset; directory_table = malloc((size + SQUASHFS_METADATA_SIZE * 2 - 1) & ~(SQUASHFS_METADATA_SIZE - 1)); if(directory_table == NULL) MEM_ERROR(); while(bytes < size) { int expected = (size - bytes) >= SQUASHFS_METADATA_SIZE ? SQUASHFS_METADATA_SIZE : 0; TRACE("squashfs_readdir: reading block 0x%llx, bytes read so " "far %d\n", start, bytes); last_start_block = start; byte = read_block(fd, start, &start, expected, directory_table + bytes); if(byte == 0) { ERROR("Failed to read directory\n"); ERROR("Filesystem corrupted?\n"); free(directory_table); return NULL; } bytes += byte; } if(!root_entries) goto all_done; bytes = offset; while(bytes < size) { SQUASHFS_SWAP_DIR_HEADER(directory_table + bytes, &dirh); dir_count = dirh.count + 1; TRACE("squashfs_readdir: Read directory header @ byte position " "0x%x, 0x%x directory entries\n", bytes, dir_count); bytes += sizeof(dirh); while(dir_count--) { SQUASHFS_SWAP_DIR_ENTRY(directory_table + bytes, dire); bytes += sizeof(*dire); memcpy(dire->name, directory_table + bytes, dire->size + 1); dire->name[dire->size + 1] = '\0'; TRACE("squashfs_readdir: pushing directory entry %s, " "inode %x:%x, type 0x%x\n", dire->name, dirh.start_block, dire->offset, dire->type); push_directory_entry(dire->name, SQUASHFS_MKINODE(dirh.start_block, dire->offset), dirh.inode_number + dire->inode_number, dire->type); bytes += dire->size + 1; } } all_done: *last_directory_block = (unsigned int) last_start_block - sBlk->directory_table_start; return directory_table; } unsigned int *read_id_table(int fd, struct squashfs_super_block *sBlk) { int indexes = SQUASHFS_ID_BLOCKS(sBlk->no_ids); long long index[indexes]; int bytes = SQUASHFS_ID_BYTES(sBlk->no_ids); unsigned int *id_table; int res, i; id_table = malloc(bytes); if(id_table == NULL) MEM_ERROR(); res = read_fs_bytes(fd, sBlk->id_table_start, SQUASHFS_ID_BLOCK_BYTES(sBlk->no_ids), index); if(res == 0) { ERROR("Failed to read id table index\n"); ERROR("Filesystem corrupted?\n"); free(id_table); return NULL; } SQUASHFS_INSWAP_ID_BLOCKS(index, indexes); for(i = 0; i < indexes; i++) { int expected = (i + 1) != indexes ? SQUASHFS_METADATA_SIZE : bytes & (SQUASHFS_METADATA_SIZE - 1); int length = read_block(fd, index[i], NULL, expected, ((unsigned char *) id_table) + (i * SQUASHFS_METADATA_SIZE)); TRACE("Read id table block %d, from 0x%llx, length %d\n", i, index[i], length); if(length == 0) { ERROR("Failed to read id table block %d, from 0x%llx, " "length %d\n", i, index[i], length); ERROR("Filesystem corrupted?\n"); free(id_table); return NULL; } } SQUASHFS_INSWAP_INTS(id_table, sBlk->no_ids); for(i = 0; i < sBlk->no_ids; i++) { TRACE("Adding id %d to id tables\n", id_table[i]); create_id(id_table[i]); } return id_table; } int read_fragment_table(int fd, struct squashfs_super_block *sBlk, struct squashfs_fragment_entry **fragment_table) { int res, i; int bytes = SQUASHFS_FRAGMENT_BYTES(sBlk->fragments); int indexes = SQUASHFS_FRAGMENT_INDEXES(sBlk->fragments); long long fragment_table_index[indexes]; TRACE("read_fragment_table: %d fragments, reading %d fragment indexes " "from 0x%llx\n", sBlk->fragments, indexes, sBlk->fragment_table_start); if(sBlk->fragments == 0) return 1; *fragment_table = malloc(bytes); if(*fragment_table == NULL) MEM_ERROR(); res = read_fs_bytes(fd, sBlk->fragment_table_start, SQUASHFS_FRAGMENT_INDEX_BYTES(sBlk->fragments), fragment_table_index); if(res == 0) { ERROR("Failed to read fragment table index\n"); ERROR("Filesystem corrupted?\n"); free(*fragment_table); return 0; } SQUASHFS_INSWAP_FRAGMENT_INDEXES(fragment_table_index, indexes); for(i = 0; i < indexes; i++) { int expected = (i + 1) != indexes ? SQUASHFS_METADATA_SIZE : bytes & (SQUASHFS_METADATA_SIZE - 1); int length = read_block(fd, fragment_table_index[i], NULL, expected, ((unsigned char *) *fragment_table) + (i * SQUASHFS_METADATA_SIZE)); TRACE("Read fragment table block %d, from 0x%llx, length %d\n", i, fragment_table_index[i], length); if(length == 0) { ERROR("Failed to read fragment table block %d, from " "0x%llx, length %d\n", i, fragment_table_index[i], length); ERROR("Filesystem corrupted?\n"); free(*fragment_table); return 0; } } for(i = 0; i < sBlk->fragments; i++) SQUASHFS_INSWAP_FRAGMENT_ENTRY(&(*fragment_table)[i]); return 1; } int read_inode_lookup_table(int fd, struct squashfs_super_block *sBlk, squashfs_inode **inode_lookup_table) { int lookup_bytes = SQUASHFS_LOOKUP_BYTES(sBlk->inodes); int indexes = SQUASHFS_LOOKUP_BLOCKS(sBlk->inodes); long long index[indexes]; int res, i; if(sBlk->lookup_table_start == SQUASHFS_INVALID_BLK) return 1; *inode_lookup_table = malloc(lookup_bytes); if(*inode_lookup_table == NULL) MEM_ERROR(); res = read_fs_bytes(fd, sBlk->lookup_table_start, SQUASHFS_LOOKUP_BLOCK_BYTES(sBlk->inodes), index); if(res == 0) { ERROR("Failed to read inode lookup table index\n"); ERROR("Filesystem corrupted?\n"); free(*inode_lookup_table); return 0; } SQUASHFS_INSWAP_LONG_LONGS(index, indexes); for(i = 0; i < indexes; i++) { int expected = (i + 1) != indexes ? SQUASHFS_METADATA_SIZE : lookup_bytes & (SQUASHFS_METADATA_SIZE - 1); int length = read_block(fd, index[i], NULL, expected, ((unsigned char *) *inode_lookup_table) + (i * SQUASHFS_METADATA_SIZE)); TRACE("Read inode lookup table block %d, from 0x%llx, length " "%d\n", i, index[i], length); if(length == 0) { ERROR("Failed to read inode lookup table block %d, " "from 0x%llx, length %d\n", i, index[i], length); ERROR("Filesystem corrupted?\n"); free(*inode_lookup_table); return 0; } } SQUASHFS_INSWAP_LONG_LONGS(*inode_lookup_table, sBlk->inodes); return 1; } long long read_filesystem(char *root_name, int fd, struct squashfs_super_block *sBlk, char **cinode_table, char **data_cache, char **cdirectory_table, char **directory_data_cache, unsigned int *last_directory_block, unsigned int *inode_dir_offset, unsigned int *inode_dir_file_size, unsigned int *root_inode_size, unsigned int *inode_dir_start_block, int *file_count, int *sym_count, int *dev_count, int *dir_count, int *fifo_count, int *sock_count, long long *uncompressed_file, unsigned int *uncompressed_inode, unsigned int *uncompressed_directory, unsigned int *inode_dir_inode_number, unsigned int *inode_dir_parent_inode, void (push_directory_entry)(char *, squashfs_inode, int, int), struct squashfs_fragment_entry **fragment_table, squashfs_inode **inode_lookup_table) { unsigned char *inode_table = NULL, *directory_table = NULL; long long start = sBlk->inode_table_start; long long end = sBlk->directory_table_start; long long root_inode_start = start + SQUASHFS_INODE_BLK(sBlk->root_inode); unsigned int root_inode_offset = SQUASHFS_INODE_OFFSET(sBlk->root_inode); unsigned int root_inode_block; union squashfs_inode_header inode; unsigned int *id_table = NULL; int res; printf("Scanning existing filesystem...\n"); if(get_xattrs(fd, sBlk) == 0) goto error; if(read_fragment_table(fd, sBlk, fragment_table) == 0) goto error; if(read_inode_lookup_table(fd, sBlk, inode_lookup_table) == 0) goto error; id_table = read_id_table(fd, sBlk); if(id_table == NULL) goto error; res = scan_inode_table(fd, start, end, root_inode_start, root_inode_offset, sBlk, &inode, &inode_table, &root_inode_block, root_inode_size, uncompressed_file, uncompressed_directory, file_count, sym_count, dev_count, dir_count, fifo_count, sock_count, id_table); if(res == 0) goto error; *uncompressed_inode = root_inode_block; if(inode.base.inode_type == SQUASHFS_DIR_TYPE || inode.base.inode_type == SQUASHFS_LDIR_TYPE) { if(inode.base.inode_type == SQUASHFS_DIR_TYPE) { *inode_dir_start_block = inode.dir.start_block; *inode_dir_offset = inode.dir.offset; *inode_dir_file_size = inode.dir.file_size - 3; *inode_dir_inode_number = inode.dir.inode_number; *inode_dir_parent_inode = inode.dir.parent_inode; } else { *inode_dir_start_block = inode.ldir.start_block; *inode_dir_offset = inode.ldir.offset; *inode_dir_file_size = inode.ldir.file_size - 3; *inode_dir_inode_number = inode.ldir.inode_number; *inode_dir_parent_inode = inode.ldir.parent_inode; } directory_table = squashfs_readdir(fd, !root_name, *inode_dir_start_block, *inode_dir_offset, *inode_dir_file_size, last_directory_block, sBlk, push_directory_entry); if(directory_table == NULL) goto error; root_inode_start -= start; *cinode_table = malloc(root_inode_start); if(*cinode_table == NULL) MEM_ERROR(); res = read_fs_bytes(fd, start, root_inode_start, *cinode_table); if(res == 0) { ERROR("Failed to read inode table\n"); ERROR("Filesystem corrupted?\n"); goto error; } *cdirectory_table = malloc(*last_directory_block); if(*cdirectory_table == NULL) MEM_ERROR(); res = read_fs_bytes(fd, sBlk->directory_table_start, *last_directory_block, *cdirectory_table); if(res == 0) { ERROR("Failed to read directory table\n"); ERROR("Filesystem corrupted?\n"); goto error; } *data_cache = malloc(root_inode_offset + *root_inode_size); if(*data_cache == NULL) MEM_ERROR(); memcpy(*data_cache, inode_table + root_inode_block, root_inode_offset + *root_inode_size); *directory_data_cache = malloc(*inode_dir_offset + *inode_dir_file_size); if(*directory_data_cache == NULL) MEM_ERROR(); memcpy(*directory_data_cache, directory_table, *inode_dir_offset + *inode_dir_file_size); free(id_table); free(inode_table); free(directory_table); return sBlk->inode_table_start; } error: free(id_table); free(inode_table); free(directory_table); return 0; } squashfs4.3/squashfs-tools/squashfs_compat.h0000644000175000017500000006336312333330365021404 0ustar phillipphillip#ifndef SQUASHFS_COMPAT #define SQUASHFS_COMPAT /* * Squashfs * * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2014 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * squashfs_compat.h */ /* * definitions for structures on disk - layout 3.x */ #define SQUASHFS_CHECK 2 #define SQUASHFS_CHECK_DATA(flags) SQUASHFS_BIT(flags, \ SQUASHFS_CHECK) /* Max number of uids and gids */ #define SQUASHFS_UIDS 256 #define SQUASHFS_GUIDS 255 struct squashfs_super_block_3 { unsigned int s_magic; unsigned int inodes; unsigned int bytes_used_2; unsigned int uid_start_2; unsigned int guid_start_2; unsigned int inode_table_start_2; unsigned int directory_table_start_2; unsigned int s_major:16; unsigned int s_minor:16; unsigned int block_size_1:16; unsigned int block_log:16; unsigned int flags:8; unsigned int no_uids:8; unsigned int no_guids:8; int mkfs_time /* time of filesystem creation */; squashfs_inode root_inode; unsigned int block_size; unsigned int fragments; unsigned int fragment_table_start_2; long long bytes_used; long long uid_start; long long guid_start; long long inode_table_start; long long directory_table_start; long long fragment_table_start; long long lookup_table_start; } __attribute__ ((packed)); struct squashfs_dir_index_3 { unsigned int index; unsigned int start_block; unsigned char size; unsigned char name[0]; } __attribute__ ((packed)); struct squashfs_base_inode_header_3 { unsigned int inode_type:4; unsigned int mode:12; unsigned int uid:8; unsigned int guid:8; int mtime; unsigned int inode_number; } __attribute__ ((packed)); struct squashfs_ipc_inode_header_3 { unsigned int inode_type:4; unsigned int mode:12; unsigned int uid:8; unsigned int guid:8; int mtime; unsigned int inode_number; unsigned int nlink; } __attribute__ ((packed)); struct squashfs_dev_inode_header_3 { unsigned int inode_type:4; unsigned int mode:12; unsigned int uid:8; unsigned int guid:8; int mtime; unsigned int inode_number; unsigned int nlink; unsigned short rdev; } __attribute__ ((packed)); struct squashfs_symlink_inode_header_3 { unsigned int inode_type:4; unsigned int mode:12; unsigned int uid:8; unsigned int guid:8; int mtime; unsigned int inode_number; unsigned int nlink; unsigned short symlink_size; char symlink[0]; } __attribute__ ((packed)); struct squashfs_reg_inode_header_3 { unsigned int inode_type:4; unsigned int mode:12; unsigned int uid:8; unsigned int guid:8; int mtime; unsigned int inode_number; squashfs_block start_block; unsigned int fragment; unsigned int offset; unsigned int file_size; unsigned short block_list[0]; } __attribute__ ((packed)); struct squashfs_lreg_inode_header_3 { unsigned int inode_type:4; unsigned int mode:12; unsigned int uid:8; unsigned int guid:8; int mtime; unsigned int inode_number; unsigned int nlink; squashfs_block start_block; unsigned int fragment; unsigned int offset; long long file_size; unsigned short block_list[0]; } __attribute__ ((packed)); struct squashfs_dir_inode_header_3 { unsigned int inode_type:4; unsigned int mode:12; unsigned int uid:8; unsigned int guid:8; int mtime; unsigned int inode_number; unsigned int nlink; unsigned int file_size:19; unsigned int offset:13; unsigned int start_block; unsigned int parent_inode; } __attribute__ ((packed)); struct squashfs_ldir_inode_header_3 { unsigned int inode_type:4; unsigned int mode:12; unsigned int uid:8; unsigned int guid:8; int mtime; unsigned int inode_number; unsigned int nlink; unsigned int file_size:27; unsigned int offset:13; unsigned int start_block; unsigned int i_count:16; unsigned int parent_inode; struct squashfs_dir_index_3 index[0]; } __attribute__ ((packed)); union squashfs_inode_header_3 { struct squashfs_base_inode_header_3 base; struct squashfs_dev_inode_header_3 dev; struct squashfs_symlink_inode_header_3 symlink; struct squashfs_reg_inode_header_3 reg; struct squashfs_lreg_inode_header_3 lreg; struct squashfs_dir_inode_header_3 dir; struct squashfs_ldir_inode_header_3 ldir; struct squashfs_ipc_inode_header_3 ipc; }; struct squashfs_dir_entry_3 { unsigned int offset:13; unsigned int type:3; unsigned int size:8; int inode_number:16; char name[0]; } __attribute__ ((packed)); struct squashfs_dir_header_3 { unsigned int count:8; unsigned int start_block; unsigned int inode_number; } __attribute__ ((packed)); struct squashfs_fragment_entry_3 { long long start_block; unsigned int size; unsigned int pending; } __attribute__ ((packed)); typedef struct squashfs_super_block_3 squashfs_super_block_3; typedef struct squashfs_dir_index_3 squashfs_dir_index_3; typedef struct squashfs_base_inode_header_3 squashfs_base_inode_header_3; typedef struct squashfs_ipc_inode_header_3 squashfs_ipc_inode_header_3; typedef struct squashfs_dev_inode_header_3 squashfs_dev_inode_header_3; typedef struct squashfs_symlink_inode_header_3 squashfs_symlink_inode_header_3; typedef struct squashfs_reg_inode_header_3 squashfs_reg_inode_header_3; typedef struct squashfs_lreg_inode_header_3 squashfs_lreg_inode_header_3; typedef struct squashfs_dir_inode_header_3 squashfs_dir_inode_header_3; typedef struct squashfs_ldir_inode_header_3 squashfs_ldir_inode_header_3; typedef struct squashfs_dir_entry_3 squashfs_dir_entry_3; typedef struct squashfs_dir_header_3 squashfs_dir_header_3; typedef struct squashfs_fragment_entry_3 squashfs_fragment_entry_3; /* * macros to convert each packed bitfield structure from little endian to big * endian and vice versa. These are needed when creating or using a filesystem * on a machine with different byte ordering to the target architecture. * */ #define SQUASHFS_SWAP_START \ int bits;\ int b_pos;\ unsigned long long val;\ unsigned char *s;\ unsigned char *d; #define SQUASHFS_SWAP_SUPER_BLOCK_3(s, d) {\ SQUASHFS_SWAP_START\ SQUASHFS_MEMSET(s, d, sizeof(struct squashfs_super_block_3));\ SQUASHFS_SWAP((s)->s_magic, d, 0, 32);\ SQUASHFS_SWAP((s)->inodes, d, 32, 32);\ SQUASHFS_SWAP((s)->bytes_used_2, d, 64, 32);\ SQUASHFS_SWAP((s)->uid_start_2, d, 96, 32);\ SQUASHFS_SWAP((s)->guid_start_2, d, 128, 32);\ SQUASHFS_SWAP((s)->inode_table_start_2, d, 160, 32);\ SQUASHFS_SWAP((s)->directory_table_start_2, d, 192, 32);\ SQUASHFS_SWAP((s)->s_major, d, 224, 16);\ SQUASHFS_SWAP((s)->s_minor, d, 240, 16);\ SQUASHFS_SWAP((s)->block_size_1, d, 256, 16);\ SQUASHFS_SWAP((s)->block_log, d, 272, 16);\ SQUASHFS_SWAP((s)->flags, d, 288, 8);\ SQUASHFS_SWAP((s)->no_uids, d, 296, 8);\ SQUASHFS_SWAP((s)->no_guids, d, 304, 8);\ SQUASHFS_SWAP((s)->mkfs_time, d, 312, 32);\ SQUASHFS_SWAP((s)->root_inode, d, 344, 64);\ SQUASHFS_SWAP((s)->block_size, d, 408, 32);\ SQUASHFS_SWAP((s)->fragments, d, 440, 32);\ SQUASHFS_SWAP((s)->fragment_table_start_2, d, 472, 32);\ SQUASHFS_SWAP((s)->bytes_used, d, 504, 64);\ SQUASHFS_SWAP((s)->uid_start, d, 568, 64);\ SQUASHFS_SWAP((s)->guid_start, d, 632, 64);\ SQUASHFS_SWAP((s)->inode_table_start, d, 696, 64);\ SQUASHFS_SWAP((s)->directory_table_start, d, 760, 64);\ SQUASHFS_SWAP((s)->fragment_table_start, d, 824, 64);\ SQUASHFS_SWAP((s)->lookup_table_start, d, 888, 64);\ } #define SQUASHFS_SWAP_BASE_INODE_CORE_3(s, d, n)\ SQUASHFS_MEMSET(s, d, n);\ SQUASHFS_SWAP((s)->inode_type, d, 0, 4);\ SQUASHFS_SWAP((s)->mode, d, 4, 12);\ SQUASHFS_SWAP((s)->uid, d, 16, 8);\ SQUASHFS_SWAP((s)->guid, d, 24, 8);\ SQUASHFS_SWAP((s)->mtime, d, 32, 32);\ SQUASHFS_SWAP((s)->inode_number, d, 64, 32); #define SQUASHFS_SWAP_BASE_INODE_HEADER_3(s, d, n) {\ SQUASHFS_SWAP_START\ SQUASHFS_SWAP_BASE_INODE_CORE_3(s, d, n)\ } #define SQUASHFS_SWAP_IPC_INODE_HEADER_3(s, d) {\ SQUASHFS_SWAP_START\ SQUASHFS_SWAP_BASE_INODE_CORE_3(s, d, \ sizeof(struct squashfs_ipc_inode_header_3))\ SQUASHFS_SWAP((s)->nlink, d, 96, 32);\ } #define SQUASHFS_SWAP_DEV_INODE_HEADER_3(s, d) {\ SQUASHFS_SWAP_START\ SQUASHFS_SWAP_BASE_INODE_CORE_3(s, d, \ sizeof(struct squashfs_dev_inode_header_3)); \ SQUASHFS_SWAP((s)->nlink, d, 96, 32);\ SQUASHFS_SWAP((s)->rdev, d, 128, 16);\ } #define SQUASHFS_SWAP_SYMLINK_INODE_HEADER_3(s, d) {\ SQUASHFS_SWAP_START\ SQUASHFS_SWAP_BASE_INODE_CORE_3(s, d, \ sizeof(struct squashfs_symlink_inode_header_3));\ SQUASHFS_SWAP((s)->nlink, d, 96, 32);\ SQUASHFS_SWAP((s)->symlink_size, d, 128, 16);\ } #define SQUASHFS_SWAP_REG_INODE_HEADER_3(s, d) {\ SQUASHFS_SWAP_START\ SQUASHFS_SWAP_BASE_INODE_CORE_3(s, d, \ sizeof(struct squashfs_reg_inode_header_3));\ SQUASHFS_SWAP((s)->start_block, d, 96, 64);\ SQUASHFS_SWAP((s)->fragment, d, 160, 32);\ SQUASHFS_SWAP((s)->offset, d, 192, 32);\ SQUASHFS_SWAP((s)->file_size, d, 224, 32);\ } #define SQUASHFS_SWAP_LREG_INODE_HEADER_3(s, d) {\ SQUASHFS_SWAP_START\ SQUASHFS_SWAP_BASE_INODE_CORE_3(s, d, \ sizeof(struct squashfs_lreg_inode_header_3));\ SQUASHFS_SWAP((s)->nlink, d, 96, 32);\ SQUASHFS_SWAP((s)->start_block, d, 128, 64);\ SQUASHFS_SWAP((s)->fragment, d, 192, 32);\ SQUASHFS_SWAP((s)->offset, d, 224, 32);\ SQUASHFS_SWAP((s)->file_size, d, 256, 64);\ } #define SQUASHFS_SWAP_DIR_INODE_HEADER_3(s, d) {\ SQUASHFS_SWAP_START\ SQUASHFS_SWAP_BASE_INODE_CORE_3(s, d, \ sizeof(struct squashfs_dir_inode_header_3));\ SQUASHFS_SWAP((s)->nlink, d, 96, 32);\ SQUASHFS_SWAP((s)->file_size, d, 128, 19);\ SQUASHFS_SWAP((s)->offset, d, 147, 13);\ SQUASHFS_SWAP((s)->start_block, d, 160, 32);\ SQUASHFS_SWAP((s)->parent_inode, d, 192, 32);\ } #define SQUASHFS_SWAP_LDIR_INODE_HEADER_3(s, d) {\ SQUASHFS_SWAP_START\ SQUASHFS_SWAP_BASE_INODE_CORE_3(s, d, \ sizeof(struct squashfs_ldir_inode_header_3));\ SQUASHFS_SWAP((s)->nlink, d, 96, 32);\ SQUASHFS_SWAP((s)->file_size, d, 128, 27);\ SQUASHFS_SWAP((s)->offset, d, 155, 13);\ SQUASHFS_SWAP((s)->start_block, d, 168, 32);\ SQUASHFS_SWAP((s)->i_count, d, 200, 16);\ SQUASHFS_SWAP((s)->parent_inode, d, 216, 32);\ } #define SQUASHFS_SWAP_DIR_INDEX_3(s, d) {\ SQUASHFS_SWAP_START\ SQUASHFS_MEMSET(s, d, sizeof(struct squashfs_dir_index_3));\ SQUASHFS_SWAP((s)->index, d, 0, 32);\ SQUASHFS_SWAP((s)->start_block, d, 32, 32);\ SQUASHFS_SWAP((s)->size, d, 64, 8);\ } #define SQUASHFS_SWAP_DIR_HEADER_3(s, d) {\ SQUASHFS_SWAP_START\ SQUASHFS_MEMSET(s, d, sizeof(struct squashfs_dir_header_3));\ SQUASHFS_SWAP((s)->count, d, 0, 8);\ SQUASHFS_SWAP((s)->start_block, d, 8, 32);\ SQUASHFS_SWAP((s)->inode_number, d, 40, 32);\ } #define SQUASHFS_SWAP_DIR_ENTRY_3(s, d) {\ SQUASHFS_SWAP_START\ SQUASHFS_MEMSET(s, d, sizeof(struct squashfs_dir_entry_3));\ SQUASHFS_SWAP((s)->offset, d, 0, 13);\ SQUASHFS_SWAP((s)->type, d, 13, 3);\ SQUASHFS_SWAP((s)->size, d, 16, 8);\ SQUASHFS_SWAP((s)->inode_number, d, 24, 16);\ } #define SQUASHFS_SWAP_INODE_T_3(s, d) SQUASHFS_SWAP_LONG_LONGS_3(s, d, 1) #define SQUASHFS_SWAP_SHORTS_3(s, d, n) {\ int entry;\ int bit_position;\ SQUASHFS_SWAP_START\ SQUASHFS_MEMSET(s, d, n * 2);\ for(entry = 0, bit_position = 0; entry < n; entry++, bit_position += \ 16)\ SQUASHFS_SWAP(s[entry], d, bit_position, 16);\ } #define SQUASHFS_SWAP_INTS_3(s, d, n) {\ int entry;\ int bit_position;\ SQUASHFS_SWAP_START\ SQUASHFS_MEMSET(s, d, n * 4);\ for(entry = 0, bit_position = 0; entry < n; entry++, bit_position += \ 32)\ SQUASHFS_SWAP(s[entry], d, bit_position, 32);\ } #define SQUASHFS_SWAP_LONG_LONGS_3(s, d, n) {\ int entry;\ int bit_position;\ SQUASHFS_SWAP_START\ SQUASHFS_MEMSET(s, d, n * 8);\ for(entry = 0, bit_position = 0; entry < n; entry++, bit_position += \ 64)\ SQUASHFS_SWAP(s[entry], d, bit_position, 64);\ } #define SQUASHFS_SWAP_DATA(s, d, n, bits) {\ int entry;\ int bit_position;\ SQUASHFS_SWAP_START\ SQUASHFS_MEMSET(s, d, n * bits / 8);\ for(entry = 0, bit_position = 0; entry < n; entry++, bit_position += \ bits)\ SQUASHFS_SWAP(s[entry], d, bit_position, bits);\ } #define SQUASHFS_SWAP_FRAGMENT_INDEXES_3(s, d, n) SQUASHFS_SWAP_LONG_LONGS_3(s, d, n) #define SQUASHFS_SWAP_LOOKUP_BLOCKS_3(s, d, n) SQUASHFS_SWAP_LONG_LONGS_3(s, d, n) #define SQUASHFS_SWAP_FRAGMENT_ENTRY_3(s, d) {\ SQUASHFS_SWAP_START\ SQUASHFS_MEMSET(s, d, sizeof(struct squashfs_fragment_entry_3));\ SQUASHFS_SWAP((s)->start_block, d, 0, 64);\ SQUASHFS_SWAP((s)->size, d, 64, 32);\ } /* fragment and fragment table defines */ #define SQUASHFS_FRAGMENT_BYTES_3(A) ((A) * sizeof(struct squashfs_fragment_entry_3)) #define SQUASHFS_FRAGMENT_INDEX_3(A) (SQUASHFS_FRAGMENT_BYTES_3(A) / \ SQUASHFS_METADATA_SIZE) #define SQUASHFS_FRAGMENT_INDEX_OFFSET_3(A) (SQUASHFS_FRAGMENT_BYTES_3(A) % \ SQUASHFS_METADATA_SIZE) #define SQUASHFS_FRAGMENT_INDEXES_3(A) ((SQUASHFS_FRAGMENT_BYTES_3(A) + \ SQUASHFS_METADATA_SIZE - 1) / \ SQUASHFS_METADATA_SIZE) #define SQUASHFS_FRAGMENT_INDEX_BYTES_3(A) (SQUASHFS_FRAGMENT_INDEXES_3(A) *\ sizeof(long long)) /* * definitions for structures on disk - layout 1.x */ #define SQUASHFS_TYPES 5 #define SQUASHFS_IPC_TYPE 0 struct squashfs_base_inode_header_1 { unsigned int inode_type:4; unsigned int mode:12; /* protection */ unsigned int uid:4; /* index into uid table */ unsigned int guid:4; /* index into guid table */ } __attribute__ ((packed)); struct squashfs_ipc_inode_header_1 { unsigned int inode_type:4; unsigned int mode:12; /* protection */ unsigned int uid:4; /* index into uid table */ unsigned int guid:4; /* index into guid table */ unsigned int type:4; unsigned int offset:4; } __attribute__ ((packed)); struct squashfs_dev_inode_header_1 { unsigned int inode_type:4; unsigned int mode:12; /* protection */ unsigned int uid:4; /* index into uid table */ unsigned int guid:4; /* index into guid table */ unsigned short rdev; } __attribute__ ((packed)); struct squashfs_symlink_inode_header_1 { unsigned int inode_type:4; unsigned int mode:12; /* protection */ unsigned int uid:4; /* index into uid table */ unsigned int guid:4; /* index into guid table */ unsigned short symlink_size; char symlink[0]; } __attribute__ ((packed)); struct squashfs_reg_inode_header_1 { unsigned int inode_type:4; unsigned int mode:12; /* protection */ unsigned int uid:4; /* index into uid table */ unsigned int guid:4; /* index into guid table */ int mtime; unsigned int start_block; unsigned int file_size:32; unsigned short block_list[0]; } __attribute__ ((packed)); struct squashfs_dir_inode_header_1 { unsigned int inode_type:4; unsigned int mode:12; /* protection */ unsigned int uid:4; /* index into uid table */ unsigned int guid:4; /* index into guid table */ unsigned int file_size:19; unsigned int offset:13; int mtime; unsigned int start_block:24; } __attribute__ ((packed)); union squashfs_inode_header_1 { struct squashfs_base_inode_header_1 base; struct squashfs_dev_inode_header_1 dev; struct squashfs_symlink_inode_header_1 symlink; struct squashfs_reg_inode_header_1 reg; struct squashfs_dir_inode_header_1 dir; struct squashfs_ipc_inode_header_1 ipc; }; typedef struct squashfs_dir_index_1 squashfs_dir_index_1; typedef struct squashfs_base_inode_header_1 squashfs_base_inode_header_1; typedef struct squashfs_ipc_inode_header_1 squashfs_ipc_inode_header_1; typedef struct squashfs_dev_inode_header_1 squashfs_dev_inode_header_1; typedef struct squashfs_symlink_inode_header_1 squashfs_symlink_inode_header_1; typedef struct squashfs_reg_inode_header_1 squashfs_reg_inode_header_1; typedef struct squashfs_dir_inode_header_1 squashfs_dir_inode_header_1; #define SQUASHFS_SWAP_BASE_INODE_CORE_1(s, d, n) \ SQUASHFS_MEMSET(s, d, n);\ SQUASHFS_SWAP((s)->inode_type, d, 0, 4);\ SQUASHFS_SWAP((s)->mode, d, 4, 12);\ SQUASHFS_SWAP((s)->uid, d, 16, 4);\ SQUASHFS_SWAP((s)->guid, d, 20, 4); #define SQUASHFS_SWAP_BASE_INODE_HEADER_1(s, d, n) {\ SQUASHFS_SWAP_START\ SQUASHFS_SWAP_BASE_INODE_CORE_1(s, d, n)\ } #define SQUASHFS_SWAP_IPC_INODE_HEADER_1(s, d) {\ SQUASHFS_SWAP_START\ SQUASHFS_SWAP_BASE_INODE_CORE_1(s, d, \ sizeof(struct squashfs_ipc_inode_header_1));\ SQUASHFS_SWAP((s)->type, d, 24, 4);\ SQUASHFS_SWAP((s)->offset, d, 28, 4);\ } #define SQUASHFS_SWAP_DEV_INODE_HEADER_1(s, d) {\ SQUASHFS_SWAP_START\ SQUASHFS_SWAP_BASE_INODE_CORE_1(s, d, \ sizeof(struct squashfs_dev_inode_header_1));\ SQUASHFS_SWAP((s)->rdev, d, 24, 16);\ } #define SQUASHFS_SWAP_SYMLINK_INODE_HEADER_1(s, d) {\ SQUASHFS_SWAP_START\ SQUASHFS_SWAP_BASE_INODE_CORE_1(s, d, \ sizeof(struct squashfs_symlink_inode_header_1));\ SQUASHFS_SWAP((s)->symlink_size, d, 24, 16);\ } #define SQUASHFS_SWAP_REG_INODE_HEADER_1(s, d) {\ SQUASHFS_SWAP_START\ SQUASHFS_SWAP_BASE_INODE_CORE_1(s, d, \ sizeof(struct squashfs_reg_inode_header_1));\ SQUASHFS_SWAP((s)->mtime, d, 24, 32);\ SQUASHFS_SWAP((s)->start_block, d, 56, 32);\ SQUASHFS_SWAP((s)->file_size, d, 88, 32);\ } #define SQUASHFS_SWAP_DIR_INODE_HEADER_1(s, d) {\ SQUASHFS_SWAP_START\ SQUASHFS_SWAP_BASE_INODE_CORE_1(s, d, \ sizeof(struct squashfs_dir_inode_header_1));\ SQUASHFS_SWAP((s)->file_size, d, 24, 19);\ SQUASHFS_SWAP((s)->offset, d, 43, 13);\ SQUASHFS_SWAP((s)->mtime, d, 56, 32);\ SQUASHFS_SWAP((s)->start_block, d, 88, 24);\ } /* * definitions for structures on disk - layout 2.x */ struct squashfs_dir_index_2 { unsigned int index:27; unsigned int start_block:29; unsigned char size; unsigned char name[0]; } __attribute__ ((packed)); struct squashfs_base_inode_header_2 { unsigned int inode_type:4; unsigned int mode:12; /* protection */ unsigned int uid:8; /* index into uid table */ unsigned int guid:8; /* index into guid table */ } __attribute__ ((packed)); struct squashfs_ipc_inode_header_2 { unsigned int inode_type:4; unsigned int mode:12; /* protection */ unsigned int uid:8; /* index into uid table */ unsigned int guid:8; /* index into guid table */ } __attribute__ ((packed)); struct squashfs_dev_inode_header_2 { unsigned int inode_type:4; unsigned int mode:12; /* protection */ unsigned int uid:8; /* index into uid table */ unsigned int guid:8; /* index into guid table */ unsigned short rdev; } __attribute__ ((packed)); struct squashfs_symlink_inode_header_2 { unsigned int inode_type:4; unsigned int mode:12; /* protection */ unsigned int uid:8; /* index into uid table */ unsigned int guid:8; /* index into guid table */ unsigned short symlink_size; char symlink[0]; } __attribute__ ((packed)); struct squashfs_reg_inode_header_2 { unsigned int inode_type:4; unsigned int mode:12; /* protection */ unsigned int uid:8; /* index into uid table */ unsigned int guid:8; /* index into guid table */ int mtime; unsigned int start_block; unsigned int fragment; unsigned int offset; unsigned int file_size:32; unsigned short block_list[0]; } __attribute__ ((packed)); struct squashfs_dir_inode_header_2 { unsigned int inode_type:4; unsigned int mode:12; /* protection */ unsigned int uid:8; /* index into uid table */ unsigned int guid:8; /* index into guid table */ unsigned int file_size:19; unsigned int offset:13; int mtime; unsigned int start_block:24; } __attribute__ ((packed)); struct squashfs_ldir_inode_header_2 { unsigned int inode_type:4; unsigned int mode:12; /* protection */ unsigned int uid:8; /* index into uid table */ unsigned int guid:8; /* index into guid table */ unsigned int file_size:27; unsigned int offset:13; int mtime; unsigned int start_block:24; unsigned int i_count:16; struct squashfs_dir_index_2 index[0]; } __attribute__ ((packed)); union squashfs_inode_header_2 { struct squashfs_base_inode_header_2 base; struct squashfs_dev_inode_header_2 dev; struct squashfs_symlink_inode_header_2 symlink; struct squashfs_reg_inode_header_2 reg; struct squashfs_dir_inode_header_2 dir; struct squashfs_ldir_inode_header_2 ldir; struct squashfs_ipc_inode_header_2 ipc; }; struct squashfs_dir_header_2 { unsigned int count:8; unsigned int start_block:24; } __attribute__ ((packed)); struct squashfs_dir_entry_2 { unsigned int offset:13; unsigned int type:3; unsigned int size:8; char name[0]; } __attribute__ ((packed)); struct squashfs_fragment_entry_2 { unsigned int start_block; unsigned int size; } __attribute__ ((packed)); typedef struct squashfs_dir_index_2 squashfs_dir_index_2; typedef struct squashfs_base_inode_header_2 squashfs_base_inode_header_2; typedef struct squashfs_ipc_inode_header_2 squashfs_ipc_inode_header_2; typedef struct squashfs_dev_inode_header_2 squashfs_dev_inode_header_2; typedef struct squashfs_symlink_inode_header_2 squashfs_symlink_inode_header_2; typedef struct squashfs_reg_inode_header_2 squashfs_reg_inode_header_2; typedef struct squashfs_lreg_inode_header_2 squashfs_lreg_inode_header_2; typedef struct squashfs_dir_inode_header_2 squashfs_dir_inode_header_2; typedef struct squashfs_ldir_inode_header_2 squashfs_ldir_inode_header_2; typedef struct squashfs_dir_entry_2 squashfs_dir_entry_2; typedef struct squashfs_dir_header_2 squashfs_dir_header_2; typedef struct squashfs_fragment_entry_2 squashfs_fragment_entry_2; #define SQUASHFS_SWAP_BASE_INODE_CORE_2(s, d, n)\ SQUASHFS_MEMSET(s, d, n);\ SQUASHFS_SWAP((s)->inode_type, d, 0, 4);\ SQUASHFS_SWAP((s)->mode, d, 4, 12);\ SQUASHFS_SWAP((s)->uid, d, 16, 8);\ SQUASHFS_SWAP((s)->guid, d, 24, 8);\ #define SQUASHFS_SWAP_BASE_INODE_HEADER_2(s, d, n) {\ SQUASHFS_SWAP_START\ SQUASHFS_SWAP_BASE_INODE_CORE_2(s, d, n)\ } #define SQUASHFS_SWAP_IPC_INODE_HEADER_2(s, d) \ SQUASHFS_SWAP_BASE_INODE_HEADER_2(s, d, sizeof(struct squashfs_ipc_inode_header_2)) #define SQUASHFS_SWAP_DEV_INODE_HEADER_2(s, d) {\ SQUASHFS_SWAP_START\ SQUASHFS_SWAP_BASE_INODE_CORE_2(s, d, \ sizeof(struct squashfs_dev_inode_header_2)); \ SQUASHFS_SWAP((s)->rdev, d, 32, 16);\ } #define SQUASHFS_SWAP_SYMLINK_INODE_HEADER_2(s, d) {\ SQUASHFS_SWAP_START\ SQUASHFS_SWAP_BASE_INODE_CORE_2(s, d, \ sizeof(struct squashfs_symlink_inode_header_2));\ SQUASHFS_SWAP((s)->symlink_size, d, 32, 16);\ } #define SQUASHFS_SWAP_REG_INODE_HEADER_2(s, d) {\ SQUASHFS_SWAP_START\ SQUASHFS_SWAP_BASE_INODE_CORE_2(s, d, \ sizeof(struct squashfs_reg_inode_header_2));\ SQUASHFS_SWAP((s)->mtime, d, 32, 32);\ SQUASHFS_SWAP((s)->start_block, d, 64, 32);\ SQUASHFS_SWAP((s)->fragment, d, 96, 32);\ SQUASHFS_SWAP((s)->offset, d, 128, 32);\ SQUASHFS_SWAP((s)->file_size, d, 160, 32);\ } #define SQUASHFS_SWAP_DIR_INODE_HEADER_2(s, d) {\ SQUASHFS_SWAP_START\ SQUASHFS_SWAP_BASE_INODE_CORE_2(s, d, \ sizeof(struct squashfs_dir_inode_header_2));\ SQUASHFS_SWAP((s)->file_size, d, 32, 19);\ SQUASHFS_SWAP((s)->offset, d, 51, 13);\ SQUASHFS_SWAP((s)->mtime, d, 64, 32);\ SQUASHFS_SWAP((s)->start_block, d, 96, 24);\ } #define SQUASHFS_SWAP_LDIR_INODE_HEADER_2(s, d) {\ SQUASHFS_SWAP_START\ SQUASHFS_SWAP_BASE_INODE_CORE_2(s, d, \ sizeof(struct squashfs_ldir_inode_header_2));\ SQUASHFS_SWAP((s)->file_size, d, 32, 27);\ SQUASHFS_SWAP((s)->offset, d, 59, 13);\ SQUASHFS_SWAP((s)->mtime, d, 72, 32);\ SQUASHFS_SWAP((s)->start_block, d, 104, 24);\ SQUASHFS_SWAP((s)->i_count, d, 128, 16);\ } #define SQUASHFS_SWAP_DIR_INDEX_2(s, d) {\ SQUASHFS_SWAP_START\ SQUASHFS_MEMSET(s, d, sizeof(struct squashfs_dir_index_2));\ SQUASHFS_SWAP((s)->index, d, 0, 27);\ SQUASHFS_SWAP((s)->start_block, d, 27, 29);\ SQUASHFS_SWAP((s)->size, d, 56, 8);\ } #define SQUASHFS_SWAP_DIR_HEADER_2(s, d) {\ SQUASHFS_SWAP_START\ SQUASHFS_MEMSET(s, d, sizeof(struct squashfs_dir_header_2));\ SQUASHFS_SWAP((s)->count, d, 0, 8);\ SQUASHFS_SWAP((s)->start_block, d, 8, 24);\ } #define SQUASHFS_SWAP_DIR_ENTRY_2(s, d) {\ SQUASHFS_SWAP_START\ SQUASHFS_MEMSET(s, d, sizeof(struct squashfs_dir_entry_2));\ SQUASHFS_SWAP((s)->offset, d, 0, 13);\ SQUASHFS_SWAP((s)->type, d, 13, 3);\ SQUASHFS_SWAP((s)->size, d, 16, 8);\ } #define SQUASHFS_SWAP_FRAGMENT_ENTRY_2(s, d) {\ SQUASHFS_SWAP_START\ SQUASHFS_MEMSET(s, d, sizeof(struct squashfs_fragment_entry_2));\ SQUASHFS_SWAP((s)->start_block, d, 0, 32);\ SQUASHFS_SWAP((s)->size, d, 32, 32);\ } #define SQUASHFS_SWAP_FRAGMENT_INDEXES_2(s, d, n) SQUASHFS_SWAP_INTS_3(s, d, n) /* fragment and fragment table defines */ #define SQUASHFS_FRAGMENT_BYTES_2(A) ((A) * sizeof(struct squashfs_fragment_entry_2)) #define SQUASHFS_FRAGMENT_INDEX_2(A) (SQUASHFS_FRAGMENT_BYTES_2(A) / \ SQUASHFS_METADATA_SIZE) #define SQUASHFS_FRAGMENT_INDEX_OFFSET_2(A) (SQUASHFS_FRAGMENT_BYTES_2(A) % \ SQUASHFS_METADATA_SIZE) #define SQUASHFS_FRAGMENT_INDEXES_2(A) ((SQUASHFS_FRAGMENT_BYTES_2(A) + \ SQUASHFS_METADATA_SIZE - 1) / \ SQUASHFS_METADATA_SIZE) #define SQUASHFS_FRAGMENT_INDEX_BYTES_2(A) (SQUASHFS_FRAGMENT_INDEXES_2(A) *\ sizeof(int)) /* * macros used to swap each structure entry, taking into account * bitfields and different bitfield placing conventions on differing architectures */ #if __BYTE_ORDER == __BIG_ENDIAN /* convert from big endian to little endian */ #define SQUASHFS_SWAP(value, p, pos, tbits) _SQUASHFS_SWAP(value, p, pos, tbits, b_pos) #else /* convert from little endian to big endian */ #define SQUASHFS_SWAP(value, p, pos, tbits) _SQUASHFS_SWAP(value, p, pos, tbits, 64 - tbits - b_pos) #endif #define _SQUASHFS_SWAP(value, p, pos, tbits, SHIFT) {\ b_pos = pos % 8;\ val = 0;\ s = (unsigned char *)p + (pos / 8);\ d = ((unsigned char *) &val) + 7;\ for(bits = 0; bits < (tbits + b_pos); bits += 8) \ *d-- = *s++;\ value = (val >> (SHIFT))/* & ((1 << tbits) - 1)*/;\ } #define SQUASHFS_MEMSET(s, d, n) memset(s, 0, n); #endif squashfs4.3/squashfs-tools/xz_wrapper.c0000644000175000017500000003265612306776317020414 0ustar phillipphillip/* * Copyright (c) 2010, 2011, 2012, 2013 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * xz_wrapper.c * * Support for XZ (LZMA2) compression using XZ Utils liblzma * http://tukaani.org/xz/ */ #include #include #include #include #include "squashfs_fs.h" #include "xz_wrapper.h" #include "compressor.h" static struct bcj bcj[] = { { "x86", LZMA_FILTER_X86, 0 }, { "powerpc", LZMA_FILTER_POWERPC, 0 }, { "ia64", LZMA_FILTER_IA64, 0 }, { "arm", LZMA_FILTER_ARM, 0 }, { "armthumb", LZMA_FILTER_ARMTHUMB, 0 }, { "sparc", LZMA_FILTER_SPARC, 0 }, { NULL, LZMA_VLI_UNKNOWN, 0 } }; static int filter_count = 1; static int dictionary_size = 0; static float dictionary_percent = 0; /* * This function is called by the options parsing code in mksquashfs.c * to parse any -X compressor option. * * Two specific options are supported: * -Xbcj * -Xdict-size * * This function returns: * >=0 (number of additional args parsed) on success * -1 if the option was unrecognised, or * -2 if the option was recognised, but otherwise bad in * some way (e.g. invalid parameter) * * Note: this function sets internal compressor state, but does not * pass back the results of the parsing other than success/failure. * The xz_dump_options() function is called later to get the options in * a format suitable for writing to the filesystem. */ static int xz_options(char *argv[], int argc) { int i; char *name; if(strcmp(argv[0], "-Xbcj") == 0) { if(argc < 2) { fprintf(stderr, "xz: -Xbcj missing filter\n"); goto failed; } name = argv[1]; while(name[0] != '\0') { for(i = 0; bcj[i].name; i++) { int n = strlen(bcj[i].name); if((strncmp(name, bcj[i].name, n) == 0) && (name[n] == '\0' || name[n] == ',')) { if(bcj[i].selected == 0) { bcj[i].selected = 1; filter_count++; } name += name[n] == ',' ? n + 1 : n; break; } } if(bcj[i].name == NULL) { fprintf(stderr, "xz: -Xbcj unrecognised " "filter\n"); goto failed; } } return 1; } else if(strcmp(argv[0], "-Xdict-size") == 0) { char *b; float size; if(argc < 2) { fprintf(stderr, "xz: -Xdict-size missing dict-size\n"); goto failed; } size = strtof(argv[1], &b); if(*b == '%') { if(size <= 0 || size > 100) { fprintf(stderr, "xz: -Xdict-size percentage " "should be 0 < dict-size <= 100\n"); goto failed; } dictionary_percent = size; dictionary_size = 0; } else { if((float) ((int) size) != size) { fprintf(stderr, "xz: -Xdict-size can't be " "fractional unless a percentage of the" " block size\n"); goto failed; } dictionary_percent = 0; dictionary_size = (int) size; if(*b == 'k' || *b == 'K') dictionary_size *= 1024; else if(*b == 'm' || *b == 'M') dictionary_size *= 1024 * 1024; else if(*b != '\0') { fprintf(stderr, "xz: -Xdict-size invalid " "dict-size\n"); goto failed; } } return 1; } return -1; failed: return -2; } /* * This function is called after all options have been parsed. * It is used to do post-processing on the compressor options using * values that were not expected to be known at option parse time. * * In this case block_size may not be known until after -Xdict-size has * been processed (in the case where -b is specified after -Xdict-size) * * This function returns 0 on successful post processing, or * -1 on error */ static int xz_options_post(int block_size) { /* * if -Xdict-size has been specified use this to compute the datablock * dictionary size */ if(dictionary_size || dictionary_percent) { int n; if(dictionary_size) { if(dictionary_size > block_size) { fprintf(stderr, "xz: -Xdict-size is larger than" " block_size\n"); goto failed; } } else dictionary_size = block_size * dictionary_percent / 100; if(dictionary_size < 8192) { fprintf(stderr, "xz: -Xdict-size should be 8192 bytes " "or larger\n"); goto failed; } /* * dictionary_size must be storable in xz header as either * 2^n or as 2^n+2^(n+1) */ n = ffs(dictionary_size) - 1; if(dictionary_size != (1 << n) && dictionary_size != ((1 << n) + (1 << (n + 1)))) { fprintf(stderr, "xz: -Xdict-size is an unsupported " "value, dict-size must be storable in xz " "header\n"); fprintf(stderr, "as either 2^n or as 2^n+2^(n+1). " "Example dict-sizes are 75%%, 50%%, 37.5%%, " "25%%,\n"); fprintf(stderr, "or 32K, 16K, 8K etc.\n"); goto failed; } } else /* No -Xdict-size specified, use defaults */ dictionary_size = block_size; return 0; failed: return -1; } /* * This function is called by mksquashfs to dump the parsed * compressor options in a format suitable for writing to the * compressor options field in the filesystem (stored immediately * after the superblock). * * This function returns a pointer to the compression options structure * to be stored (and the size), or NULL if there are no compression * options */ static void *xz_dump_options(int block_size, int *size) { static struct comp_opts comp_opts; int flags = 0, i; /* * don't store compressor specific options in file system if the * default options are being used - no compressor options in the * file system means the default options are always assumed * * Defaults are: * metadata dictionary size: SQUASHFS_METADATA_SIZE * datablock dictionary size: block_size * 1 filter */ if(dictionary_size == block_size && filter_count == 1) return NULL; for(i = 0; bcj[i].name; i++) flags |= bcj[i].selected << i; comp_opts.dictionary_size = dictionary_size; comp_opts.flags = flags; SQUASHFS_INSWAP_COMP_OPTS(&comp_opts); *size = sizeof(comp_opts); return &comp_opts; } /* * This function is a helper specifically for the append mode of * mksquashfs. Its purpose is to set the internal compressor state * to the stored compressor options in the passed compressor options * structure. * * In effect this function sets up the compressor options * to the same state they were when the filesystem was originally * generated, this is to ensure on appending, the compressor uses * the same compression options that were used to generate the * original filesystem. * * Note, even if there are no compressor options, this function is still * called with an empty compressor structure (size == 0), to explicitly * set the default options, this is to ensure any user supplied * -X options on the appending mksquashfs command line are over-ridden * * This function returns 0 on sucessful extraction of options, and * -1 on error */ static int xz_extract_options(int block_size, void *buffer, int size) { struct comp_opts *comp_opts = buffer; int flags, i, n; if(size == 0) { /* set defaults */ dictionary_size = block_size; flags = 0; } else { /* check passed comp opts struct is of the correct length */ if(size != sizeof(struct comp_opts)) goto failed; SQUASHFS_INSWAP_COMP_OPTS(comp_opts); dictionary_size = comp_opts->dictionary_size; flags = comp_opts->flags; /* * check that the dictionary size seems correct - the dictionary * size should 2^n or 2^n+2^(n+1) */ n = ffs(dictionary_size) - 1; if(dictionary_size != (1 << n) && dictionary_size != ((1 << n) + (1 << (n + 1)))) goto failed; } filter_count = 1; for(i = 0; bcj[i].name; i++) { if((flags >> i) & 1) { bcj[i].selected = 1; filter_count ++; } else bcj[i].selected = 0; } return 0; failed: fprintf(stderr, "xz: error reading stored compressor options from " "filesystem!\n"); return -1; } void xz_display_options(void *buffer, int size) { struct comp_opts *comp_opts = buffer; int dictionary_size, flags, printed; int i, n; /* check passed comp opts struct is of the correct length */ if(size != sizeof(struct comp_opts)) goto failed; SQUASHFS_INSWAP_COMP_OPTS(comp_opts); dictionary_size = comp_opts->dictionary_size; flags = comp_opts->flags; /* * check that the dictionary size seems correct - the dictionary * size should 2^n or 2^n+2^(n+1) */ n = ffs(dictionary_size) - 1; if(dictionary_size != (1 << n) && dictionary_size != ((1 << n) + (1 << (n + 1)))) goto failed; printf("\tDictionary size %d\n", dictionary_size); printed = 0; for(i = 0; bcj[i].name; i++) { if((flags >> i) & 1) { if(printed) printf(", "); else printf("\tFilters selected: "); printf("%s", bcj[i].name); printed = 1; } } if(!printed) printf("\tNo filters specified\n"); else printf("\n"); return; failed: fprintf(stderr, "xz: error reading stored compressor options from " "filesystem!\n"); } /* * This function is called by mksquashfs to initialise the * compressor, before compress() is called. * * This function returns 0 on success, and * -1 on error */ static int xz_init(void **strm, int block_size, int datablock) { int i, j, filters = datablock ? filter_count : 1; struct filter *filter = malloc(filters * sizeof(struct filter)); struct xz_stream *stream; if(filter == NULL) goto failed; stream = *strm = malloc(sizeof(struct xz_stream)); if(stream == NULL) goto failed2; stream->filter = filter; stream->filters = filters; memset(filter, 0, filters * sizeof(struct filter)); stream->dictionary_size = datablock ? dictionary_size : SQUASHFS_METADATA_SIZE; filter[0].filter[0].id = LZMA_FILTER_LZMA2; filter[0].filter[0].options = &stream->opt; filter[0].filter[1].id = LZMA_VLI_UNKNOWN; for(i = 0, j = 1; datablock && bcj[i].name; i++) { if(bcj[i].selected) { filter[j].buffer = malloc(block_size); if(filter[j].buffer == NULL) goto failed3; filter[j].filter[0].id = bcj[i].id; filter[j].filter[1].id = LZMA_FILTER_LZMA2; filter[j].filter[1].options = &stream->opt; filter[j].filter[2].id = LZMA_VLI_UNKNOWN; j++; } } return 0; failed3: for(i = 1; i < filters; i++) free(filter[i].buffer); free(stream); failed2: free(filter); failed: return -1; } static int xz_compress(void *strm, void *dest, void *src, int size, int block_size, int *error) { int i; lzma_ret res = 0; struct xz_stream *stream = strm; struct filter *selected = NULL; stream->filter[0].buffer = dest; for(i = 0; i < stream->filters; i++) { struct filter *filter = &stream->filter[i]; if(lzma_lzma_preset(&stream->opt, LZMA_PRESET_DEFAULT)) goto failed; stream->opt.dict_size = stream->dictionary_size; filter->length = 0; res = lzma_stream_buffer_encode(filter->filter, LZMA_CHECK_CRC32, NULL, src, size, filter->buffer, &filter->length, block_size); if(res == LZMA_OK) { if(!selected || selected->length > filter->length) selected = filter; } else if(res != LZMA_BUF_ERROR) goto failed; } if(!selected) /* * Output buffer overflow. Return out of buffer space */ return 0; if(selected->buffer != dest) memcpy(dest, selected->buffer, selected->length); return (int) selected->length; failed: /* * All other errors return failure, with the compressor * specific error code in *error */ *error = res; return -1; } static int xz_uncompress(void *dest, void *src, int size, int outsize, int *error) { size_t src_pos = 0; size_t dest_pos = 0; uint64_t memlimit = MEMLIMIT; lzma_ret res = lzma_stream_buffer_decode(&memlimit, 0, NULL, src, &src_pos, size, dest, &dest_pos, outsize); if(res == LZMA_OK && size == (int) src_pos) return (int) dest_pos; else { *error = res; return -1; } } void xz_usage() { fprintf(stderr, "\t -Xbcj filter1,filter2,...,filterN\n"); fprintf(stderr, "\t\tCompress using filter1,filter2,...,filterN in"); fprintf(stderr, " turn\n\t\t(in addition to no filter), and choose"); fprintf(stderr, " the best compression.\n"); fprintf(stderr, "\t\tAvailable filters: x86, arm, armthumb,"); fprintf(stderr, " powerpc, sparc, ia64\n"); fprintf(stderr, "\t -Xdict-size \n"); fprintf(stderr, "\t\tUse as the XZ dictionary size. The"); fprintf(stderr, " dictionary size\n\t\tcan be specified as a"); fprintf(stderr, " percentage of the block size, or as an\n\t\t"); fprintf(stderr, "absolute value. The dictionary size must be less"); fprintf(stderr, " than or equal\n\t\tto the block size and 8192 bytes"); fprintf(stderr, " or larger. It must also be\n\t\tstorable in the xz"); fprintf(stderr, " header as either 2^n or as 2^n+2^(n+1).\n\t\t"); fprintf(stderr, "Example dict-sizes are 75%%, 50%%, 37.5%%, 25%%, or"); fprintf(stderr, " 32K, 16K, 8K\n\t\tetc.\n"); } struct compressor xz_comp_ops = { .init = xz_init, .compress = xz_compress, .uncompress = xz_uncompress, .options = xz_options, .options_post = xz_options_post, .dump_options = xz_dump_options, .extract_options = xz_extract_options, .display_options = xz_display_options, .usage = xz_usage, .id = XZ_COMPRESSION, .name = "xz", .supported = 1 }; squashfs4.3/squashfs-tools/info.c0000644000175000017500000001023112333330365017114 0ustar phillipphillip/* * Create a squashfs filesystem. This is a highly compressed read only * filesystem. * * Copyright (c) 2013, 2014 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * info.c */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "squashfs_fs.h" #include "mksquashfs.h" #include "error.h" #include "progressbar.h" #include "caches-queues-lists.h" static int silent = 0; static struct dir_ent *ent = NULL; pthread_t info_thread; void disable_info() { ent = NULL; } void update_info(struct dir_ent *dir_ent) { ent = dir_ent; } void print_filename() { struct dir_ent *dir_ent = ent; if(dir_ent == NULL) return; if(dir_ent->our_dir->subpath[0] != '\0') INFO("%s/%s\n", dir_ent->our_dir->subpath, dir_ent->name); else INFO("/%s\n", dir_ent->name); } void dump_state() { disable_progress_bar(); printf("Queue and Cache status dump\n"); printf("===========================\n"); printf("file buffer queue (reader thread -> deflate thread(s))\n"); dump_queue(to_deflate); printf("uncompressed fragment queue (reader thread -> fragment" " thread(s))\n"); dump_queue(to_process_frag); printf("processed fragment queue (fragment thread(s) -> main" " thread)\n"); dump_seq_queue(to_main, 1); printf("compressed block queue (deflate thread(s) -> main thread)\n"); dump_seq_queue(to_main, 0); printf("uncompressed packed fragment queue (main thread -> fragment" " deflate thread(s))\n"); dump_queue(to_frag); printf("locked frag queue (compressed frags waiting while multi-block" " file is written)\n"); dump_queue(locked_fragment); printf("compressed block queue (main & fragment deflate threads(s) ->" " writer thread)\n"); dump_queue(to_writer); printf("read cache (uncompressed blocks read by reader thread)\n"); dump_cache(reader_buffer); printf("block write cache (compressed blocks waiting for the writer" " thread)\n"); dump_cache(bwriter_buffer); printf("fragment write cache (compressed fragments waiting for the" " writer thread)\n"); dump_cache(fwriter_buffer); printf("fragment cache (frags waiting to be compressed by fragment" " deflate thread(s))\n"); dump_cache(fragment_buffer); printf("fragment reserve cache (avoids pipeline stall if frag cache" " full in dup check)\n"); dump_cache(reserve_cache); enable_progress_bar(); } void *info_thrd(void *arg) { sigset_t sigmask; struct timespec timespec = { .tv_sec = 1, .tv_nsec = 0 }; int sig, waiting = 0; sigemptyset(&sigmask); sigaddset(&sigmask, SIGQUIT); sigaddset(&sigmask, SIGHUP); while(1) { if(waiting) sig = sigtimedwait(&sigmask, NULL, ×pec); else sig = sigwaitinfo(&sigmask, NULL); if(sig == -1) { switch(errno) { case EAGAIN: /* interval timed out */ waiting = 0; /* FALLTHROUGH */ case EINTR: /* if waiting, the wait will be longer, but that's OK */ continue; default: BAD_ERROR("sigtimedwait/sigwaitinfo failed " "because %s\n", strerror(errno)); } } if(sig == SIGQUIT && !waiting) { print_filename(); /* set one second interval period, if ^\ received within then, dump queue and cache status */ waiting = 1; } else dump_state(); } } void init_info() { pthread_create(&info_thread, NULL, info_thrd, NULL); } squashfs4.3/squashfs-tools/unsquashfs_xattr.c0000644000175000017500000001024212306776317021620 0ustar phillipphillip/* * Unsquash a squashfs filesystem. This is a highly compressed read only * filesystem. * * Copyright (c) 2010, 2012 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * unsquashfs_xattr.c */ #include "unsquashfs.h" #include "xattr.h" #include #define NOSPACE_MAX 10 extern int root_process; extern int user_xattrs; void write_xattr(char *pathname, unsigned int xattr) { unsigned int count; struct xattr_list *xattr_list; int i; static int nonsuper_error = FALSE; static int ignore_xattrs = FALSE; static int nospace_error = 0; if(ignore_xattrs || xattr == SQUASHFS_INVALID_XATTR || sBlk.s.xattr_id_table_start == SQUASHFS_INVALID_BLK) return; xattr_list = get_xattr(xattr, &count, 1); if(xattr_list == NULL) { ERROR("Failed to read xattrs for file %s\n", pathname); return; } for(i = 0; i < count; i++) { int prefix = xattr_list[i].type & SQUASHFS_XATTR_PREFIX_MASK; if(user_xattrs && prefix != SQUASHFS_XATTR_USER) continue; if(root_process || prefix == SQUASHFS_XATTR_USER) { int res = lsetxattr(pathname, xattr_list[i].full_name, xattr_list[i].value, xattr_list[i].vsize, 0); if(res == -1) { if(errno == ENOTSUP) { /* * If the destination filesystem cannot * suppport xattrs, print error, and * disable xattr output as this error is * unlikely to go away, and printing * screenfulls of the same error message * is rather annoying */ ERROR("write_xattr: failed to write " "xattr %s for file %s because " "extended attributes are not " "supported by the destination " "filesystem\n", xattr_list[i].full_name, pathname); ERROR("Ignoring xattrs in " "filesystem\n"); ERROR("To avoid this error message, " "specify -no-xattrs\n"); ignore_xattrs = TRUE; } else if((errno == ENOSPC || errno == EDQUOT) && nospace_error < NOSPACE_MAX) { /* * Many filesystems like ext2/3/4 have * limits on the amount of xattr * data that can be stored per file * (typically one block or 4K), so * we shouldn't disable xattr ouput, * as the error may be restriced to one * file only. If we get a lot of these * then suppress the error messsage */ ERROR("write_xattr: failed to write " "xattr %s for file %s because " "no extended attribute space " "remaining (per file or " "filesystem limit)\n", xattr_list[i].full_name, pathname); if(++ nospace_error == NOSPACE_MAX) ERROR("%d of these errors " "printed, further error " "messages of this type " "are suppressed!\n", NOSPACE_MAX); } else ERROR("write_xattr: failed to write " "xattr %s for file %s because " "%s\n", xattr_list[i].full_name, pathname, strerror(errno)); } } else if(nonsuper_error == FALSE) { /* * if extract user xattrs only then * error message is suppressed, if not * print error, and then suppress further error * messages to avoid possible screenfulls of the * same error message! */ ERROR("write_xattr: could not write xattr %s " "for file %s because you're not " "superuser!\n", xattr_list[i].full_name, pathname); ERROR("write_xattr: to avoid this error message, either" " specify -user-xattrs, -no-xattrs, or run as " "superuser!\n"); ERROR("Further error messages of this type are " "suppressed!\n"); nonsuper_error = TRUE; } } free_xattr(xattr_list, count); } squashfs4.3/squashfs-tools/error.h0000644000175000017500000000416312333330365017326 0ustar phillipphillip#ifndef ERROR_H #define ERROR_H /* * Create a squashfs filesystem. This is a highly compressed read only * filesystem. * * Copyright (c) 2012, 2013, 2014 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * error.h */ extern int exit_on_error; extern void prep_exit(); extern void progressbar_error(char *fmt, ...); extern void progressbar_info(char *fmt, ...); #ifdef SQUASHFS_TRACE #define TRACE(s, args...) \ do { \ progressbar_info("squashfs: "s, ## args);\ } while(0) #else #define TRACE(s, args...) #endif #define INFO(s, args...) \ do {\ if(!silent)\ progressbar_info(s, ## args);\ } while(0) #define ERROR(s, args...) \ do {\ progressbar_error(s, ## args); \ } while(0) #define ERROR_START(s, args...) \ do { \ disable_progress_bar(); \ fprintf(stderr, s, ## args); \ } while(0) #define ERROR_EXIT(s, args...) \ do {\ if (exit_on_error) { \ fprintf(stderr, "\n"); \ EXIT_MKSQUASHFS(); \ } else { \ fprintf(stderr, s, ## args); \ enable_progress_bar(); \ } \ } while(0) #define EXIT_MKSQUASHFS() \ do {\ prep_exit();\ exit(1);\ } while(0) #define BAD_ERROR(s, args...) \ do {\ progressbar_error("FATAL ERROR:" s, ##args); \ EXIT_MKSQUASHFS();\ } while(0) #define EXIT_UNSQUASH(s, args...) BAD_ERROR(s, ##args) #define MEM_ERROR() \ do {\ progressbar_error("FATAL ERROR: Out of memory (%s)\n", \ __func__); \ EXIT_MKSQUASHFS();\ } while(0) #endif squashfs4.3/squashfs-tools/unsquash-4.c0000644000175000017500000002601412306776317020212 0ustar phillipphillip/* * Unsquash a squashfs filesystem. This is a highly compressed read only * filesystem. * * Copyright (c) 2009, 2010, 2011, 2012, 2013 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * unsquash-4.c */ #include "unsquashfs.h" #include "squashfs_swap.h" static struct squashfs_fragment_entry *fragment_table; static unsigned int *id_table; int read_fragment_table_4(long long *directory_table_end) { int res, i; int bytes = SQUASHFS_FRAGMENT_BYTES(sBlk.s.fragments); int indexes = SQUASHFS_FRAGMENT_INDEXES(sBlk.s.fragments); long long fragment_table_index[indexes]; TRACE("read_fragment_table: %d fragments, reading %d fragment indexes " "from 0x%llx\n", sBlk.s.fragments, indexes, sBlk.s.fragment_table_start); if(sBlk.s.fragments == 0) { *directory_table_end = sBlk.s.fragment_table_start; return TRUE; } fragment_table = malloc(bytes); if(fragment_table == NULL) EXIT_UNSQUASH("read_fragment_table: failed to allocate " "fragment table\n"); res = read_fs_bytes(fd, sBlk.s.fragment_table_start, SQUASHFS_FRAGMENT_INDEX_BYTES(sBlk.s.fragments), fragment_table_index); if(res == FALSE) { ERROR("read_fragment_table: failed to read fragment table " "index\n"); return FALSE; } SQUASHFS_INSWAP_FRAGMENT_INDEXES(fragment_table_index, indexes); for(i = 0; i < indexes; i++) { int expected = (i + 1) != indexes ? SQUASHFS_METADATA_SIZE : bytes & (SQUASHFS_METADATA_SIZE - 1); int length = read_block(fd, fragment_table_index[i], NULL, expected, ((char *) fragment_table) + (i * SQUASHFS_METADATA_SIZE)); TRACE("Read fragment table block %d, from 0x%llx, length %d\n", i, fragment_table_index[i], length); if(length == FALSE) { ERROR("read_fragment_table: failed to read fragment " "table index\n"); return FALSE; } } for(i = 0; i < sBlk.s.fragments; i++) SQUASHFS_INSWAP_FRAGMENT_ENTRY(&fragment_table[i]); *directory_table_end = fragment_table_index[0]; return TRUE; } void read_fragment_4(unsigned int fragment, long long *start_block, int *size) { TRACE("read_fragment: reading fragment %d\n", fragment); struct squashfs_fragment_entry *fragment_entry; fragment_entry = &fragment_table[fragment]; *start_block = fragment_entry->start_block; *size = fragment_entry->size; } struct inode *read_inode_4(unsigned int start_block, unsigned int offset) { static union squashfs_inode_header header; long long start = sBlk.s.inode_table_start + start_block; int bytes = lookup_entry(inode_table_hash, start); char *block_ptr = inode_table + bytes + offset; static struct inode i; TRACE("read_inode: reading inode [%d:%d]\n", start_block, offset); if(bytes == -1) EXIT_UNSQUASH("read_inode: inode table block %lld not found\n", start); SQUASHFS_SWAP_BASE_INODE_HEADER(block_ptr, &header.base); i.uid = (uid_t) id_table[header.base.uid]; i.gid = (uid_t) id_table[header.base.guid]; i.mode = lookup_type[header.base.inode_type] | header.base.mode; i.type = header.base.inode_type; i.time = header.base.mtime; i.inode_number = header.base.inode_number; switch(header.base.inode_type) { case SQUASHFS_DIR_TYPE: { struct squashfs_dir_inode_header *inode = &header.dir; SQUASHFS_SWAP_DIR_INODE_HEADER(block_ptr, inode); i.data = inode->file_size; i.offset = inode->offset; i.start = inode->start_block; i.xattr = SQUASHFS_INVALID_XATTR; break; } case SQUASHFS_LDIR_TYPE: { struct squashfs_ldir_inode_header *inode = &header.ldir; SQUASHFS_SWAP_LDIR_INODE_HEADER(block_ptr, inode); i.data = inode->file_size; i.offset = inode->offset; i.start = inode->start_block; i.xattr = inode->xattr; break; } case SQUASHFS_FILE_TYPE: { struct squashfs_reg_inode_header *inode = &header.reg; SQUASHFS_SWAP_REG_INODE_HEADER(block_ptr, inode); i.data = inode->file_size; i.frag_bytes = inode->fragment == SQUASHFS_INVALID_FRAG ? 0 : inode->file_size % sBlk.s.block_size; i.fragment = inode->fragment; i.offset = inode->offset; i.blocks = inode->fragment == SQUASHFS_INVALID_FRAG ? (i.data + sBlk.s.block_size - 1) >> sBlk.s.block_log : i.data >> sBlk.s.block_log; i.start = inode->start_block; i.sparse = 0; i.block_ptr = block_ptr + sizeof(*inode); i.xattr = SQUASHFS_INVALID_XATTR; break; } case SQUASHFS_LREG_TYPE: { struct squashfs_lreg_inode_header *inode = &header.lreg; SQUASHFS_SWAP_LREG_INODE_HEADER(block_ptr, inode); i.data = inode->file_size; i.frag_bytes = inode->fragment == SQUASHFS_INVALID_FRAG ? 0 : inode->file_size % sBlk.s.block_size; i.fragment = inode->fragment; i.offset = inode->offset; i.blocks = inode->fragment == SQUASHFS_INVALID_FRAG ? (inode->file_size + sBlk.s.block_size - 1) >> sBlk.s.block_log : inode->file_size >> sBlk.s.block_log; i.start = inode->start_block; i.sparse = inode->sparse != 0; i.block_ptr = block_ptr + sizeof(*inode); i.xattr = inode->xattr; break; } case SQUASHFS_SYMLINK_TYPE: case SQUASHFS_LSYMLINK_TYPE: { struct squashfs_symlink_inode_header *inode = &header.symlink; SQUASHFS_SWAP_SYMLINK_INODE_HEADER(block_ptr, inode); i.symlink = malloc(inode->symlink_size + 1); if(i.symlink == NULL) EXIT_UNSQUASH("read_inode: failed to malloc " "symlink data\n"); strncpy(i.symlink, block_ptr + sizeof(struct squashfs_symlink_inode_header), inode->symlink_size); i.symlink[inode->symlink_size] = '\0'; i.data = inode->symlink_size; if(header.base.inode_type == SQUASHFS_LSYMLINK_TYPE) SQUASHFS_SWAP_INTS(block_ptr + sizeof(struct squashfs_symlink_inode_header) + inode->symlink_size, &i.xattr, 1); else i.xattr = SQUASHFS_INVALID_XATTR; break; } case SQUASHFS_BLKDEV_TYPE: case SQUASHFS_CHRDEV_TYPE: { struct squashfs_dev_inode_header *inode = &header.dev; SQUASHFS_SWAP_DEV_INODE_HEADER(block_ptr, inode); i.data = inode->rdev; i.xattr = SQUASHFS_INVALID_XATTR; break; } case SQUASHFS_LBLKDEV_TYPE: case SQUASHFS_LCHRDEV_TYPE: { struct squashfs_ldev_inode_header *inode = &header.ldev; SQUASHFS_SWAP_LDEV_INODE_HEADER(block_ptr, inode); i.data = inode->rdev; i.xattr = inode->xattr; break; } case SQUASHFS_FIFO_TYPE: case SQUASHFS_SOCKET_TYPE: i.data = 0; i.xattr = SQUASHFS_INVALID_XATTR; break; case SQUASHFS_LFIFO_TYPE: case SQUASHFS_LSOCKET_TYPE: { struct squashfs_lipc_inode_header *inode = &header.lipc; SQUASHFS_SWAP_LIPC_INODE_HEADER(block_ptr, inode); i.data = 0; i.xattr = inode->xattr; break; } default: EXIT_UNSQUASH("Unknown inode type %d in read_inode!\n", header.base.inode_type); } return &i; } struct dir *squashfs_opendir_4(unsigned int block_start, unsigned int offset, struct inode **i) { struct squashfs_dir_header dirh; char buffer[sizeof(struct squashfs_dir_entry) + SQUASHFS_NAME_LEN + 1] __attribute__((aligned)); struct squashfs_dir_entry *dire = (struct squashfs_dir_entry *) buffer; long long start; int bytes; int dir_count, size; struct dir_ent *new_dir; struct dir *dir; TRACE("squashfs_opendir: inode start block %d, offset %d\n", block_start, offset); *i = s_ops.read_inode(block_start, offset); dir = malloc(sizeof(struct dir)); if(dir == NULL) EXIT_UNSQUASH("squashfs_opendir: malloc failed!\n"); dir->dir_count = 0; dir->cur_entry = 0; dir->mode = (*i)->mode; dir->uid = (*i)->uid; dir->guid = (*i)->gid; dir->mtime = (*i)->time; dir->xattr = (*i)->xattr; dir->dirs = NULL; if ((*i)->data == 3) /* * if the directory is empty, skip the unnecessary * lookup_entry, this fixes the corner case with * completely empty filesystems where lookup_entry correctly * returning -1 is incorrectly treated as an error */ return dir; start = sBlk.s.directory_table_start + (*i)->start; bytes = lookup_entry(directory_table_hash, start); if(bytes == -1) EXIT_UNSQUASH("squashfs_opendir: directory block %d not " "found!\n", block_start); bytes += (*i)->offset; size = (*i)->data + bytes - 3; while(bytes < size) { SQUASHFS_SWAP_DIR_HEADER(directory_table + bytes, &dirh); dir_count = dirh.count + 1; TRACE("squashfs_opendir: Read directory header @ byte position " "%d, %d directory entries\n", bytes, dir_count); bytes += sizeof(dirh); /* dir_count should never be larger than 256 */ if(dir_count > 256) goto corrupted; while(dir_count--) { SQUASHFS_SWAP_DIR_ENTRY(directory_table + bytes, dire); bytes += sizeof(*dire); /* size should never be larger than SQUASHFS_NAME_LEN */ if(dire->size > SQUASHFS_NAME_LEN) goto corrupted; memcpy(dire->name, directory_table + bytes, dire->size + 1); dire->name[dire->size + 1] = '\0'; TRACE("squashfs_opendir: directory entry %s, inode " "%d:%d, type %d\n", dire->name, dirh.start_block, dire->offset, dire->type); if((dir->dir_count % DIR_ENT_SIZE) == 0) { new_dir = realloc(dir->dirs, (dir->dir_count + DIR_ENT_SIZE) * sizeof(struct dir_ent)); if(new_dir == NULL) EXIT_UNSQUASH("squashfs_opendir: " "realloc failed!\n"); dir->dirs = new_dir; } strcpy(dir->dirs[dir->dir_count].name, dire->name); dir->dirs[dir->dir_count].start_block = dirh.start_block; dir->dirs[dir->dir_count].offset = dire->offset; dir->dirs[dir->dir_count].type = dire->type; dir->dir_count ++; bytes += dire->size + 1; } } return dir; corrupted: free(dir->dirs); free(dir); return NULL; } int read_uids_guids_4() { int res, i; int bytes = SQUASHFS_ID_BYTES(sBlk.s.no_ids); int indexes = SQUASHFS_ID_BLOCKS(sBlk.s.no_ids); long long id_index_table[indexes]; TRACE("read_uids_guids: no_ids %d\n", sBlk.s.no_ids); id_table = malloc(bytes); if(id_table == NULL) { ERROR("read_uids_guids: failed to allocate id table\n"); return FALSE; } res = read_fs_bytes(fd, sBlk.s.id_table_start, SQUASHFS_ID_BLOCK_BYTES(sBlk.s.no_ids), id_index_table); if(res == FALSE) { ERROR("read_uids_guids: failed to read id index table\n"); return FALSE; } SQUASHFS_INSWAP_ID_BLOCKS(id_index_table, indexes); for(i = 0; i < indexes; i++) { int expected = (i + 1) != indexes ? SQUASHFS_METADATA_SIZE : bytes & (SQUASHFS_METADATA_SIZE - 1); res = read_block(fd, id_index_table[i], NULL, expected, ((char *) id_table) + i * SQUASHFS_METADATA_SIZE); if(res == FALSE) { ERROR("read_uids_guids: failed to read id table block" "\n"); return FALSE; } } SQUASHFS_INSWAP_INTS(id_table, sBlk.s.no_ids); return TRUE; } squashfs4.3/squashfs-tools/unsquashfs.c0000644000175000017500000021353212334244273020375 0ustar phillipphillip/* * Unsquash a squashfs filesystem. This is a highly compressed read only * filesystem. * * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, * 2012, 2013, 2014 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * unsquashfs.c */ #include "unsquashfs.h" #include "squashfs_swap.h" #include "squashfs_compat.h" #include "compressor.h" #include "xattr.h" #include "unsquashfs_info.h" #include "stdarg.h" #include #include #include #include #include #include struct cache *fragment_cache, *data_cache; struct queue *to_reader, *to_inflate, *to_writer, *from_writer; pthread_t *thread, *inflator_thread; pthread_mutex_t fragment_mutex; /* user options that control parallelisation */ int processors = -1; struct super_block sBlk; squashfs_operations s_ops; struct compressor *comp; int bytes = 0, swap, file_count = 0, dir_count = 0, sym_count = 0, dev_count = 0, fifo_count = 0; char *inode_table = NULL, *directory_table = NULL; struct hash_table_entry *inode_table_hash[65536], *directory_table_hash[65536]; int fd; unsigned int *uid_table, *guid_table; unsigned int cached_frag = SQUASHFS_INVALID_FRAG; char *fragment_data; char *file_data; char *data; unsigned int block_size; unsigned int block_log; int lsonly = FALSE, info = FALSE, force = FALSE, short_ls = TRUE; int use_regex = FALSE; char **created_inode; int root_process; int columns; int rotate = 0; pthread_mutex_t screen_mutex; int progress = TRUE, progress_enabled = FALSE; unsigned int total_blocks = 0, total_files = 0, total_inodes = 0; unsigned int cur_blocks = 0; int inode_number = 1; int no_xattrs = XATTR_DEF; int user_xattrs = FALSE; int lookup_type[] = { 0, S_IFDIR, S_IFREG, S_IFLNK, S_IFBLK, S_IFCHR, S_IFIFO, S_IFSOCK, S_IFDIR, S_IFREG, S_IFLNK, S_IFBLK, S_IFCHR, S_IFIFO, S_IFSOCK }; struct test table[] = { { S_IFMT, S_IFSOCK, 0, 's' }, { S_IFMT, S_IFLNK, 0, 'l' }, { S_IFMT, S_IFBLK, 0, 'b' }, { S_IFMT, S_IFDIR, 0, 'd' }, { S_IFMT, S_IFCHR, 0, 'c' }, { S_IFMT, S_IFIFO, 0, 'p' }, { S_IRUSR, S_IRUSR, 1, 'r' }, { S_IWUSR, S_IWUSR, 2, 'w' }, { S_IRGRP, S_IRGRP, 4, 'r' }, { S_IWGRP, S_IWGRP, 5, 'w' }, { S_IROTH, S_IROTH, 7, 'r' }, { S_IWOTH, S_IWOTH, 8, 'w' }, { S_IXUSR | S_ISUID, S_IXUSR | S_ISUID, 3, 's' }, { S_IXUSR | S_ISUID, S_ISUID, 3, 'S' }, { S_IXUSR | S_ISUID, S_IXUSR, 3, 'x' }, { S_IXGRP | S_ISGID, S_IXGRP | S_ISGID, 6, 's' }, { S_IXGRP | S_ISGID, S_ISGID, 6, 'S' }, { S_IXGRP | S_ISGID, S_IXGRP, 6, 'x' }, { S_IXOTH | S_ISVTX, S_IXOTH | S_ISVTX, 9, 't' }, { S_IXOTH | S_ISVTX, S_ISVTX, 9, 'T' }, { S_IXOTH | S_ISVTX, S_IXOTH, 9, 'x' }, { 0, 0, 0, 0} }; void progress_bar(long long current, long long max, int columns); #define MAX_LINE 16384 void prep_exit() { } void sigwinch_handler() { struct winsize winsize; if(ioctl(1, TIOCGWINSZ, &winsize) == -1) { if(isatty(STDOUT_FILENO)) ERROR("TIOCGWINSZ ioctl failed, defaulting to 80 " "columns\n"); columns = 80; } else columns = winsize.ws_col; } void sigalrm_handler() { rotate = (rotate + 1) % 4; } int add_overflow(int a, int b) { return (INT_MAX - a) < b; } int shift_overflow(int a, int shift) { return (INT_MAX >> shift) < a; } int multiply_overflow(int a, int multiplier) { return (INT_MAX / multiplier) < a; } struct queue *queue_init(int size) { struct queue *queue = malloc(sizeof(struct queue)); if(queue == NULL) EXIT_UNSQUASH("Out of memory in queue_init\n"); if(add_overflow(size, 1) || multiply_overflow(size + 1, sizeof(void *))) EXIT_UNSQUASH("Size too large in queue_init\n"); queue->data = malloc(sizeof(void *) * (size + 1)); if(queue->data == NULL) EXIT_UNSQUASH("Out of memory in queue_init\n"); queue->size = size + 1; queue->readp = queue->writep = 0; pthread_mutex_init(&queue->mutex, NULL); pthread_cond_init(&queue->empty, NULL); pthread_cond_init(&queue->full, NULL); return queue; } void queue_put(struct queue *queue, void *data) { int nextp; pthread_mutex_lock(&queue->mutex); while((nextp = (queue->writep + 1) % queue->size) == queue->readp) pthread_cond_wait(&queue->full, &queue->mutex); queue->data[queue->writep] = data; queue->writep = nextp; pthread_cond_signal(&queue->empty); pthread_mutex_unlock(&queue->mutex); } void *queue_get(struct queue *queue) { void *data; pthread_mutex_lock(&queue->mutex); while(queue->readp == queue->writep) pthread_cond_wait(&queue->empty, &queue->mutex); data = queue->data[queue->readp]; queue->readp = (queue->readp + 1) % queue->size; pthread_cond_signal(&queue->full); pthread_mutex_unlock(&queue->mutex); return data; } void dump_queue(struct queue *queue) { pthread_mutex_lock(&queue->mutex); printf("Max size %d, size %d%s\n", queue->size - 1, queue->readp <= queue->writep ? queue->writep - queue->readp : queue->size - queue->readp + queue->writep, queue->readp == queue->writep ? " (EMPTY)" : ((queue->writep + 1) % queue->size) == queue->readp ? " (FULL)" : ""); pthread_mutex_unlock(&queue->mutex); } /* Called with the cache mutex held */ void insert_hash_table(struct cache *cache, struct cache_entry *entry) { int hash = CALCULATE_HASH(entry->block); entry->hash_next = cache->hash_table[hash]; cache->hash_table[hash] = entry; entry->hash_prev = NULL; if(entry->hash_next) entry->hash_next->hash_prev = entry; } /* Called with the cache mutex held */ void remove_hash_table(struct cache *cache, struct cache_entry *entry) { if(entry->hash_prev) entry->hash_prev->hash_next = entry->hash_next; else cache->hash_table[CALCULATE_HASH(entry->block)] = entry->hash_next; if(entry->hash_next) entry->hash_next->hash_prev = entry->hash_prev; entry->hash_prev = entry->hash_next = NULL; } /* Called with the cache mutex held */ void insert_free_list(struct cache *cache, struct cache_entry *entry) { if(cache->free_list) { entry->free_next = cache->free_list; entry->free_prev = cache->free_list->free_prev; cache->free_list->free_prev->free_next = entry; cache->free_list->free_prev = entry; } else { cache->free_list = entry; entry->free_prev = entry->free_next = entry; } } /* Called with the cache mutex held */ void remove_free_list(struct cache *cache, struct cache_entry *entry) { if(entry->free_prev == NULL || entry->free_next == NULL) /* not in free list */ return; else if(entry->free_prev == entry && entry->free_next == entry) { /* only this entry in the free list */ cache->free_list = NULL; } else { /* more than one entry in the free list */ entry->free_next->free_prev = entry->free_prev; entry->free_prev->free_next = entry->free_next; if(cache->free_list == entry) cache->free_list = entry->free_next; } entry->free_prev = entry->free_next = NULL; } struct cache *cache_init(int buffer_size, int max_buffers) { struct cache *cache = malloc(sizeof(struct cache)); if(cache == NULL) EXIT_UNSQUASH("Out of memory in cache_init\n"); cache->max_buffers = max_buffers; cache->buffer_size = buffer_size; cache->count = 0; cache->used = 0; cache->free_list = NULL; memset(cache->hash_table, 0, sizeof(struct cache_entry *) * 65536); cache->wait_free = FALSE; cache->wait_pending = FALSE; pthread_mutex_init(&cache->mutex, NULL); pthread_cond_init(&cache->wait_for_free, NULL); pthread_cond_init(&cache->wait_for_pending, NULL); return cache; } struct cache_entry *cache_get(struct cache *cache, long long block, int size) { /* * Get a block out of the cache. If the block isn't in the cache * it is added and queued to the reader() and inflate() threads for * reading off disk and decompression. The cache grows until max_blocks * is reached, once this occurs existing discarded blocks on the free * list are reused */ int hash = CALCULATE_HASH(block); struct cache_entry *entry; pthread_mutex_lock(&cache->mutex); for(entry = cache->hash_table[hash]; entry; entry = entry->hash_next) if(entry->block == block) break; if(entry) { /* * found the block in the cache. If the block is currently unused * remove it from the free list and increment cache used count. */ if(entry->used == 0) { cache->used ++; remove_free_list(cache, entry); } entry->used ++; pthread_mutex_unlock(&cache->mutex); } else { /* * not in the cache * * first try to allocate new block */ if(cache->count < cache->max_buffers) { entry = malloc(sizeof(struct cache_entry)); if(entry == NULL) EXIT_UNSQUASH("Out of memory in cache_get\n"); entry->data = malloc(cache->buffer_size); if(entry->data == NULL) EXIT_UNSQUASH("Out of memory in cache_get\n"); entry->cache = cache; entry->free_prev = entry->free_next = NULL; cache->count ++; } else { /* * try to get from free list */ while(cache->free_list == NULL) { cache->wait_free = TRUE; pthread_cond_wait(&cache->wait_for_free, &cache->mutex); } entry = cache->free_list; remove_free_list(cache, entry); remove_hash_table(cache, entry); } /* * Initialise block and insert into the hash table. * Increment used which tracks how many buffers in the * cache are actively in use (the other blocks, count - used, * are in the cache and available for lookup, but can also be * re-used). */ entry->block = block; entry->size = size; entry->used = 1; entry->error = FALSE; entry->pending = TRUE; insert_hash_table(cache, entry); cache->used ++; /* * queue to read thread to read and ultimately (via the * decompress threads) decompress the buffer */ pthread_mutex_unlock(&cache->mutex); queue_put(to_reader, entry); } return entry; } void cache_block_ready(struct cache_entry *entry, int error) { /* * mark cache entry as being complete, reading and (if necessary) * decompression has taken place, and the buffer is valid for use. * If an error occurs reading or decompressing, the buffer also * becomes ready but with an error... */ pthread_mutex_lock(&entry->cache->mutex); entry->pending = FALSE; entry->error = error; /* * if the wait_pending flag is set, one or more threads may be waiting * on this buffer */ if(entry->cache->wait_pending) { entry->cache->wait_pending = FALSE; pthread_cond_broadcast(&entry->cache->wait_for_pending); } pthread_mutex_unlock(&entry->cache->mutex); } void cache_block_wait(struct cache_entry *entry) { /* * wait for this cache entry to become ready, when reading and (if * necessary) decompression has taken place */ pthread_mutex_lock(&entry->cache->mutex); while(entry->pending) { entry->cache->wait_pending = TRUE; pthread_cond_wait(&entry->cache->wait_for_pending, &entry->cache->mutex); } pthread_mutex_unlock(&entry->cache->mutex); } void cache_block_put(struct cache_entry *entry) { /* * finished with this cache entry, once the usage count reaches zero it * can be reused and is put onto the free list. As it remains * accessible via the hash table it can be found getting a new lease of * life before it is reused. */ pthread_mutex_lock(&entry->cache->mutex); entry->used --; if(entry->used == 0) { insert_free_list(entry->cache, entry); entry->cache->used --; /* * if the wait_free flag is set, one or more threads may be * waiting on this buffer */ if(entry->cache->wait_free) { entry->cache->wait_free = FALSE; pthread_cond_broadcast(&entry->cache->wait_for_free); } } pthread_mutex_unlock(&entry->cache->mutex); } void dump_cache(struct cache *cache) { pthread_mutex_lock(&cache->mutex); printf("Max buffers %d, Current size %d, Used %d, %s\n", cache->max_buffers, cache->count, cache->used, cache->free_list ? "Free buffers" : "No free buffers"); pthread_mutex_unlock(&cache->mutex); } char *modestr(char *str, int mode) { int i; strcpy(str, "----------"); for(i = 0; table[i].mask != 0; i++) { if((mode & table[i].mask) == table[i].value) str[table[i].position] = table[i].mode; } return str; } #define TOTALCHARS 25 int print_filename(char *pathname, struct inode *inode) { char str[11], dummy[12], dummy2[12]; /* overflow safe */ char *userstr, *groupstr; int padchars; struct passwd *user; struct group *group; struct tm *t; if(short_ls) { printf("%s\n", pathname); return 1; } user = getpwuid(inode->uid); if(user == NULL) { int res = snprintf(dummy, 12, "%d", inode->uid); if(res < 0) EXIT_UNSQUASH("snprintf failed in print_filename()\n"); else if(res >= 12) /* unsigned int shouldn't ever need more than 11 bytes * (including terminating '\0') to print in base 10 */ userstr = "*"; else userstr = dummy; } else userstr = user->pw_name; group = getgrgid(inode->gid); if(group == NULL) { int res = snprintf(dummy2, 12, "%d", inode->gid); if(res < 0) EXIT_UNSQUASH("snprintf failed in print_filename()\n"); else if(res >= 12) /* unsigned int shouldn't ever need more than 11 bytes * (including terminating '\0') to print in base 10 */ groupstr = "*"; else groupstr = dummy2; } else groupstr = group->gr_name; printf("%s %s/%s ", modestr(str, inode->mode), userstr, groupstr); switch(inode->mode & S_IFMT) { case S_IFREG: case S_IFDIR: case S_IFSOCK: case S_IFIFO: case S_IFLNK: padchars = TOTALCHARS - strlen(userstr) - strlen(groupstr); printf("%*lld ", padchars > 0 ? padchars : 0, inode->data); break; case S_IFCHR: case S_IFBLK: padchars = TOTALCHARS - strlen(userstr) - strlen(groupstr) - 7; printf("%*s%3d,%3d ", padchars > 0 ? padchars : 0, " ", (int) inode->data >> 8, (int) inode->data & 0xff); break; } t = localtime(&inode->time); printf("%d-%02d-%02d %02d:%02d %s", t->tm_year + 1900, t->tm_mon + 1, t->tm_mday, t->tm_hour, t->tm_min, pathname); if((inode->mode & S_IFMT) == S_IFLNK) printf(" -> %s", inode->symlink); printf("\n"); return 1; } void add_entry(struct hash_table_entry *hash_table[], long long start, int bytes) { int hash = CALCULATE_HASH(start); struct hash_table_entry *hash_table_entry; hash_table_entry = malloc(sizeof(struct hash_table_entry)); if(hash_table_entry == NULL) EXIT_UNSQUASH("Out of memory in add_entry\n"); hash_table_entry->start = start; hash_table_entry->bytes = bytes; hash_table_entry->next = hash_table[hash]; hash_table[hash] = hash_table_entry; } int lookup_entry(struct hash_table_entry *hash_table[], long long start) { int hash = CALCULATE_HASH(start); struct hash_table_entry *hash_table_entry; for(hash_table_entry = hash_table[hash]; hash_table_entry; hash_table_entry = hash_table_entry->next) if(hash_table_entry->start == start) return hash_table_entry->bytes; return -1; } int read_fs_bytes(int fd, long long byte, int bytes, void *buff) { off_t off = byte; int res, count; TRACE("read_bytes: reading from position 0x%llx, bytes %d\n", byte, bytes); if(lseek(fd, off, SEEK_SET) == -1) { ERROR("Lseek failed because %s\n", strerror(errno)); return FALSE; } for(count = 0; count < bytes; count += res) { res = read(fd, buff + count, bytes - count); if(res < 1) { if(res == 0) { ERROR("Read on filesystem failed because " "EOF\n"); return FALSE; } else if(errno != EINTR) { ERROR("Read on filesystem failed because %s\n", strerror(errno)); return FALSE; } else res = 0; } } return TRUE; } int read_block(int fd, long long start, long long *next, int expected, void *block) { unsigned short c_byte; int offset = 2, res, compressed; int outlen = expected ? expected : SQUASHFS_METADATA_SIZE; if(swap) { if(read_fs_bytes(fd, start, 2, &c_byte) == FALSE) goto failed; c_byte = (c_byte >> 8) | ((c_byte & 0xff) << 8); } else if(read_fs_bytes(fd, start, 2, &c_byte) == FALSE) goto failed; TRACE("read_block: block @0x%llx, %d %s bytes\n", start, SQUASHFS_COMPRESSED_SIZE(c_byte), SQUASHFS_COMPRESSED(c_byte) ? "compressed" : "uncompressed"); if(SQUASHFS_CHECK_DATA(sBlk.s.flags)) offset = 3; compressed = SQUASHFS_COMPRESSED(c_byte); c_byte = SQUASHFS_COMPRESSED_SIZE(c_byte); /* * The block size should not be larger than * the uncompressed size (or max uncompressed size if * expected is 0) */ if(c_byte > outlen) return 0; if(compressed) { char buffer[c_byte]; int error; res = read_fs_bytes(fd, start + offset, c_byte, buffer); if(res == FALSE) goto failed; res = compressor_uncompress(comp, block, buffer, c_byte, outlen, &error); if(res == -1) { ERROR("%s uncompress failed with error code %d\n", comp->name, error); goto failed; } } else { res = read_fs_bytes(fd, start + offset, c_byte, block); if(res == FALSE) goto failed; res = c_byte; } if(next) *next = start + offset + c_byte; /* * if expected, then check the (uncompressed) return data * is of the expected size */ if(expected && expected != res) return 0; else return res; failed: ERROR("read_block: failed to read block @0x%llx\n", start); return FALSE; } int read_data_block(long long start, unsigned int size, char *block) { int error, res; int c_byte = SQUASHFS_COMPRESSED_SIZE_BLOCK(size); TRACE("read_data_block: block @0x%llx, %d %s bytes\n", start, c_byte, SQUASHFS_COMPRESSED_BLOCK(size) ? "compressed" : "uncompressed"); if(SQUASHFS_COMPRESSED_BLOCK(size)) { if(read_fs_bytes(fd, start, c_byte, data) == FALSE) goto failed; res = compressor_uncompress(comp, block, data, c_byte, block_size, &error); if(res == -1) { ERROR("%s uncompress failed with error code %d\n", comp->name, error); goto failed; } return res; } else { if(read_fs_bytes(fd, start, c_byte, block) == FALSE) goto failed; return c_byte; } failed: ERROR("read_data_block: failed to read block @0x%llx, size %d\n", start, c_byte); return FALSE; } int read_inode_table(long long start, long long end) { int size = 0, bytes = 0, res; TRACE("read_inode_table: start %lld, end %lld\n", start, end); while(start < end) { if(size - bytes < SQUASHFS_METADATA_SIZE) { inode_table = realloc(inode_table, size += SQUASHFS_METADATA_SIZE); if(inode_table == NULL) { ERROR("Out of memory in read_inode_table"); goto failed; } } add_entry(inode_table_hash, start, bytes); res = read_block(fd, start, &start, 0, inode_table + bytes); if(res == 0) { ERROR("read_inode_table: failed to read block\n"); goto failed; } bytes += res; /* * If this is not the last metadata block in the inode table * then it should be SQUASHFS_METADATA_SIZE in size. * Note, we can't use expected in read_block() above for this * because we don't know if this is the last block until * after reading. */ if(start != end && res != SQUASHFS_METADATA_SIZE) { ERROR("read_inode_table: metadata block should be %d " "bytes in length, it is %d bytes\n", SQUASHFS_METADATA_SIZE, res); goto failed; } } return TRUE; failed: free(inode_table); return FALSE; } int set_attributes(char *pathname, int mode, uid_t uid, gid_t guid, time_t time, unsigned int xattr, unsigned int set_mode) { struct utimbuf times = { time, time }; write_xattr(pathname, xattr); if(utime(pathname, ×) == -1) { ERROR("set_attributes: failed to set time on %s, because %s\n", pathname, strerror(errno)); return FALSE; } if(root_process) { if(chown(pathname, uid, guid) == -1) { ERROR("set_attributes: failed to change uid and gids " "on %s, because %s\n", pathname, strerror(errno)); return FALSE; } } else mode &= ~07000; if((set_mode || (mode & 07000)) && chmod(pathname, (mode_t) mode) == -1) { ERROR("set_attributes: failed to change mode %s, because %s\n", pathname, strerror(errno)); return FALSE; } return TRUE; } int write_bytes(int fd, char *buff, int bytes) { int res, count; for(count = 0; count < bytes; count += res) { res = write(fd, buff + count, bytes - count); if(res == -1) { if(errno != EINTR) { ERROR("Write on output file failed because " "%s\n", strerror(errno)); return -1; } res = 0; } } return 0; } int lseek_broken = FALSE; char *zero_data = NULL; int write_block(int file_fd, char *buffer, int size, long long hole, int sparse) { off_t off = hole; if(hole) { if(sparse && lseek_broken == FALSE) { int error = lseek(file_fd, off, SEEK_CUR); if(error == -1) /* failed to seek beyond end of file */ lseek_broken = TRUE; } if((sparse == FALSE || lseek_broken) && zero_data == NULL) { if((zero_data = malloc(block_size)) == NULL) EXIT_UNSQUASH("write_block: failed to alloc " "zero data block\n"); memset(zero_data, 0, block_size); } if(sparse == FALSE || lseek_broken) { int blocks = (hole + block_size -1) / block_size; int avail_bytes, i; for(i = 0; i < blocks; i++, hole -= avail_bytes) { avail_bytes = hole > block_size ? block_size : hole; if(write_bytes(file_fd, zero_data, avail_bytes) == -1) goto failure; } } } if(write_bytes(file_fd, buffer, size) == -1) goto failure; return TRUE; failure: return FALSE; } pthread_mutex_t open_mutex = PTHREAD_MUTEX_INITIALIZER; pthread_cond_t open_empty = PTHREAD_COND_INITIALIZER; int open_unlimited, open_count; #define OPEN_FILE_MARGIN 10 void open_init(int count) { open_count = count; open_unlimited = count == -1; } int open_wait(char *pathname, int flags, mode_t mode) { if (!open_unlimited) { pthread_mutex_lock(&open_mutex); while (open_count == 0) pthread_cond_wait(&open_empty, &open_mutex); open_count --; pthread_mutex_unlock(&open_mutex); } return open(pathname, flags, mode); } void close_wake(int fd) { close(fd); if (!open_unlimited) { pthread_mutex_lock(&open_mutex); open_count ++; pthread_cond_signal(&open_empty); pthread_mutex_unlock(&open_mutex); } } void queue_file(char *pathname, int file_fd, struct inode *inode) { struct squashfs_file *file = malloc(sizeof(struct squashfs_file)); if(file == NULL) EXIT_UNSQUASH("queue_file: unable to malloc file\n"); file->fd = file_fd; file->file_size = inode->data; file->mode = inode->mode; file->gid = inode->gid; file->uid = inode->uid; file->time = inode->time; file->pathname = strdup(pathname); file->blocks = inode->blocks + (inode->frag_bytes > 0); file->sparse = inode->sparse; file->xattr = inode->xattr; queue_put(to_writer, file); } void queue_dir(char *pathname, struct dir *dir) { struct squashfs_file *file = malloc(sizeof(struct squashfs_file)); if(file == NULL) EXIT_UNSQUASH("queue_dir: unable to malloc file\n"); file->fd = -1; file->mode = dir->mode; file->gid = dir->guid; file->uid = dir->uid; file->time = dir->mtime; file->pathname = strdup(pathname); file->xattr = dir->xattr; queue_put(to_writer, file); } int write_file(struct inode *inode, char *pathname) { unsigned int file_fd, i; unsigned int *block_list; int file_end = inode->data / block_size; long long start = inode->start; TRACE("write_file: regular file, blocks %d\n", inode->blocks); file_fd = open_wait(pathname, O_CREAT | O_WRONLY | (force ? O_TRUNC : 0), (mode_t) inode->mode & 0777); if(file_fd == -1) { ERROR("write_file: failed to create file %s, because %s\n", pathname, strerror(errno)); return FALSE; } block_list = malloc(inode->blocks * sizeof(unsigned int)); if(block_list == NULL) EXIT_UNSQUASH("write_file: unable to malloc block list\n"); s_ops.read_block_list(block_list, inode->block_ptr, inode->blocks); /* * the writer thread is queued a squashfs_file structure describing the * file. If the file has one or more blocks or a fragment they are * queued separately (references to blocks in the cache). */ queue_file(pathname, file_fd, inode); for(i = 0; i < inode->blocks; i++) { int c_byte = SQUASHFS_COMPRESSED_SIZE_BLOCK(block_list[i]); struct file_entry *block = malloc(sizeof(struct file_entry)); if(block == NULL) EXIT_UNSQUASH("write_file: unable to malloc file\n"); block->offset = 0; block->size = i == file_end ? inode->data & (block_size - 1) : block_size; if(block_list[i] == 0) /* sparse block */ block->buffer = NULL; else { block->buffer = cache_get(data_cache, start, block_list[i]); start += c_byte; } queue_put(to_writer, block); } if(inode->frag_bytes) { int size; long long start; struct file_entry *block = malloc(sizeof(struct file_entry)); if(block == NULL) EXIT_UNSQUASH("write_file: unable to malloc file\n"); s_ops.read_fragment(inode->fragment, &start, &size); block->buffer = cache_get(fragment_cache, start, size); block->offset = inode->offset; block->size = inode->frag_bytes; queue_put(to_writer, block); } free(block_list); return TRUE; } int create_inode(char *pathname, struct inode *i) { TRACE("create_inode: pathname %s\n", pathname); if(created_inode[i->inode_number - 1]) { TRACE("create_inode: hard link\n"); if(force) unlink(pathname); if(link(created_inode[i->inode_number - 1], pathname) == -1) { ERROR("create_inode: failed to create hardlink, " "because %s\n", strerror(errno)); return FALSE; } return TRUE; } switch(i->type) { case SQUASHFS_FILE_TYPE: case SQUASHFS_LREG_TYPE: TRACE("create_inode: regular file, file_size %lld, " "blocks %d\n", i->data, i->blocks); if(write_file(i, pathname)) file_count ++; break; case SQUASHFS_SYMLINK_TYPE: case SQUASHFS_LSYMLINK_TYPE: TRACE("create_inode: symlink, symlink_size %lld\n", i->data); if(force) unlink(pathname); if(symlink(i->symlink, pathname) == -1) { ERROR("create_inode: failed to create symlink " "%s, because %s\n", pathname, strerror(errno)); break; } write_xattr(pathname, i->xattr); if(root_process) { if(lchown(pathname, i->uid, i->gid) == -1) ERROR("create_inode: failed to change " "uid and gids on %s, because " "%s\n", pathname, strerror(errno)); } sym_count ++; break; case SQUASHFS_BLKDEV_TYPE: case SQUASHFS_CHRDEV_TYPE: case SQUASHFS_LBLKDEV_TYPE: case SQUASHFS_LCHRDEV_TYPE: { int chrdev = i->type == SQUASHFS_CHRDEV_TYPE; TRACE("create_inode: dev, rdev 0x%llx\n", i->data); if(root_process) { if(force) unlink(pathname); if(mknod(pathname, chrdev ? S_IFCHR : S_IFBLK, makedev((i->data >> 8) & 0xff, i->data & 0xff)) == -1) { ERROR("create_inode: failed to create " "%s device %s, because %s\n", chrdev ? "character" : "block", pathname, strerror(errno)); break; } set_attributes(pathname, i->mode, i->uid, i->gid, i->time, i->xattr, TRUE); dev_count ++; } else ERROR("create_inode: could not create %s " "device %s, because you're not " "superuser!\n", chrdev ? "character" : "block", pathname); break; } case SQUASHFS_FIFO_TYPE: case SQUASHFS_LFIFO_TYPE: TRACE("create_inode: fifo\n"); if(force) unlink(pathname); if(mknod(pathname, S_IFIFO, 0) == -1) { ERROR("create_inode: failed to create fifo %s, " "because %s\n", pathname, strerror(errno)); break; } set_attributes(pathname, i->mode, i->uid, i->gid, i->time, i->xattr, TRUE); fifo_count ++; break; case SQUASHFS_SOCKET_TYPE: case SQUASHFS_LSOCKET_TYPE: TRACE("create_inode: socket\n"); ERROR("create_inode: socket %s ignored\n", pathname); break; default: ERROR("Unknown inode type %d in create_inode_table!\n", i->type); return FALSE; } created_inode[i->inode_number - 1] = strdup(pathname); return TRUE; } int read_directory_table(long long start, long long end) { int bytes = 0, size = 0, res; TRACE("read_directory_table: start %lld, end %lld\n", start, end); while(start < end) { if(size - bytes < SQUASHFS_METADATA_SIZE) { directory_table = realloc(directory_table, size += SQUASHFS_METADATA_SIZE); if(directory_table == NULL) { ERROR("Out of memory in " "read_directory_table\n"); goto failed; } } add_entry(directory_table_hash, start, bytes); res = read_block(fd, start, &start, 0, directory_table + bytes); if(res == 0) { ERROR("read_directory_table: failed to read block\n"); goto failed; } bytes += res; /* * If this is not the last metadata block in the directory table * then it should be SQUASHFS_METADATA_SIZE in size. * Note, we can't use expected in read_block() above for this * because we don't know if this is the last block until * after reading. */ if(start != end && res != SQUASHFS_METADATA_SIZE) { ERROR("read_directory_table: metadata block " "should be %d bytes in length, it is %d " "bytes\n", SQUASHFS_METADATA_SIZE, res); goto failed; } } return TRUE; failed: free(directory_table); return FALSE; } int squashfs_readdir(struct dir *dir, char **name, unsigned int *start_block, unsigned int *offset, unsigned int *type) { if(dir->cur_entry == dir->dir_count) return FALSE; *name = dir->dirs[dir->cur_entry].name; *start_block = dir->dirs[dir->cur_entry].start_block; *offset = dir->dirs[dir->cur_entry].offset; *type = dir->dirs[dir->cur_entry].type; dir->cur_entry ++; return TRUE; } void squashfs_closedir(struct dir *dir) { free(dir->dirs); free(dir); } char *get_component(char *target, char **targname) { char *start; while(*target == '/') target ++; start = target; while(*target != '/' && *target != '\0') target ++; *targname = strndup(start, target - start); while(*target == '/') target ++; return target; } void free_path(struct pathname *paths) { int i; for(i = 0; i < paths->names; i++) { if(paths->name[i].paths) free_path(paths->name[i].paths); free(paths->name[i].name); if(paths->name[i].preg) { regfree(paths->name[i].preg); free(paths->name[i].preg); } } free(paths); } struct pathname *add_path(struct pathname *paths, char *target, char *alltarget) { char *targname; int i, error; TRACE("add_path: adding \"%s\" extract file\n", target); target = get_component(target, &targname); if(paths == NULL) { paths = malloc(sizeof(struct pathname)); if(paths == NULL) EXIT_UNSQUASH("failed to allocate paths\n"); paths->names = 0; paths->name = NULL; } for(i = 0; i < paths->names; i++) if(strcmp(paths->name[i].name, targname) == 0) break; if(i == paths->names) { /* * allocate new name entry */ paths->names ++; paths->name = realloc(paths->name, (i + 1) * sizeof(struct path_entry)); if(paths->name == NULL) EXIT_UNSQUASH("Out of memory in add_path\n"); paths->name[i].name = targname; paths->name[i].paths = NULL; if(use_regex) { paths->name[i].preg = malloc(sizeof(regex_t)); if(paths->name[i].preg == NULL) EXIT_UNSQUASH("Out of memory in add_path\n"); error = regcomp(paths->name[i].preg, targname, REG_EXTENDED|REG_NOSUB); if(error) { char str[1024]; /* overflow safe */ regerror(error, paths->name[i].preg, str, 1024); EXIT_UNSQUASH("invalid regex %s in export %s, " "because %s\n", targname, alltarget, str); } } else paths->name[i].preg = NULL; if(target[0] == '\0') /* * at leaf pathname component */ paths->name[i].paths = NULL; else /* * recurse adding child components */ paths->name[i].paths = add_path(NULL, target, alltarget); } else { /* * existing matching entry */ free(targname); if(paths->name[i].paths == NULL) { /* * No sub-directory which means this is the leaf * component of a pre-existing extract which subsumes * the extract currently being added, in which case stop * adding components */ } else if(target[0] == '\0') { /* * at leaf pathname component and child components exist * from more specific extracts, delete as they're * subsumed by this extract */ free_path(paths->name[i].paths); paths->name[i].paths = NULL; } else /* * recurse adding child components */ add_path(paths->name[i].paths, target, alltarget); } return paths; } struct pathnames *init_subdir() { struct pathnames *new = malloc(sizeof(struct pathnames)); if(new == NULL) EXIT_UNSQUASH("Out of memory in init_subdir\n"); new->count = 0; return new; } struct pathnames *add_subdir(struct pathnames *paths, struct pathname *path) { if(paths->count % PATHS_ALLOC_SIZE == 0) { paths = realloc(paths, sizeof(struct pathnames *) + (paths->count + PATHS_ALLOC_SIZE) * sizeof(struct pathname *)); if(paths == NULL) EXIT_UNSQUASH("Out of memory in add_subdir\n"); } paths->path[paths->count++] = path; return paths; } void free_subdir(struct pathnames *paths) { free(paths); } int matches(struct pathnames *paths, char *name, struct pathnames **new) { int i, n; if(paths == NULL) { *new = NULL; return TRUE; } *new = init_subdir(); for(n = 0; n < paths->count; n++) { struct pathname *path = paths->path[n]; for(i = 0; i < path->names; i++) { int match = use_regex ? regexec(path->name[i].preg, name, (size_t) 0, NULL, 0) == 0 : fnmatch(path->name[i].name, name, FNM_PATHNAME|FNM_PERIOD|FNM_EXTMATCH) == 0; if(match && path->name[i].paths == NULL) /* * match on a leaf component, any subdirectories * will implicitly match, therefore return an * empty new search set */ goto empty_set; if(match) /* * match on a non-leaf component, add any * subdirectories to the new set of * subdirectories to scan for this name */ *new = add_subdir(*new, path->name[i].paths); } } if((*new)->count == 0) { /* * no matching names found, delete empty search set, and return * FALSE */ free_subdir(*new); *new = NULL; return FALSE; } /* * one or more matches with sub-directories found (no leaf matches), * return new search set and return TRUE */ return TRUE; empty_set: /* * found matching leaf exclude, return empty search set and return TRUE */ free_subdir(*new); *new = NULL; return TRUE; } void pre_scan(char *parent_name, unsigned int start_block, unsigned int offset, struct pathnames *paths) { unsigned int type; char *name; struct pathnames *new; struct inode *i; struct dir *dir = s_ops.squashfs_opendir(start_block, offset, &i); if(dir == NULL) return; while(squashfs_readdir(dir, &name, &start_block, &offset, &type)) { struct inode *i; char *pathname; int res; TRACE("pre_scan: name %s, start_block %d, offset %d, type %d\n", name, start_block, offset, type); if(!matches(paths, name, &new)) continue; res = asprintf(&pathname, "%s/%s", parent_name, name); if(res == -1) EXIT_UNSQUASH("asprintf failed in dir_scan\n"); if(type == SQUASHFS_DIR_TYPE) pre_scan(parent_name, start_block, offset, new); else if(new == NULL) { if(type == SQUASHFS_FILE_TYPE || type == SQUASHFS_LREG_TYPE) { i = s_ops.read_inode(start_block, offset); if(created_inode[i->inode_number - 1] == NULL) { created_inode[i->inode_number - 1] = (char *) i; total_blocks += (i->data + (block_size - 1)) >> block_log; } total_files ++; } total_inodes ++; } free_subdir(new); free(pathname); } squashfs_closedir(dir); } void dir_scan(char *parent_name, unsigned int start_block, unsigned int offset, struct pathnames *paths) { unsigned int type; char *name; struct pathnames *new; struct inode *i; struct dir *dir = s_ops.squashfs_opendir(start_block, offset, &i); if(dir == NULL) { ERROR("dir_scan: failed to read directory %s, skipping\n", parent_name); return; } if(lsonly || info) print_filename(parent_name, i); if(!lsonly) { /* * Make directory with default User rwx permissions rather than * the permissions from the filesystem, as these may not have * write/execute permission. These are fixed up later in * set_attributes(). */ int res = mkdir(parent_name, S_IRUSR|S_IWUSR|S_IXUSR); if(res == -1) { /* * Skip directory if mkdir fails, unless we're * forcing and the error is -EEXIST */ if(!force || errno != EEXIST) { ERROR("dir_scan: failed to make directory %s, " "because %s\n", parent_name, strerror(errno)); squashfs_closedir(dir); return; } /* * Try to change permissions of existing directory so * that we can write to it */ res = chmod(parent_name, S_IRUSR|S_IWUSR|S_IXUSR); if (res == -1) ERROR("dir_scan: failed to change permissions " "for directory %s, because %s\n", parent_name, strerror(errno)); } } while(squashfs_readdir(dir, &name, &start_block, &offset, &type)) { char *pathname; int res; TRACE("dir_scan: name %s, start_block %d, offset %d, type %d\n", name, start_block, offset, type); if(!matches(paths, name, &new)) continue; res = asprintf(&pathname, "%s/%s", parent_name, name); if(res == -1) EXIT_UNSQUASH("asprintf failed in dir_scan\n"); if(type == SQUASHFS_DIR_TYPE) { dir_scan(pathname, start_block, offset, new); free(pathname); } else if(new == NULL) { update_info(pathname); i = s_ops.read_inode(start_block, offset); if(lsonly || info) print_filename(pathname, i); if(!lsonly) create_inode(pathname, i); if(i->type == SQUASHFS_SYMLINK_TYPE || i->type == SQUASHFS_LSYMLINK_TYPE) free(i->symlink); } else free(pathname); free_subdir(new); } if(!lsonly) queue_dir(parent_name, dir); squashfs_closedir(dir); dir_count ++; } void squashfs_stat(char *source) { time_t mkfs_time = (time_t) sBlk.s.mkfs_time; char *mkfs_str = ctime(&mkfs_time); #if __BYTE_ORDER == __BIG_ENDIAN printf("Found a valid %sSQUASHFS %d:%d superblock on %s.\n", sBlk.s.s_major == 4 ? "" : swap ? "little endian " : "big endian ", sBlk.s.s_major, sBlk.s.s_minor, source); #else printf("Found a valid %sSQUASHFS %d:%d superblock on %s.\n", sBlk.s.s_major == 4 ? "" : swap ? "big endian " : "little endian ", sBlk.s.s_major, sBlk.s.s_minor, source); #endif printf("Creation or last append time %s", mkfs_str ? mkfs_str : "failed to get time\n"); printf("Filesystem size %.2f Kbytes (%.2f Mbytes)\n", sBlk.s.bytes_used / 1024.0, sBlk.s.bytes_used / (1024.0 * 1024.0)); if(sBlk.s.s_major == 4) { printf("Compression %s\n", comp->name); if(SQUASHFS_COMP_OPTS(sBlk.s.flags)) { char buffer[SQUASHFS_METADATA_SIZE] __attribute__ ((aligned)); int bytes; bytes = read_block(fd, sizeof(sBlk.s), NULL, 0, buffer); if(bytes == 0) { ERROR("Failed to read compressor options\n"); return; } compressor_display_options(comp, buffer, bytes); } } printf("Block size %d\n", sBlk.s.block_size); printf("Filesystem is %sexportable via NFS\n", SQUASHFS_EXPORTABLE(sBlk.s.flags) ? "" : "not "); printf("Inodes are %scompressed\n", SQUASHFS_UNCOMPRESSED_INODES(sBlk.s.flags) ? "un" : ""); printf("Data is %scompressed\n", SQUASHFS_UNCOMPRESSED_DATA(sBlk.s.flags) ? "un" : ""); if(sBlk.s.s_major > 1) { if(SQUASHFS_NO_FRAGMENTS(sBlk.s.flags)) printf("Fragments are not stored\n"); else { printf("Fragments are %scompressed\n", SQUASHFS_UNCOMPRESSED_FRAGMENTS(sBlk.s.flags) ? "un" : ""); printf("Always-use-fragments option is %sspecified\n", SQUASHFS_ALWAYS_FRAGMENTS(sBlk.s.flags) ? "" : "not "); } } if(sBlk.s.s_major == 4) { if(SQUASHFS_NO_XATTRS(sBlk.s.flags)) printf("Xattrs are not stored\n"); else printf("Xattrs are %scompressed\n", SQUASHFS_UNCOMPRESSED_XATTRS(sBlk.s.flags) ? "un" : ""); } if(sBlk.s.s_major < 4) printf("Check data is %spresent in the filesystem\n", SQUASHFS_CHECK_DATA(sBlk.s.flags) ? "" : "not "); if(sBlk.s.s_major > 1) printf("Duplicates are %sremoved\n", SQUASHFS_DUPLICATES(sBlk.s.flags) ? "" : "not "); else printf("Duplicates are removed\n"); if(sBlk.s.s_major > 1) printf("Number of fragments %d\n", sBlk.s.fragments); printf("Number of inodes %d\n", sBlk.s.inodes); if(sBlk.s.s_major == 4) printf("Number of ids %d\n", sBlk.s.no_ids); else { printf("Number of uids %d\n", sBlk.no_uids); printf("Number of gids %d\n", sBlk.no_guids); } TRACE("sBlk.s.inode_table_start 0x%llx\n", sBlk.s.inode_table_start); TRACE("sBlk.s.directory_table_start 0x%llx\n", sBlk.s.directory_table_start); if(sBlk.s.s_major > 1) TRACE("sBlk.s.fragment_table_start 0x%llx\n\n", sBlk.s.fragment_table_start); if(sBlk.s.s_major > 2) TRACE("sBlk.s.lookup_table_start 0x%llx\n\n", sBlk.s.lookup_table_start); if(sBlk.s.s_major == 4) { TRACE("sBlk.s.id_table_start 0x%llx\n", sBlk.s.id_table_start); TRACE("sBlk.s.xattr_id_table_start 0x%llx\n", sBlk.s.xattr_id_table_start); } else { TRACE("sBlk.uid_start 0x%llx\n", sBlk.uid_start); TRACE("sBlk.guid_start 0x%llx\n", sBlk.guid_start); } } int check_compression(struct compressor *comp) { int res, bytes = 0; char buffer[SQUASHFS_METADATA_SIZE] __attribute__ ((aligned)); if(!comp->supported) { ERROR("Filesystem uses %s compression, this is " "unsupported by this version\n", comp->name); ERROR("Decompressors available:\n"); display_compressors("", ""); return 0; } /* * Read compression options from disk if present, and pass to * the compressor to ensure we know how to decompress a filesystem * compressed with these compression options. * * Note, even if there is no compression options we still call the * compressor because some compression options may be mandatory * for some compressors. */ if(SQUASHFS_COMP_OPTS(sBlk.s.flags)) { bytes = read_block(fd, sizeof(sBlk.s), NULL, 0, buffer); if(bytes == 0) { ERROR("Failed to read compressor options\n"); return 0; } } res = compressor_check_options(comp, sBlk.s.block_size, buffer, bytes); return res != -1; } int read_super(char *source) { squashfs_super_block_3 sBlk_3; struct squashfs_super_block sBlk_4; /* * Try to read a Squashfs 4 superblock */ read_fs_bytes(fd, SQUASHFS_START, sizeof(struct squashfs_super_block), &sBlk_4); swap = sBlk_4.s_magic != SQUASHFS_MAGIC; SQUASHFS_INSWAP_SUPER_BLOCK(&sBlk_4); if(sBlk_4.s_magic == SQUASHFS_MAGIC && sBlk_4.s_major == 4 && sBlk_4.s_minor == 0) { s_ops.squashfs_opendir = squashfs_opendir_4; s_ops.read_fragment = read_fragment_4; s_ops.read_fragment_table = read_fragment_table_4; s_ops.read_block_list = read_block_list_2; s_ops.read_inode = read_inode_4; s_ops.read_uids_guids = read_uids_guids_4; memcpy(&sBlk, &sBlk_4, sizeof(sBlk_4)); /* * Check the compression type */ comp = lookup_compressor_id(sBlk.s.compression); return TRUE; } /* * Not a Squashfs 4 superblock, try to read a squashfs 3 superblock * (compatible with 1 and 2 filesystems) */ read_fs_bytes(fd, SQUASHFS_START, sizeof(squashfs_super_block_3), &sBlk_3); /* * Check it is a SQUASHFS superblock */ swap = 0; if(sBlk_3.s_magic != SQUASHFS_MAGIC) { if(sBlk_3.s_magic == SQUASHFS_MAGIC_SWAP) { squashfs_super_block_3 sblk; ERROR("Reading a different endian SQUASHFS filesystem " "on %s\n", source); SQUASHFS_SWAP_SUPER_BLOCK_3(&sblk, &sBlk_3); memcpy(&sBlk_3, &sblk, sizeof(squashfs_super_block_3)); swap = 1; } else { ERROR("Can't find a SQUASHFS superblock on %s\n", source); goto failed_mount; } } sBlk.s.s_magic = sBlk_3.s_magic; sBlk.s.inodes = sBlk_3.inodes; sBlk.s.mkfs_time = sBlk_3.mkfs_time; sBlk.s.block_size = sBlk_3.block_size; sBlk.s.fragments = sBlk_3.fragments; sBlk.s.block_log = sBlk_3.block_log; sBlk.s.flags = sBlk_3.flags; sBlk.s.s_major = sBlk_3.s_major; sBlk.s.s_minor = sBlk_3.s_minor; sBlk.s.root_inode = sBlk_3.root_inode; sBlk.s.bytes_used = sBlk_3.bytes_used; sBlk.s.inode_table_start = sBlk_3.inode_table_start; sBlk.s.directory_table_start = sBlk_3.directory_table_start; sBlk.s.fragment_table_start = sBlk_3.fragment_table_start; sBlk.s.lookup_table_start = sBlk_3.lookup_table_start; sBlk.no_uids = sBlk_3.no_uids; sBlk.no_guids = sBlk_3.no_guids; sBlk.uid_start = sBlk_3.uid_start; sBlk.guid_start = sBlk_3.guid_start; sBlk.s.xattr_id_table_start = SQUASHFS_INVALID_BLK; /* Check the MAJOR & MINOR versions */ if(sBlk.s.s_major == 1 || sBlk.s.s_major == 2) { sBlk.s.bytes_used = sBlk_3.bytes_used_2; sBlk.uid_start = sBlk_3.uid_start_2; sBlk.guid_start = sBlk_3.guid_start_2; sBlk.s.inode_table_start = sBlk_3.inode_table_start_2; sBlk.s.directory_table_start = sBlk_3.directory_table_start_2; if(sBlk.s.s_major == 1) { sBlk.s.block_size = sBlk_3.block_size_1; sBlk.s.fragment_table_start = sBlk.uid_start; s_ops.squashfs_opendir = squashfs_opendir_1; s_ops.read_fragment_table = read_fragment_table_1; s_ops.read_block_list = read_block_list_1; s_ops.read_inode = read_inode_1; s_ops.read_uids_guids = read_uids_guids_1; } else { sBlk.s.fragment_table_start = sBlk_3.fragment_table_start_2; s_ops.squashfs_opendir = squashfs_opendir_1; s_ops.read_fragment = read_fragment_2; s_ops.read_fragment_table = read_fragment_table_2; s_ops.read_block_list = read_block_list_2; s_ops.read_inode = read_inode_2; s_ops.read_uids_guids = read_uids_guids_1; } } else if(sBlk.s.s_major == 3) { s_ops.squashfs_opendir = squashfs_opendir_3; s_ops.read_fragment = read_fragment_3; s_ops.read_fragment_table = read_fragment_table_3; s_ops.read_block_list = read_block_list_2; s_ops.read_inode = read_inode_3; s_ops.read_uids_guids = read_uids_guids_1; } else { ERROR("Filesystem on %s is (%d:%d), ", source, sBlk.s.s_major, sBlk.s.s_minor); ERROR("which is a later filesystem version than I support!\n"); goto failed_mount; } /* * 1.x, 2.x and 3.x filesystems use gzip compression. */ comp = lookup_compressor("gzip"); return TRUE; failed_mount: return FALSE; } struct pathname *process_extract_files(struct pathname *path, char *filename) { FILE *fd; char buffer[MAX_LINE + 1]; /* overflow safe */ char *name; fd = fopen(filename, "r"); if(fd == NULL) EXIT_UNSQUASH("Failed to open extract file \"%s\" because %s\n", filename, strerror(errno)); while(fgets(name = buffer, MAX_LINE + 1, fd) != NULL) { int len = strlen(name); if(len == MAX_LINE && name[len - 1] != '\n') /* line too large */ EXIT_UNSQUASH("Line too long when reading " "extract file \"%s\", larger than %d " "bytes\n", filename, MAX_LINE); /* * Remove '\n' terminator if it exists (the last line * in the file may not be '\n' terminated) */ if(len && name[len - 1] == '\n') name[len - 1] = '\0'; /* Skip any leading whitespace */ while(isspace(*name)) name ++; /* if comment line, skip */ if(*name == '#') continue; /* check for initial backslash, to accommodate * filenames with leading space or leading # character */ if(*name == '\\') name ++; /* if line is now empty after skipping characters, skip it */ if(*name == '\0') continue; path = add_path(path, name, name); } if(ferror(fd)) EXIT_UNSQUASH("Reading extract file \"%s\" failed because %s\n", filename, strerror(errno)); fclose(fd); return path; } /* * reader thread. This thread processes read requests queued by the * cache_get() routine. */ void *reader(void *arg) { while(1) { struct cache_entry *entry = queue_get(to_reader); int res = read_fs_bytes(fd, entry->block, SQUASHFS_COMPRESSED_SIZE_BLOCK(entry->size), entry->data); if(res && SQUASHFS_COMPRESSED_BLOCK(entry->size)) /* * queue successfully read block to the inflate * thread(s) for further processing */ queue_put(to_inflate, entry); else /* * block has either been successfully read and is * uncompressed, or an error has occurred, clear pending * flag, set error appropriately, and wake up any * threads waiting on this buffer */ cache_block_ready(entry, !res); } } /* * writer thread. This processes file write requests queued by the * write_file() routine. */ void *writer(void *arg) { int i; while(1) { struct squashfs_file *file = queue_get(to_writer); int file_fd; long long hole = 0; int failed = FALSE; int error; if(file == NULL) { queue_put(from_writer, NULL); continue; } else if(file->fd == -1) { /* write attributes for directory file->pathname */ set_attributes(file->pathname, file->mode, file->uid, file->gid, file->time, file->xattr, TRUE); free(file->pathname); free(file); continue; } TRACE("writer: regular file, blocks %d\n", file->blocks); file_fd = file->fd; for(i = 0; i < file->blocks; i++, cur_blocks ++) { struct file_entry *block = queue_get(to_writer); if(block->buffer == 0) { /* sparse file */ hole += block->size; free(block); continue; } cache_block_wait(block->buffer); if(block->buffer->error) failed = TRUE; if(failed) continue; error = write_block(file_fd, block->buffer->data + block->offset, block->size, hole, file->sparse); if(error == FALSE) { ERROR("writer: failed to write data block %d\n", i); failed = TRUE; } hole = 0; cache_block_put(block->buffer); free(block); } if(hole && failed == FALSE) { /* * corner case for hole extending to end of file */ if(file->sparse == FALSE || lseek(file_fd, hole, SEEK_CUR) == -1) { /* * for files which we don't want to write * sparsely, or for broken lseeks which cannot * seek beyond end of file, write_block will do * the right thing */ hole --; if(write_block(file_fd, "\0", 1, hole, file->sparse) == FALSE) { ERROR("writer: failed to write sparse " "data block\n"); failed = TRUE; } } else if(ftruncate(file_fd, file->file_size) == -1) { ERROR("writer: failed to write sparse data " "block\n"); failed = TRUE; } } close_wake(file_fd); if(failed == FALSE) set_attributes(file->pathname, file->mode, file->uid, file->gid, file->time, file->xattr, force); else { ERROR("Failed to write %s, skipping\n", file->pathname); unlink(file->pathname); } free(file->pathname); free(file); } } /* * decompress thread. This decompresses buffers queued by the read thread */ void *inflator(void *arg) { char tmp[block_size]; while(1) { struct cache_entry *entry = queue_get(to_inflate); int error, res; res = compressor_uncompress(comp, tmp, entry->data, SQUASHFS_COMPRESSED_SIZE_BLOCK(entry->size), block_size, &error); if(res == -1) ERROR("%s uncompress failed with error code %d\n", comp->name, error); else memcpy(entry->data, tmp, res); /* * block has been either successfully decompressed, or an error * occurred, clear pending flag, set error appropriately and * wake up any threads waiting on this block */ cache_block_ready(entry, res == -1); } } void *progress_thread(void *arg) { struct timespec requested_time, remaining; struct itimerval itimerval; struct winsize winsize; if(ioctl(1, TIOCGWINSZ, &winsize) == -1) { if(isatty(STDOUT_FILENO)) ERROR("TIOCGWINSZ ioctl failed, defaulting to 80 " "columns\n"); columns = 80; } else columns = winsize.ws_col; signal(SIGWINCH, sigwinch_handler); signal(SIGALRM, sigalrm_handler); itimerval.it_value.tv_sec = 0; itimerval.it_value.tv_usec = 250000; itimerval.it_interval.tv_sec = 0; itimerval.it_interval.tv_usec = 250000; setitimer(ITIMER_REAL, &itimerval, NULL); requested_time.tv_sec = 0; requested_time.tv_nsec = 250000000; while(1) { int res = nanosleep(&requested_time, &remaining); if(res == -1 && errno != EINTR) EXIT_UNSQUASH("nanosleep failed in progress thread\n"); if(progress_enabled) { pthread_mutex_lock(&screen_mutex); progress_bar(sym_count + dev_count + fifo_count + cur_blocks, total_inodes - total_files + total_blocks, columns); pthread_mutex_unlock(&screen_mutex); } } } void initialise_threads(int fragment_buffer_size, int data_buffer_size) { struct rlimit rlim; int i, max_files, res; sigset_t sigmask, old_mask; /* block SIGQUIT and SIGHUP, these are handled by the info thread */ sigemptyset(&sigmask); sigaddset(&sigmask, SIGQUIT); sigaddset(&sigmask, SIGHUP); if(pthread_sigmask(SIG_BLOCK, &sigmask, NULL) == -1) EXIT_UNSQUASH("Failed to set signal mask in initialise_threads" "\n"); /* * temporarily block these signals so the created sub-threads will * ignore them, ensuring the main thread handles them */ sigemptyset(&sigmask); sigaddset(&sigmask, SIGINT); sigaddset(&sigmask, SIGTERM); if(pthread_sigmask(SIG_BLOCK, &sigmask, &old_mask) == -1) EXIT_UNSQUASH("Failed to set signal mask in initialise_threads" "\n"); if(processors == -1) { #ifndef linux int mib[2]; size_t len = sizeof(processors); mib[0] = CTL_HW; #ifdef HW_AVAILCPU mib[1] = HW_AVAILCPU; #else mib[1] = HW_NCPU; #endif if(sysctl(mib, 2, &processors, &len, NULL, 0) == -1) { ERROR("Failed to get number of available processors. " "Defaulting to 1\n"); processors = 1; } #else processors = sysconf(_SC_NPROCESSORS_ONLN); #endif } if(add_overflow(processors, 3) || multiply_overflow(processors + 3, sizeof(pthread_t))) EXIT_UNSQUASH("Processors too large\n"); thread = malloc((3 + processors) * sizeof(pthread_t)); if(thread == NULL) EXIT_UNSQUASH("Out of memory allocating thread descriptors\n"); inflator_thread = &thread[3]; /* * dimensioning the to_reader and to_inflate queues. The size of * these queues is directly related to the amount of block * read-ahead possible. To_reader queues block read requests to * the reader thread and to_inflate queues block decompression * requests to the inflate thread(s) (once the block has been read by * the reader thread). The amount of read-ahead is determined by * the combined size of the data_block and fragment caches which * determine the total number of blocks which can be "in flight" * at any one time (either being read or being decompressed) * * The maximum file open limit, however, affects the read-ahead * possible, in that for normal sizes of the fragment and data block * caches, where the incoming files have few data blocks or one fragment * only, the file open limit is likely to be reached before the * caches are full. This means the worst case sizing of the combined * sizes of the caches is unlikely to ever be necessary. However, is is * obvious read-ahead up to the data block cache size is always possible * irrespective of the file open limit, because a single file could * contain that number of blocks. * * Choosing the size as "file open limit + data block cache size" seems * to be a reasonable estimate. We can reasonably assume the maximum * likely read-ahead possible is data block cache size + one fragment * per open file. * * dimensioning the to_writer queue. The size of this queue is * directly related to the amount of block read-ahead possible. * However, unlike the to_reader and to_inflate queues, this is * complicated by the fact the to_writer queue not only contains * entries for fragments and data_blocks but it also contains * file entries, one per open file in the read-ahead. * * Choosing the size as "2 * (file open limit) + * data block cache size" seems to be a reasonable estimate. * We can reasonably assume the maximum likely read-ahead possible * is data block cache size + one fragment per open file, and then * we will have a file_entry for each open file. */ res = getrlimit(RLIMIT_NOFILE, &rlim); if (res == -1) { ERROR("failed to get open file limit! Defaulting to 1\n"); rlim.rlim_cur = 1; } if (rlim.rlim_cur != RLIM_INFINITY) { /* * leave OPEN_FILE_MARGIN free (rlim_cur includes fds used by * stdin, stdout, stderr and filesystem fd */ if (rlim.rlim_cur <= OPEN_FILE_MARGIN) /* no margin, use minimum possible */ max_files = 1; else max_files = rlim.rlim_cur - OPEN_FILE_MARGIN; } else max_files = -1; /* set amount of available files for use by open_wait and close_wake */ open_init(max_files); /* * allocate to_reader, to_inflate and to_writer queues. Set based on * open file limit and cache size, unless open file limit is unlimited, * in which case set purely based on cache limits * * In doing so, check that the user supplied values do not overflow * a signed int */ if (max_files != -1) { if(add_overflow(data_buffer_size, max_files) || add_overflow(data_buffer_size, max_files * 2)) EXIT_UNSQUASH("Data queue size is too large\n"); to_reader = queue_init(max_files + data_buffer_size); to_inflate = queue_init(max_files + data_buffer_size); to_writer = queue_init(max_files * 2 + data_buffer_size); } else { int all_buffers_size; if(add_overflow(fragment_buffer_size, data_buffer_size)) EXIT_UNSQUASH("Data and fragment queues combined are" " too large\n"); all_buffers_size = fragment_buffer_size + data_buffer_size; if(add_overflow(all_buffers_size, all_buffers_size)) EXIT_UNSQUASH("Data and fragment queues combined are" " too large\n"); to_reader = queue_init(all_buffers_size); to_inflate = queue_init(all_buffers_size); to_writer = queue_init(all_buffers_size * 2); } from_writer = queue_init(1); fragment_cache = cache_init(block_size, fragment_buffer_size); data_cache = cache_init(block_size, data_buffer_size); pthread_create(&thread[0], NULL, reader, NULL); pthread_create(&thread[1], NULL, writer, NULL); pthread_create(&thread[2], NULL, progress_thread, NULL); init_info(); pthread_mutex_init(&fragment_mutex, NULL); for(i = 0; i < processors; i++) { if(pthread_create(&inflator_thread[i], NULL, inflator, NULL) != 0) EXIT_UNSQUASH("Failed to create thread\n"); } printf("Parallel unsquashfs: Using %d processor%s\n", processors, processors == 1 ? "" : "s"); if(pthread_sigmask(SIG_SETMASK, &old_mask, NULL) == -1) EXIT_UNSQUASH("Failed to set signal mask in initialise_threads" "\n"); } void enable_progress_bar() { pthread_mutex_lock(&screen_mutex); progress_enabled = progress; pthread_mutex_unlock(&screen_mutex); } void disable_progress_bar() { pthread_mutex_lock(&screen_mutex); if(progress_enabled) { progress_bar(sym_count + dev_count + fifo_count + cur_blocks, total_inodes - total_files + total_blocks, columns); printf("\n"); } progress_enabled = FALSE; pthread_mutex_unlock(&screen_mutex); } void progressbar_error(char *fmt, ...) { va_list ap; pthread_mutex_lock(&screen_mutex); if(progress_enabled) fprintf(stderr, "\n"); va_start(ap, fmt); vfprintf(stderr, fmt, ap); va_end(ap); pthread_mutex_unlock(&screen_mutex); } void progressbar_info(char *fmt, ...) { va_list ap; pthread_mutex_lock(&screen_mutex); if(progress_enabled) printf("\n"); va_start(ap, fmt); vprintf(fmt, ap); va_end(ap); pthread_mutex_unlock(&screen_mutex); } void progress_bar(long long current, long long max, int columns) { char rotate_list[] = { '|', '/', '-', '\\' }; int max_digits, used, hashes, spaces; static int tty = -1; if(max == 0) return; max_digits = floor(log10(max)) + 1; used = max_digits * 2 + 11; hashes = (current * (columns - used)) / max; spaces = columns - used - hashes; if((current > max) || (columns - used < 0)) return; if(tty == -1) tty = isatty(STDOUT_FILENO); if(!tty) { static long long previous = -1; /* * Updating much more frequently than this results in huge * log files. */ if((current % 100) != 0 && current != max) return; /* Don't update just to rotate the spinner. */ if(current == previous) return; previous = current; } printf("\r["); while (hashes --) putchar('='); putchar(rotate_list[rotate]); while(spaces --) putchar(' '); printf("] %*lld/%*lld", max_digits, current, max_digits, max); printf(" %3lld%%", current * 100 / max); fflush(stdout); } int parse_number(char *arg, int *res) { char *b; long number = strtol(arg, &b, 10); /* check for trailing junk after number */ if(*b != '\0') return 0; /* * check for strtol underflow or overflow in conversion. * Note: strtol can validly return LONG_MIN and LONG_MAX * if the user entered these values, but, additional code * to distinguish this scenario is unnecessary, because for * our purposes LONG_MIN and LONG_MAX are too large anyway */ if(number == LONG_MIN || number == LONG_MAX) return 0; /* reject negative numbers as invalid */ if(number < 0) return 0; /* check if long result will overflow signed int */ if(number > INT_MAX) return 0; *res = number; return 1; } #define VERSION() \ printf("unsquashfs version 4.3 (2014/05/12)\n");\ printf("copyright (C) 2014 Phillip Lougher "\ "\n\n");\ printf("This program is free software; you can redistribute it and/or"\ "\n");\ printf("modify it under the terms of the GNU General Public License"\ "\n");\ printf("as published by the Free Software Foundation; either version "\ "2,\n");\ printf("or (at your option) any later version.\n\n");\ printf("This program is distributed in the hope that it will be "\ "useful,\n");\ printf("but WITHOUT ANY WARRANTY; without even the implied warranty of"\ "\n");\ printf("MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the"\ "\n");\ printf("GNU General Public License for more details.\n"); int main(int argc, char *argv[]) { char *dest = "squashfs-root"; int i, stat_sys = FALSE, version = FALSE; int n; struct pathnames *paths = NULL; struct pathname *path = NULL; long long directory_table_end; int fragment_buffer_size = FRAGMENT_BUFFER_DEFAULT; int data_buffer_size = DATA_BUFFER_DEFAULT; pthread_mutex_init(&screen_mutex, NULL); root_process = geteuid() == 0; if(root_process) umask(0); for(i = 1; i < argc; i++) { if(*argv[i] != '-') break; if(strcmp(argv[i], "-version") == 0 || strcmp(argv[i], "-v") == 0) { VERSION(); version = TRUE; } else if(strcmp(argv[i], "-info") == 0 || strcmp(argv[i], "-i") == 0) info = TRUE; else if(strcmp(argv[i], "-ls") == 0 || strcmp(argv[i], "-l") == 0) lsonly = TRUE; else if(strcmp(argv[i], "-no-progress") == 0 || strcmp(argv[i], "-n") == 0) progress = FALSE; else if(strcmp(argv[i], "-no-xattrs") == 0 || strcmp(argv[i], "-no") == 0) no_xattrs = TRUE; else if(strcmp(argv[i], "-xattrs") == 0 || strcmp(argv[i], "-x") == 0) no_xattrs = FALSE; else if(strcmp(argv[i], "-user-xattrs") == 0 || strcmp(argv[i], "-u") == 0) { user_xattrs = TRUE; no_xattrs = FALSE; } else if(strcmp(argv[i], "-dest") == 0 || strcmp(argv[i], "-d") == 0) { if(++i == argc) { fprintf(stderr, "%s: -dest missing filename\n", argv[0]); exit(1); } dest = argv[i]; } else if(strcmp(argv[i], "-processors") == 0 || strcmp(argv[i], "-p") == 0) { if((++i == argc) || !parse_number(argv[i], &processors)) { ERROR("%s: -processors missing or invalid " "processor number\n", argv[0]); exit(1); } if(processors < 1) { ERROR("%s: -processors should be 1 or larger\n", argv[0]); exit(1); } } else if(strcmp(argv[i], "-data-queue") == 0 || strcmp(argv[i], "-da") == 0) { if((++i == argc) || !parse_number(argv[i], &data_buffer_size)) { ERROR("%s: -data-queue missing or invalid " "queue size\n", argv[0]); exit(1); } if(data_buffer_size < 1) { ERROR("%s: -data-queue should be 1 Mbyte or " "larger\n", argv[0]); exit(1); } } else if(strcmp(argv[i], "-frag-queue") == 0 || strcmp(argv[i], "-fr") == 0) { if((++i == argc) || !parse_number(argv[i], &fragment_buffer_size)) { ERROR("%s: -frag-queue missing or invalid " "queue size\n", argv[0]); exit(1); } if(fragment_buffer_size < 1) { ERROR("%s: -frag-queue should be 1 Mbyte or " "larger\n", argv[0]); exit(1); } } else if(strcmp(argv[i], "-force") == 0 || strcmp(argv[i], "-f") == 0) force = TRUE; else if(strcmp(argv[i], "-stat") == 0 || strcmp(argv[i], "-s") == 0) stat_sys = TRUE; else if(strcmp(argv[i], "-lls") == 0 || strcmp(argv[i], "-ll") == 0) { lsonly = TRUE; short_ls = FALSE; } else if(strcmp(argv[i], "-linfo") == 0 || strcmp(argv[i], "-li") == 0) { info = TRUE; short_ls = FALSE; } else if(strcmp(argv[i], "-ef") == 0 || strcmp(argv[i], "-e") == 0) { if(++i == argc) { fprintf(stderr, "%s: -ef missing filename\n", argv[0]); exit(1); } path = process_extract_files(path, argv[i]); } else if(strcmp(argv[i], "-regex") == 0 || strcmp(argv[i], "-r") == 0) use_regex = TRUE; else goto options; } if(lsonly || info) progress = FALSE; #ifdef SQUASHFS_TRACE /* * Disable progress bar if full debug tracing is enabled. * The progress bar in this case just gets in the way of the * debug trace output */ progress = FALSE; #endif if(i == argc) { if(!version) { options: ERROR("SYNTAX: %s [options] filesystem [directories or " "files to extract]\n", argv[0]); ERROR("\t-v[ersion]\t\tprint version, licence and " "copyright information\n"); ERROR("\t-d[est] \tunsquash to , " "default \"squashfs-root\"\n"); ERROR("\t-n[o-progress]\t\tdon't display the progress " "bar\n"); ERROR("\t-no[-xattrs]\t\tdon't extract xattrs in file system" NOXOPT_STR"\n"); ERROR("\t-x[attrs]\t\textract xattrs in file system" XOPT_STR "\n"); ERROR("\t-u[ser-xattrs]\t\tonly extract user xattrs in " "file system.\n\t\t\t\tEnables extracting " "xattrs\n"); ERROR("\t-p[rocessors] \tuse " "processors. By default will use\n"); ERROR("\t\t\t\tnumber of processors available\n"); ERROR("\t-i[nfo]\t\t\tprint files as they are " "unsquashed\n"); ERROR("\t-li[nfo]\t\tprint files as they are " "unsquashed with file\n"); ERROR("\t\t\t\tattributes (like ls -l output)\n"); ERROR("\t-l[s]\t\t\tlist filesystem, but don't unsquash" "\n"); ERROR("\t-ll[s]\t\t\tlist filesystem with file " "attributes (like\n"); ERROR("\t\t\t\tls -l output), but don't unsquash\n"); ERROR("\t-f[orce]\t\tif file already exists then " "overwrite\n"); ERROR("\t-s[tat]\t\t\tdisplay filesystem superblock " "information\n"); ERROR("\t-e[f] \tlist of directories or " "files to extract.\n\t\t\t\tOne per line\n"); ERROR("\t-da[ta-queue] \tSet data queue to " " Mbytes. Default %d\n\t\t\t\tMbytes\n", DATA_BUFFER_DEFAULT); ERROR("\t-fr[ag-queue] \tSet fragment queue to " " Mbytes. Default\n\t\t\t\t%d Mbytes\n", FRAGMENT_BUFFER_DEFAULT); ERROR("\t-r[egex]\t\ttreat extract names as POSIX " "regular expressions\n"); ERROR("\t\t\t\trather than use the default shell " "wildcard\n\t\t\t\texpansion (globbing)\n"); ERROR("\nDecompressors available:\n"); display_compressors("", ""); } exit(1); } for(n = i + 1; n < argc; n++) path = add_path(path, argv[n], argv[n]); if((fd = open(argv[i], O_RDONLY)) == -1) { ERROR("Could not open %s, because %s\n", argv[i], strerror(errno)); exit(1); } if(read_super(argv[i]) == FALSE) exit(1); if(stat_sys) { squashfs_stat(argv[i]); exit(0); } if(!check_compression(comp)) exit(1); block_size = sBlk.s.block_size; block_log = sBlk.s.block_log; /* * Sanity check block size and block log. * * Check they're within correct limits */ if(block_size > SQUASHFS_FILE_MAX_SIZE || block_log > SQUASHFS_FILE_MAX_LOG) EXIT_UNSQUASH("Block size or block_log too large." " File system is corrupt.\n"); /* * Check block_size and block_log match */ if(block_size != (1 << block_log)) EXIT_UNSQUASH("Block size and block_log do not match." " File system is corrupt.\n"); /* * convert from queue size in Mbytes to queue size in * blocks. * * In doing so, check that the user supplied values do not * overflow a signed int */ if(shift_overflow(fragment_buffer_size, 20 - block_log)) EXIT_UNSQUASH("Fragment queue size is too large\n"); else fragment_buffer_size <<= 20 - block_log; if(shift_overflow(data_buffer_size, 20 - block_log)) EXIT_UNSQUASH("Data queue size is too large\n"); else data_buffer_size <<= 20 - block_log; initialise_threads(fragment_buffer_size, data_buffer_size); fragment_data = malloc(block_size); if(fragment_data == NULL) EXIT_UNSQUASH("failed to allocate fragment_data\n"); file_data = malloc(block_size); if(file_data == NULL) EXIT_UNSQUASH("failed to allocate file_data"); data = malloc(block_size); if(data == NULL) EXIT_UNSQUASH("failed to allocate data\n"); created_inode = malloc(sBlk.s.inodes * sizeof(char *)); if(created_inode == NULL) EXIT_UNSQUASH("failed to allocate created_inode\n"); memset(created_inode, 0, sBlk.s.inodes * sizeof(char *)); if(s_ops.read_uids_guids() == FALSE) EXIT_UNSQUASH("failed to uid/gid table\n"); if(s_ops.read_fragment_table(&directory_table_end) == FALSE) EXIT_UNSQUASH("failed to read fragment table\n"); if(read_inode_table(sBlk.s.inode_table_start, sBlk.s.directory_table_start) == FALSE) EXIT_UNSQUASH("failed to read inode table\n"); if(read_directory_table(sBlk.s.directory_table_start, directory_table_end) == FALSE) EXIT_UNSQUASH("failed to read directory table\n"); if(no_xattrs) sBlk.s.xattr_id_table_start = SQUASHFS_INVALID_BLK; if(read_xattrs_from_disk(fd, &sBlk.s) == 0) EXIT_UNSQUASH("failed to read the xattr table\n"); if(path) { paths = init_subdir(); paths = add_subdir(paths, path); } pre_scan(dest, SQUASHFS_INODE_BLK(sBlk.s.root_inode), SQUASHFS_INODE_OFFSET(sBlk.s.root_inode), paths); memset(created_inode, 0, sBlk.s.inodes * sizeof(char *)); inode_number = 1; printf("%d inodes (%d blocks) to write\n\n", total_inodes, total_inodes - total_files + total_blocks); enable_progress_bar(); dir_scan(dest, SQUASHFS_INODE_BLK(sBlk.s.root_inode), SQUASHFS_INODE_OFFSET(sBlk.s.root_inode), paths); queue_put(to_writer, NULL); queue_get(from_writer); disable_progress_bar(); if(!lsonly) { printf("\n"); printf("created %d files\n", file_count); printf("created %d directories\n", dir_count); printf("created %d symlinks\n", sym_count); printf("created %d devices\n", dev_count); printf("created %d fifos\n", fifo_count); } return 0; } squashfs4.3/squashfs-tools/action.h0000644000175000017500000001266012333330365017453 0ustar phillipphillip#ifndef ACTION_H #define ACTION_H /* * Create a squashfs filesystem. This is a highly compressed read only * filesystem. * * Copyright (c) 2011, 2012, 2013, 2014 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * action.h */ /* * Lexical analyser definitions */ #define TOK_OPEN_BRACKET 0 #define TOK_CLOSE_BRACKET 1 #define TOK_AND 2 #define TOK_OR 3 #define TOK_NOT 4 #define TOK_COMMA 5 #define TOK_AT 6 #define TOK_WHITE_SPACE 7 #define TOK_STRING 8 #define TOK_EOF 9 #define TOK_TO_STR(OP, S) ({ \ char *s; \ switch(OP) { \ case TOK_EOF: \ s = "EOF"; \ break; \ case TOK_STRING: \ s = S; \ break; \ default: \ s = token_table[OP].string; \ break; \ } \ s; \ }) struct token_entry { char *string; int token; int size; }; /* * Expression parser definitions */ #define OP_TYPE 0 #define ATOM_TYPE 1 #define UNARY_TYPE 2 #define SYNTAX_ERROR(S, ARGS...) { \ char *src = strdup(source); \ src[cur_ptr - source] = '\0'; \ fprintf(stderr, "Failed to parse action \"%s\"\n", source); \ fprintf(stderr, "Syntax error: "S, ##ARGS); \ fprintf(stderr, "Got here \"%s\"\n", src); \ free(src); \ } #define TEST_SYNTAX_ERROR(TEST, ARG, S, ARGS...) { \ char *src = strdup(source); \ src[cur_ptr - source] = '\0'; \ fprintf(stderr, "Failed to parse action \"%s\"\n", source); \ fprintf(stderr, "Syntax error in \"%s()\", arg %d: "S, TEST->name, \ ARG, ##ARGS); \ fprintf(stderr, "Got here \"%s\"\n", src); \ free(src); \ } struct expr; struct expr_op { struct expr *lhs; struct expr *rhs; int op; }; struct atom { struct test_entry *test; char **argv; void *data; }; struct unary_op { struct expr *expr; int op; }; struct expr { int type; union { struct atom atom; struct expr_op expr_op; struct unary_op unary_op; }; }; /* * Test operation definitions */ #define NUM_EQ 1 #define NUM_LESS 2 #define NUM_GREATER 3 struct test_number_arg { long long size; int range; }; struct test_range_args { long long start; long long end; }; struct action; struct action_data; struct test_entry { char *name; int args; int (*fn)(struct atom *, struct action_data *); int (*parse_args)(struct test_entry *, struct atom *); }; /* * Type test specific definitions */ struct type_entry { int value; char type; }; /* * Action definitions */ #define FRAGMENT_ACTION 0 #define EXCLUDE_ACTION 1 #define FRAGMENTS_ACTION 2 #define NO_FRAGMENTS_ACTION 3 #define ALWAYS_FRAGS_ACTION 4 #define NO_ALWAYS_FRAGS_ACTION 5 #define COMPRESSED_ACTION 6 #define UNCOMPRESSED_ACTION 7 #define UID_ACTION 8 #define GID_ACTION 9 #define GUID_ACTION 10 #define MODE_ACTION 11 #define EMPTY_ACTION 12 #define MOVE_ACTION 13 /* * Define what file types each action operates over */ #define ACTION_DIR S_IFDIR #define ACTION_REG S_IFREG #define ACTION_ALL_LNK (S_IFDIR | S_IFREG | S_IFBLK | S_IFCHR | S_IFSOCK | \ S_IFIFO | S_IFLNK) #define ACTION_ALL (S_IFDIR | S_IFREG | S_IFBLK | S_IFCHR | S_IFSOCK | S_IFIFO) struct action_entry { char *name; int type; int args; int file_types; int (*parse_args)(struct action_entry *, int, char **, void **); void (*run_action)(struct action *, struct dir_ent *); }; struct action_data { int depth; char *name; char *pathname; char *subpath; struct stat *buf; }; struct action { int type; struct action_entry *action; int args; char **argv; struct expr *expr; void *data; }; /* * Uid/gid action specific definitions */ struct uid_info { uid_t uid; }; struct gid_info { gid_t gid; }; struct guid_info { uid_t uid; gid_t gid; }; /* * Mode action specific definitions */ #define ACTION_MODE_SET 0 #define ACTION_MODE_ADD 1 #define ACTION_MODE_REM 2 #define ACTION_MODE_OCT 3 struct mode_data { struct mode_data *next; int operation; int mode; unsigned int mask; char X; }; /* * Empty action specific definitions */ #define EMPTY_ALL 0 #define EMPTY_SOURCE 1 #define EMPTY_EXCLUDED 2 struct empty_data { int val; }; /* * Move action specific definitions */ #define ACTION_MOVE_RENAME 1 #define ACTION_MOVE_MOVE 2 struct move_ent { int ops; struct dir_ent *dir_ent; char *name; struct dir_info *dest; struct move_ent *next; }; /* * External function definitions */ extern int parse_action(char *); extern void dump_actions(); extern void *eval_frag_actions(struct dir_ent *); extern void *get_frag_action(void *); extern int eval_exclude_actions(char *, char *, char *, struct stat *, int); extern void eval_actions(struct dir_ent *); extern int eval_empty_actions(struct dir_ent *dir_ent); extern void eval_move_actions(struct dir_info *, struct dir_ent *); extern void do_move_actions(); extern int read_bytes(int, void *, int); extern int actions(); extern int move_actions(); extern int empty_actions(); extern int read_action_file(char *); extern int exclude_actions(); #endif squashfs4.3/squashfs-tools/lz4_wrapper.c0000644000175000017500000001634512306776316020460 0ustar phillipphillip/* * Copyright (c) 2013 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * lz4_wrapper.c * * Support for LZ4 compression http://fastcompression.blogspot.com/p/lz4.html */ #include #include #include #include #include #include "squashfs_fs.h" #include "lz4_wrapper.h" #include "compressor.h" static int hc = 0; /* * This function is called by the options parsing code in mksquashfs.c * to parse any -X compressor option. * * This function returns: * >=0 (number of additional args parsed) on success * -1 if the option was unrecognised, or * -2 if the option was recognised, but otherwise bad in * some way (e.g. invalid parameter) * * Note: this function sets internal compressor state, but does not * pass back the results of the parsing other than success/failure. * The lz4_dump_options() function is called later to get the options in * a format suitable for writing to the filesystem. */ static int lz4_options(char *argv[], int argc) { if(strcmp(argv[0], "-Xhc") == 0) { hc = 1; return 0; } return -1; } /* * This function is called by mksquashfs to dump the parsed * compressor options in a format suitable for writing to the * compressor options field in the filesystem (stored immediately * after the superblock). * * This function returns a pointer to the compression options structure * to be stored (and the size), or NULL if there are no compression * options * * Currently LZ4 always returns a comp_opts structure, with * the version indicating LZ4_LEGACY stream fomat. This is to * easily accomodate changes in the kernel code to different * stream formats */ static void *lz4_dump_options(int block_size, int *size) { static struct lz4_comp_opts comp_opts; comp_opts.version = LZ4_LEGACY; comp_opts.flags = hc ? LZ4_HC : 0; SQUASHFS_INSWAP_COMP_OPTS(&comp_opts); *size = sizeof(comp_opts); return &comp_opts; } /* * This function is a helper specifically for the append mode of * mksquashfs. Its purpose is to set the internal compressor state * to the stored compressor options in the passed compressor options * structure. * * In effect this function sets up the compressor options * to the same state they were when the filesystem was originally * generated, this is to ensure on appending, the compressor uses * the same compression options that were used to generate the * original filesystem. * * Note, even if there are no compressor options, this function is still * called with an empty compressor structure (size == 0), to explicitly * set the default options, this is to ensure any user supplied * -X options on the appending mksquashfs command line are over-ridden * * This function returns 0 on sucessful extraction of options, and * -1 on error */ static int lz4_extract_options(int block_size, void *buffer, int size) { struct lz4_comp_opts *comp_opts = buffer; /* we expect a comp_opts structure to be present */ if(size < sizeof(*comp_opts)) goto failed; SQUASHFS_INSWAP_COMP_OPTS(comp_opts); /* we expect the stream format to be LZ4_LEGACY */ if(comp_opts->version != LZ4_LEGACY) { fprintf(stderr, "lz4: unknown LZ4 version\n"); goto failed; } /* * Check compression flags, currently only LZ4_HC ("high compression") * can be set. */ if(comp_opts->flags == LZ4_HC) hc = 1; else if(comp_opts->flags != 0) { fprintf(stderr, "lz4: unknown LZ4 flags\n"); goto failed; } return 0; failed: fprintf(stderr, "lz4: error reading stored compressor options from " "filesystem!\n"); return -1; } /* * This function is a helper specifically for unsquashfs. * Its purpose is to check that the compression options are * understood by this version of LZ4. * * This is important for LZ4 because the format understood by the * Linux kernel may change from the already obsolete legacy format * currently supported. * * If this does happen, then this version of LZ4 will not be able to decode * the newer format. So we need to check for this. * * This function returns 0 on sucessful checking of options, and * -1 on error */ static int lz4_check_options(int block_size, void *buffer, int size) { struct lz4_comp_opts *comp_opts = buffer; /* we expect a comp_opts structure to be present */ if(size < sizeof(*comp_opts)) goto failed; SQUASHFS_INSWAP_COMP_OPTS(comp_opts); /* we expect the stream format to be LZ4_LEGACY */ if(comp_opts->version != LZ4_LEGACY) { fprintf(stderr, "lz4: unknown LZ4 version\n"); goto failed; } return 0; failed: fprintf(stderr, "lz4: error reading stored compressor options from " "filesystem!\n"); return -1; } void lz4_display_options(void *buffer, int size) { struct lz4_comp_opts *comp_opts = buffer; /* check passed comp opts struct is of the correct length */ if(size < sizeof(*comp_opts)) goto failed; SQUASHFS_INSWAP_COMP_OPTS(comp_opts); /* we expect the stream format to be LZ4_LEGACY */ if(comp_opts->version != LZ4_LEGACY) { fprintf(stderr, "lz4: unknown LZ4 version\n"); goto failed; } /* * Check compression flags, currently only LZ4_HC ("high compression") * can be set. */ if(comp_opts->flags & ~LZ4_FLAGS_MASK) { fprintf(stderr, "lz4: unknown LZ4 flags\n"); goto failed; } if(comp_opts->flags & LZ4_HC) printf("\tHigh Compression option specified (-Xhc)\n"); return; failed: fprintf(stderr, "lz4: error reading stored compressor options from " "filesystem!\n"); } static int lz4_compress(void *strm, void *dest, void *src, int size, int block_size, int *error) { int res; if(hc) res = LZ4_compressHC_limitedOutput(src, dest, size, block_size); else res = LZ4_compress_limitedOutput(src, dest, size, block_size); if(res == 0) { /* * Output buffer overflow. Return out of buffer space */ return 0; } else if(res < 0) { /* * All other errors return failure, with the compressor * specific error code in *error */ *error = res; return -1; } return res; } static int lz4_uncompress(void *dest, void *src, int size, int outsize, int *error) { int res = LZ4_decompress_safe(src, dest, size, outsize); if(res < 0) { *error = res; return -1; } return res; } void lz4_usage() { fprintf(stderr, "\t -Xhc\n"); fprintf(stderr, "\t\tCompress using LZ4 High Compression\n"); } struct compressor lz4_comp_ops = { .compress = lz4_compress, .uncompress = lz4_uncompress, .options = lz4_options, .dump_options = lz4_dump_options, .extract_options = lz4_extract_options, .check_options = lz4_check_options, .display_options = lz4_display_options, .usage = lz4_usage, .id = LZ4_COMPRESSION, .name = "lz4", .supported = 1 }; squashfs4.3/squashfs-tools/unsquash-1.c0000644000175000017500000002330012306776317020202 0ustar phillipphillip/* * Unsquash a squashfs filesystem. This is a highly compressed read only * filesystem. * * Copyright (c) 2009, 2010, 2011, 2012 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * unsquash-1.c */ #include "unsquashfs.h" #include "squashfs_compat.h" void read_block_list_1(unsigned int *block_list, char *block_ptr, int blocks) { unsigned short block_size; int i; TRACE("read_block_list: blocks %d\n", blocks); for(i = 0; i < blocks; i++, block_ptr += 2) { if(swap) { unsigned short sblock_size; memcpy(&sblock_size, block_ptr, sizeof(unsigned short)); SQUASHFS_SWAP_SHORTS_3((&block_size), &sblock_size, 1); } else memcpy(&block_size, block_ptr, sizeof(unsigned short)); block_list[i] = SQUASHFS_COMPRESSED_SIZE(block_size) | (SQUASHFS_COMPRESSED(block_size) ? 0 : SQUASHFS_COMPRESSED_BIT_BLOCK); } } int read_fragment_table_1(long long *directory_table_end) { TRACE("read_fragment_table\n"); *directory_table_end = sBlk.s.fragment_table_start; return TRUE; } struct inode *read_inode_1(unsigned int start_block, unsigned int offset) { static union squashfs_inode_header_1 header; long long start = sBlk.s.inode_table_start + start_block; int bytes = lookup_entry(inode_table_hash, start); char *block_ptr = inode_table + bytes + offset; static struct inode i; TRACE("read_inode: reading inode [%d:%d]\n", start_block, offset); if(bytes == -1) EXIT_UNSQUASH("read_inode: inode table block %lld not found\n", start); if(swap) { squashfs_base_inode_header_1 sinode; memcpy(&sinode, block_ptr, sizeof(header.base)); SQUASHFS_SWAP_BASE_INODE_HEADER_1(&header.base, &sinode, sizeof(squashfs_base_inode_header_1)); } else memcpy(&header.base, block_ptr, sizeof(header.base)); i.uid = (uid_t) uid_table[(header.base.inode_type - 1) / SQUASHFS_TYPES * 16 + header.base.uid]; if(header.base.inode_type == SQUASHFS_IPC_TYPE) { squashfs_ipc_inode_header_1 *inodep = &header.ipc; if(swap) { squashfs_ipc_inode_header_1 sinodep; memcpy(&sinodep, block_ptr, sizeof(sinodep)); SQUASHFS_SWAP_IPC_INODE_HEADER_1(inodep, &sinodep); } else memcpy(inodep, block_ptr, sizeof(*inodep)); if(inodep->type == SQUASHFS_SOCKET_TYPE) { i.mode = S_IFSOCK | header.base.mode; i.type = SQUASHFS_SOCKET_TYPE; } else { i.mode = S_IFIFO | header.base.mode; i.type = SQUASHFS_FIFO_TYPE; } i.uid = (uid_t) uid_table[inodep->offset * 16 + inodep->uid]; } else { i.mode = lookup_type[(header.base.inode_type - 1) % SQUASHFS_TYPES + 1] | header.base.mode; i.type = (header.base.inode_type - 1) % SQUASHFS_TYPES + 1; } i.xattr = SQUASHFS_INVALID_XATTR; i.gid = header.base.guid == 15 ? i.uid : (uid_t) guid_table[header.base.guid]; i.time = sBlk.s.mkfs_time; i.inode_number = inode_number ++; switch(i.type) { case SQUASHFS_DIR_TYPE: { squashfs_dir_inode_header_1 *inode = &header.dir; if(swap) { squashfs_dir_inode_header_1 sinode; memcpy(&sinode, block_ptr, sizeof(header.dir)); SQUASHFS_SWAP_DIR_INODE_HEADER_1(inode, &sinode); } else memcpy(inode, block_ptr, sizeof(header.dir)); i.data = inode->file_size; i.offset = inode->offset; i.start = inode->start_block; i.time = inode->mtime; break; } case SQUASHFS_FILE_TYPE: { squashfs_reg_inode_header_1 *inode = &header.reg; if(swap) { squashfs_reg_inode_header_1 sinode; memcpy(&sinode, block_ptr, sizeof(sinode)); SQUASHFS_SWAP_REG_INODE_HEADER_1(inode, &sinode); } else memcpy(inode, block_ptr, sizeof(*inode)); i.data = inode->file_size; i.time = inode->mtime; i.blocks = (i.data + sBlk.s.block_size - 1) >> sBlk.s.block_log; i.start = inode->start_block; i.block_ptr = block_ptr + sizeof(*inode); i.fragment = 0; i.frag_bytes = 0; i.offset = 0; i.sparse = 0; break; } case SQUASHFS_SYMLINK_TYPE: { squashfs_symlink_inode_header_1 *inodep = &header.symlink; if(swap) { squashfs_symlink_inode_header_1 sinodep; memcpy(&sinodep, block_ptr, sizeof(sinodep)); SQUASHFS_SWAP_SYMLINK_INODE_HEADER_1(inodep, &sinodep); } else memcpy(inodep, block_ptr, sizeof(*inodep)); i.symlink = malloc(inodep->symlink_size + 1); if(i.symlink == NULL) EXIT_UNSQUASH("read_inode: failed to malloc " "symlink data\n"); strncpy(i.symlink, block_ptr + sizeof(squashfs_symlink_inode_header_1), inodep->symlink_size); i.symlink[inodep->symlink_size] = '\0'; i.data = inodep->symlink_size; break; } case SQUASHFS_BLKDEV_TYPE: case SQUASHFS_CHRDEV_TYPE: { squashfs_dev_inode_header_1 *inodep = &header.dev; if(swap) { squashfs_dev_inode_header_1 sinodep; memcpy(&sinodep, block_ptr, sizeof(sinodep)); SQUASHFS_SWAP_DEV_INODE_HEADER_1(inodep, &sinodep); } else memcpy(inodep, block_ptr, sizeof(*inodep)); i.data = inodep->rdev; break; } case SQUASHFS_FIFO_TYPE: case SQUASHFS_SOCKET_TYPE: { i.data = 0; break; } default: EXIT_UNSQUASH("Unknown inode type %d in " " read_inode_header_1!\n", header.base.inode_type); } return &i; } struct dir *squashfs_opendir_1(unsigned int block_start, unsigned int offset, struct inode **i) { squashfs_dir_header_2 dirh; char buffer[sizeof(squashfs_dir_entry_2) + SQUASHFS_NAME_LEN + 1] __attribute__((aligned)); squashfs_dir_entry_2 *dire = (squashfs_dir_entry_2 *) buffer; long long start; int bytes; int dir_count, size; struct dir_ent *new_dir; struct dir *dir; TRACE("squashfs_opendir: inode start block %d, offset %d\n", block_start, offset); *i = s_ops.read_inode(block_start, offset); dir = malloc(sizeof(struct dir)); if(dir == NULL) EXIT_UNSQUASH("squashfs_opendir: malloc failed!\n"); dir->dir_count = 0; dir->cur_entry = 0; dir->mode = (*i)->mode; dir->uid = (*i)->uid; dir->guid = (*i)->gid; dir->mtime = (*i)->time; dir->xattr = (*i)->xattr; dir->dirs = NULL; if ((*i)->data == 0) /* * if the directory is empty, skip the unnecessary * lookup_entry, this fixes the corner case with * completely empty filesystems where lookup_entry correctly * returning -1 is incorrectly treated as an error */ return dir; start = sBlk.s.directory_table_start + (*i)->start; bytes = lookup_entry(directory_table_hash, start); if(bytes == -1) EXIT_UNSQUASH("squashfs_opendir: directory block %d not " "found!\n", block_start); bytes += (*i)->offset; size = (*i)->data + bytes; while(bytes < size) { if(swap) { squashfs_dir_header_2 sdirh; memcpy(&sdirh, directory_table + bytes, sizeof(sdirh)); SQUASHFS_SWAP_DIR_HEADER_2(&dirh, &sdirh); } else memcpy(&dirh, directory_table + bytes, sizeof(dirh)); dir_count = dirh.count + 1; TRACE("squashfs_opendir: Read directory header @ byte position " "%d, %d directory entries\n", bytes, dir_count); bytes += sizeof(dirh); /* dir_count should never be larger than 256 */ if(dir_count > 256) goto corrupted; while(dir_count--) { if(swap) { squashfs_dir_entry_2 sdire; memcpy(&sdire, directory_table + bytes, sizeof(sdire)); SQUASHFS_SWAP_DIR_ENTRY_2(dire, &sdire); } else memcpy(dire, directory_table + bytes, sizeof(*dire)); bytes += sizeof(*dire); /* size should never be larger than SQUASHFS_NAME_LEN */ if(dire->size > SQUASHFS_NAME_LEN) goto corrupted; memcpy(dire->name, directory_table + bytes, dire->size + 1); dire->name[dire->size + 1] = '\0'; TRACE("squashfs_opendir: directory entry %s, inode " "%d:%d, type %d\n", dire->name, dirh.start_block, dire->offset, dire->type); if((dir->dir_count % DIR_ENT_SIZE) == 0) { new_dir = realloc(dir->dirs, (dir->dir_count + DIR_ENT_SIZE) * sizeof(struct dir_ent)); if(new_dir == NULL) EXIT_UNSQUASH("squashfs_opendir: " "realloc failed!\n"); dir->dirs = new_dir; } strcpy(dir->dirs[dir->dir_count].name, dire->name); dir->dirs[dir->dir_count].start_block = dirh.start_block; dir->dirs[dir->dir_count].offset = dire->offset; dir->dirs[dir->dir_count].type = dire->type; dir->dir_count ++; bytes += dire->size + 1; } } return dir; corrupted: free(dir->dirs); free(dir); return NULL; } int read_uids_guids_1() { int res; TRACE("read_uids_guids: no_uids %d, no_guids %d\n", sBlk.no_uids, sBlk.no_guids); uid_table = malloc((sBlk.no_uids + sBlk.no_guids) * sizeof(unsigned int)); if(uid_table == NULL) { ERROR("read_uids_guids: failed to allocate uid/gid table\n"); return FALSE; } guid_table = uid_table + sBlk.no_uids; if(swap) { unsigned int suid_table[sBlk.no_uids + sBlk.no_guids]; res = read_fs_bytes(fd, sBlk.uid_start, (sBlk.no_uids + sBlk.no_guids) * sizeof(unsigned int), suid_table); if(res == FALSE) { ERROR("read_uids_guids: failed to read uid/gid table" "\n"); return FALSE; } SQUASHFS_SWAP_INTS_3(uid_table, suid_table, sBlk.no_uids + sBlk.no_guids); } else { res = read_fs_bytes(fd, sBlk.uid_start, (sBlk.no_uids + sBlk.no_guids) * sizeof(unsigned int), uid_table); if(res == FALSE) { ERROR("read_uids_guids: failed to read uid/gid table" "\n"); return FALSE; } } return TRUE; } squashfs4.3/squashfs-tools/unsquash-3.c0000644000175000017500000002622212306776317020212 0ustar phillipphillip/* * Unsquash a squashfs filesystem. This is a highly compressed read only * filesystem. * * Copyright (c) 2009, 2010, 2011, 2012, 2013 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * unsquash-3.c */ #include "unsquashfs.h" #include "squashfs_compat.h" static squashfs_fragment_entry_3 *fragment_table; int read_fragment_table_3(long long *directory_table_end) { int res, i; int bytes = SQUASHFS_FRAGMENT_BYTES_3(sBlk.s.fragments); int indexes = SQUASHFS_FRAGMENT_INDEXES_3(sBlk.s.fragments); long long fragment_table_index[indexes]; TRACE("read_fragment_table: %d fragments, reading %d fragment indexes " "from 0x%llx\n", sBlk.s.fragments, indexes, sBlk.s.fragment_table_start); if(sBlk.s.fragments == 0) { *directory_table_end = sBlk.s.fragment_table_start; return TRUE; } fragment_table = malloc(bytes); if(fragment_table == NULL) EXIT_UNSQUASH("read_fragment_table: failed to allocate " "fragment table\n"); if(swap) { long long sfragment_table_index[indexes]; res = read_fs_bytes(fd, sBlk.s.fragment_table_start, SQUASHFS_FRAGMENT_INDEX_BYTES_3(sBlk.s.fragments), sfragment_table_index); if(res == FALSE) { ERROR("read_fragment_table: failed to read fragment " "table index\n"); return FALSE; } SQUASHFS_SWAP_FRAGMENT_INDEXES_3(fragment_table_index, sfragment_table_index, indexes); } else { res = read_fs_bytes(fd, sBlk.s.fragment_table_start, SQUASHFS_FRAGMENT_INDEX_BYTES_3(sBlk.s.fragments), fragment_table_index); if(res == FALSE) { ERROR("read_fragment_table: failed to read fragment " "table index\n"); return FALSE; } } for(i = 0; i < indexes; i++) { int expected = (i + 1) != indexes ? SQUASHFS_METADATA_SIZE : bytes & (SQUASHFS_METADATA_SIZE - 1); int length = read_block(fd, fragment_table_index[i], NULL, expected, ((char *) fragment_table) + (i * SQUASHFS_METADATA_SIZE)); TRACE("Read fragment table block %d, from 0x%llx, length %d\n", i, fragment_table_index[i], length); if(length == FALSE) { ERROR("read_fragment_table: failed to read fragment " "table block\n"); return FALSE; } } if(swap) { squashfs_fragment_entry_3 sfragment; for(i = 0; i < sBlk.s.fragments; i++) { SQUASHFS_SWAP_FRAGMENT_ENTRY_3((&sfragment), (&fragment_table[i])); memcpy((char *) &fragment_table[i], (char *) &sfragment, sizeof(squashfs_fragment_entry_3)); } } *directory_table_end = fragment_table_index[0]; return TRUE; } void read_fragment_3(unsigned int fragment, long long *start_block, int *size) { TRACE("read_fragment: reading fragment %d\n", fragment); squashfs_fragment_entry_3 *fragment_entry = &fragment_table[fragment]; *start_block = fragment_entry->start_block; *size = fragment_entry->size; } struct inode *read_inode_3(unsigned int start_block, unsigned int offset) { static union squashfs_inode_header_3 header; long long start = sBlk.s.inode_table_start + start_block; int bytes = lookup_entry(inode_table_hash, start); char *block_ptr = inode_table + bytes + offset; static struct inode i; TRACE("read_inode: reading inode [%d:%d]\n", start_block, offset); if(bytes == -1) EXIT_UNSQUASH("read_inode: inode table block %lld not found\n", start); if(swap) { squashfs_base_inode_header_3 sinode; memcpy(&sinode, block_ptr, sizeof(header.base)); SQUASHFS_SWAP_BASE_INODE_HEADER_3(&header.base, &sinode, sizeof(squashfs_base_inode_header_3)); } else memcpy(&header.base, block_ptr, sizeof(header.base)); i.xattr = SQUASHFS_INVALID_XATTR; i.uid = (uid_t) uid_table[header.base.uid]; i.gid = header.base.guid == SQUASHFS_GUIDS ? i.uid : (uid_t) guid_table[header.base.guid]; i.mode = lookup_type[header.base.inode_type] | header.base.mode; i.type = header.base.inode_type; i.time = header.base.mtime; i.inode_number = header.base.inode_number; switch(header.base.inode_type) { case SQUASHFS_DIR_TYPE: { squashfs_dir_inode_header_3 *inode = &header.dir; if(swap) { squashfs_dir_inode_header_3 sinode; memcpy(&sinode, block_ptr, sizeof(header.dir)); SQUASHFS_SWAP_DIR_INODE_HEADER_3(&header.dir, &sinode); } else memcpy(&header.dir, block_ptr, sizeof(header.dir)); i.data = inode->file_size; i.offset = inode->offset; i.start = inode->start_block; break; } case SQUASHFS_LDIR_TYPE: { squashfs_ldir_inode_header_3 *inode = &header.ldir; if(swap) { squashfs_ldir_inode_header_3 sinode; memcpy(&sinode, block_ptr, sizeof(header.ldir)); SQUASHFS_SWAP_LDIR_INODE_HEADER_3(&header.ldir, &sinode); } else memcpy(&header.ldir, block_ptr, sizeof(header.ldir)); i.data = inode->file_size; i.offset = inode->offset; i.start = inode->start_block; break; } case SQUASHFS_FILE_TYPE: { squashfs_reg_inode_header_3 *inode = &header.reg; if(swap) { squashfs_reg_inode_header_3 sinode; memcpy(&sinode, block_ptr, sizeof(sinode)); SQUASHFS_SWAP_REG_INODE_HEADER_3(inode, &sinode); } else memcpy(inode, block_ptr, sizeof(*inode)); i.data = inode->file_size; i.frag_bytes = inode->fragment == SQUASHFS_INVALID_FRAG ? 0 : inode->file_size % sBlk.s.block_size; i.fragment = inode->fragment; i.offset = inode->offset; i.blocks = inode->fragment == SQUASHFS_INVALID_FRAG ? (i.data + sBlk.s.block_size - 1) >> sBlk.s.block_log : i.data >> sBlk.s.block_log; i.start = inode->start_block; i.sparse = 1; i.block_ptr = block_ptr + sizeof(*inode); break; } case SQUASHFS_LREG_TYPE: { squashfs_lreg_inode_header_3 *inode = &header.lreg; if(swap) { squashfs_lreg_inode_header_3 sinode; memcpy(&sinode, block_ptr, sizeof(sinode)); SQUASHFS_SWAP_LREG_INODE_HEADER_3(inode, &sinode); } else memcpy(inode, block_ptr, sizeof(*inode)); i.data = inode->file_size; i.frag_bytes = inode->fragment == SQUASHFS_INVALID_FRAG ? 0 : inode->file_size % sBlk.s.block_size; i.fragment = inode->fragment; i.offset = inode->offset; i.blocks = inode->fragment == SQUASHFS_INVALID_FRAG ? (inode->file_size + sBlk.s.block_size - 1) >> sBlk.s.block_log : inode->file_size >> sBlk.s.block_log; i.start = inode->start_block; i.sparse = 1; i.block_ptr = block_ptr + sizeof(*inode); break; } case SQUASHFS_SYMLINK_TYPE: { squashfs_symlink_inode_header_3 *inodep = &header.symlink; if(swap) { squashfs_symlink_inode_header_3 sinodep; memcpy(&sinodep, block_ptr, sizeof(sinodep)); SQUASHFS_SWAP_SYMLINK_INODE_HEADER_3(inodep, &sinodep); } else memcpy(inodep, block_ptr, sizeof(*inodep)); i.symlink = malloc(inodep->symlink_size + 1); if(i.symlink == NULL) EXIT_UNSQUASH("read_inode: failed to malloc " "symlink data\n"); strncpy(i.symlink, block_ptr + sizeof(squashfs_symlink_inode_header_3), inodep->symlink_size); i.symlink[inodep->symlink_size] = '\0'; i.data = inodep->symlink_size; break; } case SQUASHFS_BLKDEV_TYPE: case SQUASHFS_CHRDEV_TYPE: { squashfs_dev_inode_header_3 *inodep = &header.dev; if(swap) { squashfs_dev_inode_header_3 sinodep; memcpy(&sinodep, block_ptr, sizeof(sinodep)); SQUASHFS_SWAP_DEV_INODE_HEADER_3(inodep, &sinodep); } else memcpy(inodep, block_ptr, sizeof(*inodep)); i.data = inodep->rdev; break; } case SQUASHFS_FIFO_TYPE: case SQUASHFS_SOCKET_TYPE: i.data = 0; break; default: EXIT_UNSQUASH("Unknown inode type %d in read_inode!\n", header.base.inode_type); } return &i; } struct dir *squashfs_opendir_3(unsigned int block_start, unsigned int offset, struct inode **i) { squashfs_dir_header_3 dirh; char buffer[sizeof(squashfs_dir_entry_3) + SQUASHFS_NAME_LEN + 1] __attribute__((aligned)); squashfs_dir_entry_3 *dire = (squashfs_dir_entry_3 *) buffer; long long start; int bytes; int dir_count, size; struct dir_ent *new_dir; struct dir *dir; TRACE("squashfs_opendir: inode start block %d, offset %d\n", block_start, offset); *i = s_ops.read_inode(block_start, offset); dir = malloc(sizeof(struct dir)); if(dir == NULL) EXIT_UNSQUASH("squashfs_opendir: malloc failed!\n"); dir->dir_count = 0; dir->cur_entry = 0; dir->mode = (*i)->mode; dir->uid = (*i)->uid; dir->guid = (*i)->gid; dir->mtime = (*i)->time; dir->xattr = (*i)->xattr; dir->dirs = NULL; if ((*i)->data == 3) /* * if the directory is empty, skip the unnecessary * lookup_entry, this fixes the corner case with * completely empty filesystems where lookup_entry correctly * returning -1 is incorrectly treated as an error */ return dir; start = sBlk.s.directory_table_start + (*i)->start; bytes = lookup_entry(directory_table_hash, start); if(bytes == -1) EXIT_UNSQUASH("squashfs_opendir: directory block %d not " "found!\n", block_start); bytes += (*i)->offset; size = (*i)->data + bytes - 3; while(bytes < size) { if(swap) { squashfs_dir_header_3 sdirh; memcpy(&sdirh, directory_table + bytes, sizeof(sdirh)); SQUASHFS_SWAP_DIR_HEADER_3(&dirh, &sdirh); } else memcpy(&dirh, directory_table + bytes, sizeof(dirh)); dir_count = dirh.count + 1; TRACE("squashfs_opendir: Read directory header @ byte position " "%d, %d directory entries\n", bytes, dir_count); bytes += sizeof(dirh); /* dir_count should never be larger than 256 */ if(dir_count > 256) goto corrupted; while(dir_count--) { if(swap) { squashfs_dir_entry_3 sdire; memcpy(&sdire, directory_table + bytes, sizeof(sdire)); SQUASHFS_SWAP_DIR_ENTRY_3(dire, &sdire); } else memcpy(dire, directory_table + bytes, sizeof(*dire)); bytes += sizeof(*dire); /* size should never be larger than SQUASHFS_NAME_LEN */ if(dire->size > SQUASHFS_NAME_LEN) goto corrupted; memcpy(dire->name, directory_table + bytes, dire->size + 1); dire->name[dire->size + 1] = '\0'; TRACE("squashfs_opendir: directory entry %s, inode " "%d:%d, type %d\n", dire->name, dirh.start_block, dire->offset, dire->type); if((dir->dir_count % DIR_ENT_SIZE) == 0) { new_dir = realloc(dir->dirs, (dir->dir_count + DIR_ENT_SIZE) * sizeof(struct dir_ent)); if(new_dir == NULL) EXIT_UNSQUASH("squashfs_opendir: " "realloc failed!\n"); dir->dirs = new_dir; } strcpy(dir->dirs[dir->dir_count].name, dire->name); dir->dirs[dir->dir_count].start_block = dirh.start_block; dir->dirs[dir->dir_count].offset = dire->offset; dir->dirs[dir->dir_count].type = dire->type; dir->dir_count ++; bytes += dire->size + 1; } } return dir; corrupted: free(dir->dirs); free(dir); return NULL; } squashfs4.3/squashfs-tools/read_fs.h0000644000175000017500000000260612306776317017613 0ustar phillipphillip#ifndef READ_FS_H #define READ_FS_H /* * Squashfs * * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2013 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * read_fs.h * */ extern struct compressor *read_super(int, struct squashfs_super_block *, char *); extern long long read_filesystem(char *, int, struct squashfs_super_block *, char **, char **, char **, char **, unsigned int *, unsigned int *, unsigned int *, unsigned int *, unsigned int *, int *, int *, int *, int *, int *, int *, long long *, unsigned int *, unsigned int *, unsigned int *, unsigned int *, void (push_directory_entry)(char *, squashfs_inode, int, int), struct squashfs_fragment_entry **, squashfs_inode **); #endif squashfs4.3/squashfs-tools/process_fragments.h0000644000175000017500000000177012333330365021722 0ustar phillipphillip#ifndef PROCESS_FRAGMENTS_H #define PROCESS_FRAGMENTS_H /* * Create a squashfs filesystem. This is a highly compressed read only * filesystem. * * Copyright (c) 2014 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * process_fragments.h */ #define DUP_HASH(a) (a & 0xffff) extern void *frag_thrd(void *); #endif squashfs4.3/squashfs-tools/read_xattrs.c0000644000175000017500000002410012306776317020514 0ustar phillipphillip/* * Read a squashfs filesystem. This is a highly compressed read only * filesystem. * * Copyright (c) 2010, 2012, 2013 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * read_xattrs.c */ /* * Common xattr read code shared between mksquashfs and unsquashfs */ #define TRUE 1 #define FALSE 0 #include #include #ifndef linux #define __BYTE_ORDER BYTE_ORDER #define __BIG_ENDIAN BIG_ENDIAN #define __LITTLE_ENDIAN LITTLE_ENDIAN #else #include #endif #include "squashfs_fs.h" #include "squashfs_swap.h" #include "xattr.h" #include "error.h" #include extern int read_fs_bytes(int, long long, int, void *); extern int read_block(int, long long, long long *, int, void *); static struct hash_entry { long long start; unsigned int offset; struct hash_entry *next; } *hash_table[65536]; static struct squashfs_xattr_id *xattr_ids; static void *xattrs = NULL; static long long xattr_table_start; /* * Prefix lookup table, storing mapping to/from prefix string and prefix id */ struct prefix prefix_table[] = { { "user.", SQUASHFS_XATTR_USER }, { "trusted.", SQUASHFS_XATTR_TRUSTED }, { "security.", SQUASHFS_XATTR_SECURITY }, { "", -1 } }; /* * store mapping from location of compressed block in fs -> * location of uncompressed block in memory */ static void save_xattr_block(long long start, int offset) { struct hash_entry *hash_entry = malloc(sizeof(*hash_entry)); int hash = start & 0xffff; TRACE("save_xattr_block: start %lld, offset %d\n", start, offset); if(hash_entry == NULL) MEM_ERROR(); hash_entry->start = start; hash_entry->offset = offset; hash_entry->next = hash_table[hash]; hash_table[hash] = hash_entry; } /* * map from location of compressed block in fs -> * location of uncompressed block in memory */ static int get_xattr_block(long long start) { int hash = start & 0xffff; struct hash_entry *hash_entry = hash_table[hash]; for(; hash_entry; hash_entry = hash_entry->next) if(hash_entry->start == start) break; TRACE("get_xattr_block: start %lld, offset %d\n", start, hash_entry ? hash_entry->offset : -1); return hash_entry ? hash_entry->offset : -1; } /* * construct the xattr_list entry from the fs xattr, including * mapping name and prefix into a full name */ static int read_xattr_entry(struct xattr_list *xattr, struct squashfs_xattr_entry *entry, void *name) { int i, len, type = entry->type & XATTR_PREFIX_MASK; for(i = 0; prefix_table[i].type != -1; i++) if(prefix_table[i].type == type) break; if(prefix_table[i].type == -1) { ERROR("Unrecognised type in read_xattr_entry\n"); return 0; } len = strlen(prefix_table[i].prefix); xattr->full_name = malloc(len + entry->size + 1); if(xattr->full_name == NULL) MEM_ERROR(); memcpy(xattr->full_name, prefix_table[i].prefix, len); memcpy(xattr->full_name + len, name, entry->size); xattr->full_name[len + entry->size] = '\0'; xattr->name = xattr->full_name + len; xattr->size = entry->size; xattr->type = type; return 1; } /* * Read and decompress the xattr id table and the xattr metadata. * This is cached in memory for later use by get_xattr() */ int read_xattrs_from_disk(int fd, struct squashfs_super_block *sBlk) { int res, bytes, i, indexes, index_bytes, ids; long long *index, start, end; struct squashfs_xattr_table id_table; TRACE("read_xattrs_from_disk\n"); if(sBlk->xattr_id_table_start == SQUASHFS_INVALID_BLK) return SQUASHFS_INVALID_BLK; /* * Read xattr id table, containing start of xattr metadata and the * number of xattrs in the file system */ res = read_fs_bytes(fd, sBlk->xattr_id_table_start, sizeof(id_table), &id_table); if(res == 0) return 0; SQUASHFS_INSWAP_XATTR_TABLE(&id_table); /* * Allocate and read the index to the xattr id table metadata * blocks */ ids = id_table.xattr_ids; xattr_table_start = id_table.xattr_table_start; index_bytes = SQUASHFS_XATTR_BLOCK_BYTES(ids); indexes = SQUASHFS_XATTR_BLOCKS(ids); index = malloc(index_bytes); if(index == NULL) MEM_ERROR(); res = read_fs_bytes(fd, sBlk->xattr_id_table_start + sizeof(id_table), index_bytes, index); if(res ==0) goto failed1; SQUASHFS_INSWAP_LONG_LONGS(index, indexes); /* * Allocate enough space for the uncompressed xattr id table, and * read and decompress it */ bytes = SQUASHFS_XATTR_BYTES(ids); xattr_ids = malloc(bytes); if(xattr_ids == NULL) MEM_ERROR(); for(i = 0; i < indexes; i++) { int expected = (i + 1) != indexes ? SQUASHFS_METADATA_SIZE : bytes & (SQUASHFS_METADATA_SIZE - 1); int length = read_block(fd, index[i], NULL, expected, ((unsigned char *) xattr_ids) + (i * SQUASHFS_METADATA_SIZE)); TRACE("Read xattr id table block %d, from 0x%llx, length " "%d\n", i, index[i], length); if(length == 0) { ERROR("Failed to read xattr id table block %d, " "from 0x%llx, length %d\n", i, index[i], length); goto failed2; } } /* * Read and decompress the xattr metadata * * Note the first xattr id table metadata block is immediately after * the last xattr metadata block, so we can use index[0] to work out * the end of the xattr metadata */ start = xattr_table_start; end = index[0]; for(i = 0; start < end; i++) { int length; xattrs = realloc(xattrs, (i + 1) * SQUASHFS_METADATA_SIZE); if(xattrs == NULL) MEM_ERROR(); /* store mapping from location of compressed block in fs -> * location of uncompressed block in memory */ save_xattr_block(start, i * SQUASHFS_METADATA_SIZE); length = read_block(fd, start, &start, 0, ((unsigned char *) xattrs) + (i * SQUASHFS_METADATA_SIZE)); TRACE("Read xattr block %d, length %d\n", i, length); if(length == 0) { ERROR("Failed to read xattr block %d\n", i); goto failed3; } /* * If this is not the last metadata block in the xattr metadata * then it should be SQUASHFS_METADATA_SIZE in size. * Note, we can't use expected in read_block() above for this * because we don't know if this is the last block until * after reading. */ if(start != end && length != SQUASHFS_METADATA_SIZE) { ERROR("Xattr block %d should be %d bytes in length, " "it is %d bytes\n", i, SQUASHFS_METADATA_SIZE, length); goto failed3; } } /* swap if necessary the xattr id entries */ for(i = 0; i < ids; i++) SQUASHFS_INSWAP_XATTR_ID(&xattr_ids[i]); free(index); return ids; failed3: free(xattrs); failed2: free(xattr_ids); failed1: free(index); return 0; } void free_xattr(struct xattr_list *xattr_list, int count) { int i; for(i = 0; i < count; i++) free(xattr_list[i].full_name); free(xattr_list); } /* * Construct and return the list of xattr name:value pairs for the passed xattr * id * * There are two users for get_xattr(), Mksquashfs uses it to read the * xattrs from the filesystem on appending, and Unsquashfs uses it * to retrieve the xattrs for writing to disk. * * Unfortunately, the two users disagree on what to do with unknown * xattr prefixes, Mksquashfs wants to treat this as fatal otherwise * this will cause xattrs to be be lost on appending. Unsquashfs * on the otherhand wants to retrieve the xattrs which are known and * to ignore the rest, this allows Unsquashfs to cope more gracefully * with future versions which may have unknown xattrs, as long as the * general xattr structure is adhered to, Unsquashfs should be able * to safely ignore unknown xattrs, and to write the ones it knows about, * this is better than completely refusing to retrieve all the xattrs. * * If ignore is TRUE then don't treat unknown xattr prefixes as * a failure to read the xattr. */ struct xattr_list *get_xattr(int i, unsigned int *count, int ignore) { long long start; struct xattr_list *xattr_list = NULL; unsigned int offset; void *xptr; int j = 0, res = 1; TRACE("get_xattr\n"); *count = xattr_ids[i].count; start = SQUASHFS_XATTR_BLK(xattr_ids[i].xattr) + xattr_table_start; offset = SQUASHFS_XATTR_OFFSET(xattr_ids[i].xattr); xptr = xattrs + get_xattr_block(start) + offset; TRACE("get_xattr: xattr_id %d, count %d, start %lld, offset %d\n", i, *count, start, offset); while(j < *count) { struct squashfs_xattr_entry entry; struct squashfs_xattr_val val; if(res != 0) { xattr_list = realloc(xattr_list, (j + 1) * sizeof(struct xattr_list)); if(xattr_list == NULL) MEM_ERROR(); } SQUASHFS_SWAP_XATTR_ENTRY(xptr, &entry); xptr += sizeof(entry); res = read_xattr_entry(&xattr_list[j], &entry, xptr); if(ignore && res == 0) { /* unknown prefix, but ignore flag is set */ (*count) --; continue; } if(res != 1) goto failed; xptr += entry.size; TRACE("get_xattr: xattr %d, type %d, size %d, name %s\n", j, entry.type, entry.size, xattr_list[j].full_name); if(entry.type & SQUASHFS_XATTR_VALUE_OOL) { long long xattr; void *ool_xptr; xptr += sizeof(val); SQUASHFS_SWAP_LONG_LONGS(xptr, &xattr, 1); xptr += sizeof(xattr); start = SQUASHFS_XATTR_BLK(xattr) + xattr_table_start; offset = SQUASHFS_XATTR_OFFSET(xattr); ool_xptr = xattrs + get_xattr_block(start) + offset; SQUASHFS_SWAP_XATTR_VAL(ool_xptr, &val); xattr_list[j].value = ool_xptr + sizeof(val); } else { SQUASHFS_SWAP_XATTR_VAL(xptr, &val); xattr_list[j].value = xptr + sizeof(val); xptr += sizeof(val) + val.vsize; } TRACE("get_xattr: xattr %d, vsize %d\n", j, val.vsize); xattr_list[j ++].vsize = val.vsize; } if(*count == 0) goto failed; return xattr_list; failed: free_xattr(xattr_list, j); return NULL; } squashfs4.3/squashfs-tools/read_file.c0000644000175000017500000000660312306776317020116 0ustar phillipphillip/* * Create a squashfs filesystem. This is a highly compressed read only * filesystem. * * Copyright (c) 2012 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * read_file.c */ #include #include #include #include #include #include "error.h" #define TRUE 1 #define FALSE 0 #define MAX_LINE 16384 /* * Read file, passing each line to parse_line() for * parsing. * * Lines can be split across multiple lines using "\". * * Blank lines and comment lines indicated by # are supported. */ int read_file(char *filename, char *type, int (parse_line)(char *)) { FILE *fd; char *def, *err, *line = NULL; int res, size = 0; fd = fopen(filename, "r"); if(fd == NULL) { ERROR("Could not open %s device file \"%s\" because %s\n", type, filename, strerror(errno)); return FALSE; } while(1) { int total = 0; while(1) { int len; if(total + (MAX_LINE + 1) > size) { line = realloc(line, size += (MAX_LINE + 1)); if(line == NULL) MEM_ERROR(); } err = fgets(line + total, MAX_LINE + 1, fd); if(err == NULL) break; len = strlen(line + total); total += len; if(len == MAX_LINE && line[total - 1] != '\n') { /* line too large */ ERROR("Line too long when reading " "%s file \"%s\", larger than " "%d bytes\n", type, filename, MAX_LINE); goto failed; } /* * Remove '\n' terminator if it exists (the last line * in the file may not be '\n' terminated) */ if(len && line[total - 1] == '\n') { line[-- total] = '\0'; len --; } /* * If no line continuation then jump out to * process line. Note, we have to be careful to * check for "\\" (backslashed backslash) and to * ensure we don't look at the previous line */ if(len == 0 || line[total - 1] != '\\' || (len >= 2 && strcmp(line + total - 2, "\\\\") == 0)) break; else total --; } if(err == NULL) { if(ferror(fd)) { ERROR("Reading %s file \"%s\" failed " "because %s\n", type, filename, strerror(errno)); goto failed; } /* * At EOF, normally we'll be finished, but, have to * check for special case where we had "\" line * continuation and then hit EOF immediately afterwards */ if(total == 0) break; else line[total] = '\0'; } /* Skip any leading whitespace */ for(def = line; isspace(*def); def ++); /* if line is now empty after skipping characters, skip it */ if(*def == '\0') continue; /* if comment line, skip */ if(*def == '#') continue; res = parse_line(def); if(res == FALSE) goto failed; } fclose(fd); free(line); return TRUE; failed: fclose(fd); free(line); return FALSE; } squashfs4.3/squashfs-tools/lz4_wrapper.h0000644000175000017500000000303112306776316020451 0ustar phillipphillip#ifndef LZ4_WRAPPER_H #define LZ4_WRAPPER_H /* * Squashfs * * Copyright (c) 2013 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * lz4_wrapper.h * */ #ifndef linux #define __BYTE_ORDER BYTE_ORDER #define __BIG_ENDIAN BIG_ENDIAN #define __LITTLE_ENDIAN LITTLE_ENDIAN #else #include #endif #if __BYTE_ORDER == __BIG_ENDIAN extern unsigned int inswap_le32(unsigned int); #define SQUASHFS_INSWAP_COMP_OPTS(s) { \ (s)->version = inswap_le32((s)->version); \ (s)->flags = inswap_le32((s)->flags); \ } #else #define SQUASHFS_INSWAP_COMP_OPTS(s) #endif /* * Define the various stream formats recognised. * Currently omly legacy stream format is supported by the * kernel */ #define LZ4_LEGACY 1 #define LZ4_FLAGS_MASK 1 /* Define the compression flags recognised. */ #define LZ4_HC 1 struct lz4_comp_opts { int version; int flags; }; #endif squashfs4.3/squashfs-tools/pseudo.c0000644000175000017500000002627412333330365017476 0ustar phillipphillip/* * Create a squashfs filesystem. This is a highly compressed read only * filesystem. * * Copyright (c) 2009, 2010, 2012, 2014 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * pseudo.c */ #include #include #include #include #include #include #include #include #include #include #include #include "pseudo.h" #include "error.h" #include "progressbar.h" #define TRUE 1 #define FALSE 0 extern int read_file(char *filename, char *type, int (parse_line)(char *)); struct pseudo_dev **pseudo_file = NULL; struct pseudo *pseudo = NULL; int pseudo_count = 0; static char *get_component(char *target, char **targname) { char *start; while(*target == '/') target ++; start = target; while(*target != '/' && *target != '\0') target ++; *targname = strndup(start, target - start); while(*target == '/') target ++; return target; } /* * Add pseudo device target to the set of pseudo devices. Pseudo_dev * describes the pseudo device attributes. */ struct pseudo *add_pseudo(struct pseudo *pseudo, struct pseudo_dev *pseudo_dev, char *target, char *alltarget) { char *targname; int i; target = get_component(target, &targname); if(pseudo == NULL) { pseudo = malloc(sizeof(struct pseudo)); if(pseudo == NULL) MEM_ERROR(); pseudo->names = 0; pseudo->count = 0; pseudo->name = NULL; } for(i = 0; i < pseudo->names; i++) if(strcmp(pseudo->name[i].name, targname) == 0) break; if(i == pseudo->names) { /* allocate new name entry */ pseudo->names ++; pseudo->name = realloc(pseudo->name, (i + 1) * sizeof(struct pseudo_entry)); if(pseudo->name == NULL) MEM_ERROR(); pseudo->name[i].name = targname; if(target[0] == '\0') { /* at leaf pathname component */ pseudo->name[i].pseudo = NULL; pseudo->name[i].pathname = strdup(alltarget); pseudo->name[i].dev = pseudo_dev; } else { /* recurse adding child components */ pseudo->name[i].dev = NULL; pseudo->name[i].pseudo = add_pseudo(NULL, pseudo_dev, target, alltarget); } } else { /* existing matching entry */ free(targname); if(pseudo->name[i].pseudo == NULL) { /* No sub-directory which means this is the leaf * component of a pre-existing pseudo file. */ if(target[0] != '\0') { /* * entry must exist as either a 'd' type or * 'm' type pseudo file */ if(pseudo->name[i].dev->type == 'd' || pseudo->name[i].dev->type == 'm') /* recurse adding child components */ pseudo->name[i].pseudo = add_pseudo(NULL, pseudo_dev, target, alltarget); else { ERROR_START("%s already exists as a " "non directory.", pseudo->name[i].name); ERROR_EXIT(". Ignoring %s!\n", alltarget); } } else if(memcmp(pseudo_dev, pseudo->name[i].dev, sizeof(struct pseudo_dev)) != 0) { ERROR_START("%s already exists as a different " "pseudo definition.", alltarget); ERROR_EXIT(" Ignoring!\n"); } else { ERROR_START("%s already exists as an identical " "pseudo definition!", alltarget); ERROR_EXIT(" Ignoring!\n"); } } else { if(target[0] == '\0') { /* * sub-directory exists, which means we can only * add a pseudo file of type 'd' or type 'm' */ if(pseudo->name[i].dev == NULL && (pseudo_dev->type == 'd' || pseudo_dev->type == 'm')) { pseudo->name[i].pathname = strdup(alltarget); pseudo->name[i].dev = pseudo_dev; } else { ERROR_START("%s already exists as a " "different pseudo definition.", pseudo->name[i].name); ERROR_EXIT(" Ignoring %s!\n", alltarget); } } else /* recurse adding child components */ add_pseudo(pseudo->name[i].pseudo, pseudo_dev, target, alltarget); } } return pseudo; } /* * Find subdirectory in pseudo directory referenced by pseudo, matching * filename. If filename doesn't exist or if filename is a leaf file * return NULL */ struct pseudo *pseudo_subdir(char *filename, struct pseudo *pseudo) { int i; if(pseudo == NULL) return NULL; for(i = 0; i < pseudo->names; i++) if(strcmp(filename, pseudo->name[i].name) == 0) return pseudo->name[i].pseudo; return NULL; } struct pseudo_entry *pseudo_readdir(struct pseudo *pseudo) { if(pseudo == NULL) return NULL; while(pseudo->count < pseudo->names) { if(pseudo->name[pseudo->count].dev != NULL) return &pseudo->name[pseudo->count++]; else pseudo->count++; } return NULL; } int pseudo_exec_file(struct pseudo_dev *dev, int *child) { int res, pipefd[2]; res = pipe(pipefd); if(res == -1) { ERROR("Executing dynamic pseudo file, pipe failed\n"); return 0; } *child = fork(); if(*child == -1) { ERROR("Executing dynamic pseudo file, fork failed\n"); goto failed; } if(*child == 0) { close(pipefd[0]); close(STDOUT_FILENO); res = dup(pipefd[1]); if(res == -1) exit(EXIT_FAILURE); execl("/bin/sh", "sh", "-c", dev->command, (char *) NULL); exit(EXIT_FAILURE); } close(pipefd[1]); return pipefd[0]; failed: close(pipefd[0]); close(pipefd[1]); return 0; } void add_pseudo_file(struct pseudo_dev *dev) { pseudo_file = realloc(pseudo_file, (pseudo_count + 1) * sizeof(struct pseudo_dev *)); if(pseudo_file == NULL) MEM_ERROR(); dev->pseudo_id = pseudo_count; pseudo_file[pseudo_count ++] = dev; } struct pseudo_dev *get_pseudo_file(int pseudo_id) { return pseudo_file[pseudo_id]; } int read_pseudo_def(char *def) { int n, bytes; unsigned int major = 0, minor = 0, mode; char type, *ptr; char suid[100], sgid[100]; /* overflow safe */ char *filename, *name; char *orig_def = def; long long uid, gid; struct pseudo_dev *dev; /* * Scan for filename, don't use sscanf() and "%s" because * that can't handle filenames with spaces */ filename = malloc(strlen(def) + 1); if(filename == NULL) MEM_ERROR(); for(name = filename; !isspace(*def) && *def != '\0';) { if(*def == '\\') { def ++; if (*def == '\0') break; } *name ++ = *def ++; } *name = '\0'; if(*filename == '\0') { ERROR("Not enough or invalid arguments in pseudo file " "definition \"%s\"\n", orig_def); goto error; } n = sscanf(def, " %c %o %99s %99s %n", &type, &mode, suid, sgid, &bytes); def += bytes; if(n < 4) { ERROR("Not enough or invalid arguments in pseudo file " "definition \"%s\"\n", orig_def); switch(n) { case -1: /* FALLTHROUGH */ case 0: ERROR("Read filename, but failed to read or match " "type\n"); break; case 1: ERROR("Read filename and type, but failed to read or " "match octal mode\n"); break; case 2: ERROR("Read filename, type and mode, but failed to " "read or match uid\n"); break; default: ERROR("Read filename, type, mode and uid, but failed " "to read or match gid\n"); break; } goto error; } switch(type) { case 'b': /* FALLTHROUGH */ case 'c': n = sscanf(def, "%u %u %n", &major, &minor, &bytes); def += bytes; if(n < 2) { ERROR("Not enough or invalid arguments in %s device " "pseudo file definition \"%s\"\n", type == 'b' ? "block" : "character", orig_def); if(n < 1) ERROR("Read filename, type, mode, uid and gid, " "but failed to read or match major\n"); else ERROR("Read filename, type, mode, uid, gid " "and major, but failed to read or " "match minor\n"); goto error; } if(major > 0xfff) { ERROR("Major %d out of range\n", major); goto error; } if(minor > 0xfffff) { ERROR("Minor %d out of range\n", minor); goto error; } /* FALLTHROUGH */ case 'd': /* FALLTHROUGH */ case 'm': /* * Check for trailing junk after expected arguments */ if(def[0] != '\0') { ERROR("Unexpected tailing characters in pseudo file " "definition \"%s\"\n", orig_def); goto error; } break; case 'f': if(def[0] == '\0') { ERROR("Not enough arguments in dynamic file pseudo " "definition \"%s\"\n", orig_def); ERROR("Expected command, which can be an executable " "or a piece of shell script\n"); goto error; } break; default: ERROR("Unsupported type %c\n", type); goto error; } if(mode > 07777) { ERROR("Mode %o out of range\n", mode); goto error; } uid = strtoll(suid, &ptr, 10); if(*ptr == '\0') { if(uid < 0 || uid > ((1LL << 32) - 1)) { ERROR("Uid %s out of range\n", suid); goto error; } } else { struct passwd *pwuid = getpwnam(suid); if(pwuid) uid = pwuid->pw_uid; else { ERROR("Uid %s invalid uid or unknown user\n", suid); goto error; } } gid = strtoll(sgid, &ptr, 10); if(*ptr == '\0') { if(gid < 0 || gid > ((1LL << 32) - 1)) { ERROR("Gid %s out of range\n", sgid); goto error; } } else { struct group *grgid = getgrnam(sgid); if(grgid) gid = grgid->gr_gid; else { ERROR("Gid %s invalid uid or unknown user\n", sgid); goto error; } } switch(type) { case 'b': mode |= S_IFBLK; break; case 'c': mode |= S_IFCHR; break; case 'd': mode |= S_IFDIR; break; case 'f': mode |= S_IFREG; break; } dev = malloc(sizeof(struct pseudo_dev)); if(dev == NULL) MEM_ERROR(); dev->type = type; dev->mode = mode; dev->uid = uid; dev->gid = gid; dev->major = major; dev->minor = minor; if(type == 'f') { dev->command = strdup(def); add_pseudo_file(dev); } pseudo = add_pseudo(pseudo, dev, filename, filename); free(filename); return TRUE; error: ERROR("Pseudo definitions should be of format\n"); ERROR("\tfilename d mode uid gid\n"); ERROR("\tfilename m mode uid gid\n"); ERROR("\tfilename b mode uid gid major minor\n"); ERROR("\tfilename c mode uid gid major minor\n"); ERROR("\tfilename f mode uid command\n"); free(filename); return FALSE; } int read_pseudo_file(char *filename) { return read_file(filename, "pseudo", read_pseudo_def); } struct pseudo *get_pseudo() { return pseudo; } #ifdef SQUASHFS_TRACE static void dump_pseudo(struct pseudo *pseudo, char *string) { int i, res; char *path; for(i = 0; i < pseudo->names; i++) { struct pseudo_entry *entry = &pseudo->name[i]; if(string) { res = asprintf(&path, "%s/%s", string, entry->name); if(res == -1) BAD_ERROR("asprintf failed in dump_pseudo\n"); } else path = entry->name; if(entry->dev) ERROR("%s %c 0%o %d %d %d %d\n", path, entry->dev->type, entry->dev->mode & ~S_IFMT, entry->dev->uid, entry->dev->gid, entry->dev->major, entry->dev->minor); if(entry->pseudo) dump_pseudo(entry->pseudo, path); if(string) free(path); } } void dump_pseudos() { dump_pseudo(pseudo, NULL); } #else void dump_pseudos() { } #endif squashfs4.3/squashfs-tools/lzma_wrapper.c0000644000175000017500000000604412306776316020705 0ustar phillipphillip/* * Copyright (c) 2009, 2010, 2013 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * lzma_wrapper.c * * Support for LZMA1 compression using LZMA SDK (4.65 used in * development, other versions may work) http://www.7-zip.org/sdk.html */ #include #include "squashfs_fs.h" #include "compressor.h" #define LZMA_HEADER_SIZE (LZMA_PROPS_SIZE + 8) static int lzma_compress(void *strm, void *dest, void *src, int size, int block_size, int *error) { unsigned char *d = dest; size_t props_size = LZMA_PROPS_SIZE, outlen = block_size - LZMA_HEADER_SIZE; int res; res = LzmaCompress(dest + LZMA_HEADER_SIZE, &outlen, src, size, dest, &props_size, 5, block_size, 3, 0, 2, 32, 1); if(res == SZ_ERROR_OUTPUT_EOF) { /* * Output buffer overflow. Return out of buffer space error */ return 0; } if(res != SZ_OK) { /* * All other errors return failure, with the compressor * specific error code in *error */ *error = res; return -1; } /* * Fill in the 8 byte little endian uncompressed size field in the * LZMA header. 8 bytes is excessively large for squashfs but * this is the standard LZMA header and which is expected by the kernel * code */ d[LZMA_PROPS_SIZE] = size & 255; d[LZMA_PROPS_SIZE + 1] = (size >> 8) & 255; d[LZMA_PROPS_SIZE + 2] = (size >> 16) & 255; d[LZMA_PROPS_SIZE + 3] = (size >> 24) & 255; d[LZMA_PROPS_SIZE + 4] = 0; d[LZMA_PROPS_SIZE + 5] = 0; d[LZMA_PROPS_SIZE + 6] = 0; d[LZMA_PROPS_SIZE + 7] = 0; /* * Success, return the compressed size. Outlen returned by the LZMA * compressor does not include the LZMA header space */ return outlen + LZMA_HEADER_SIZE; } static int lzma_uncompress(void *dest, void *src, int size, int outsize, int *error) { unsigned char *s = src; size_t outlen, inlen = size - LZMA_HEADER_SIZE; int res; outlen = s[LZMA_PROPS_SIZE] | (s[LZMA_PROPS_SIZE + 1] << 8) | (s[LZMA_PROPS_SIZE + 2] << 16) | (s[LZMA_PROPS_SIZE + 3] << 24); if(outlen > outsize) { *error = 0; return -1; } res = LzmaUncompress(dest, &outlen, src + LZMA_HEADER_SIZE, &inlen, src, LZMA_PROPS_SIZE); if(res == SZ_OK) return outlen; else { *error = res; return -1; } } struct compressor lzma_comp_ops = { .init = NULL, .compress = lzma_compress, .uncompress = lzma_uncompress, .options = NULL, .usage = NULL, .id = LZMA_COMPRESSION, .name = "lzma", .supported = 1 }; squashfs4.3/squashfs-tools/sort.h0000644000175000017500000000227712306776317017203 0ustar phillipphillip#ifndef SORT_H #define SORT_H /* * Squashfs * * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2013 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * sort.h */ struct priority_entry { struct dir_ent *dir; struct priority_entry *next; }; extern int read_sort_file(char *, int, char *[]); extern void sort_files_and_write(struct dir_info *); extern void generate_file_priorities(struct dir_info *, int priority, struct stat *); extern struct priority_entry *priority_list[65536]; #endif squashfs4.3/squashfs-tools/gzip_wrapper.h0000644000175000017500000000344012334003731020676 0ustar phillipphillip#ifndef GZIP_WRAPPER_H #define GZIP_WRAPPER_H /* * Squashfs * * Copyright (c) 2014 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * gzip_wrapper.h * */ #ifndef linux #define __BYTE_ORDER BYTE_ORDER #define __BIG_ENDIAN BIG_ENDIAN #define __LITTLE_ENDIAN LITTLE_ENDIAN #else #include #endif #if __BYTE_ORDER == __BIG_ENDIAN extern unsigned int inswap_le16(unsigned short); extern unsigned int inswap_le32(unsigned int); #define SQUASHFS_INSWAP_COMP_OPTS(s) { \ (s)->compression_level = inswap_le32((s)->compression_level); \ (s)->window_size = inswap_le16((s)->window_size); \ (s)->strategy = inswap_le16((s)->strategy); \ } #else #define SQUASHFS_INSWAP_COMP_OPTS(s) #endif /* Default compression */ #define GZIP_DEFAULT_COMPRESSION_LEVEL 9 #define GZIP_DEFAULT_WINDOW_SIZE 15 struct gzip_comp_opts { int compression_level; short window_size; short strategy; }; struct strategy { char *name; int strategy; int selected; }; struct gzip_strategy { int strategy; int length; void *buffer; }; struct gzip_stream { z_stream stream; int strategies; struct gzip_strategy strategy[0]; }; #endif squashfs4.3/squashfs-tools/process_fragments.c0000644000175000017500000002366212333330365021721 0ustar phillipphillip/* * Create a squashfs filesystem. This is a highly compressed read only * filesystem. * * Copyright (c) 2014 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * process_fragments.c */ #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include #include "caches-queues-lists.h" #include "squashfs_fs.h" #include "mksquashfs.h" #include "error.h" #include "progressbar.h" #include "info.h" #include "compressor.h" #include "process_fragments.h" #define FALSE 0 #define TRUE 1 extern struct queue *to_process_frag; extern struct seq_queue *to_main; extern int sparse_files; /* * Compute 16 bit BSD checksum over the data, and check for sparseness */ static int checksum_sparse(struct file_buffer *file_buffer) { unsigned char *b = (unsigned char *) file_buffer->data; unsigned short chksum = 0; int bytes = file_buffer->size, sparse = TRUE, value; while(bytes --) { chksum = (chksum & 1) ? (chksum >> 1) | 0x8000 : chksum >> 1; value = *b++; if(value) { sparse = FALSE; chksum += value; } } file_buffer->checksum = chksum; return sparse; } static int read_filesystem(int fd, long long byte, int bytes, void *buff) { off_t off = byte; TRACE("read_filesystem: reading from position 0x%llx, bytes %d\n", byte, bytes); if(lseek(fd, off, SEEK_SET) == -1) { ERROR("read_filesystem: Lseek on destination failed because %s, " "offset=0x%llx\n", strerror(errno), off); return 0; } else if(read_bytes(fd, buff, bytes) < bytes) { ERROR("Read on destination failed\n"); return 0; } return 1; } static struct file_buffer *get_fragment(struct fragment *fragment, char *data_buffer, int fd) { struct squashfs_fragment_entry *disk_fragment; struct file_buffer *buffer, *compressed_buffer; long long start_block; int res, size, index = fragment->index; char locked; /* * Lookup fragment block in cache. * If the fragment block doesn't exist, then get the compressed version * from the writer cache or off disk, and decompress it. * * This routine has two things which complicate the code: * * 1. Multiple threads can simultaneously lookup/create the * same buffer. This means a buffer needs to be "locked" * when it is being filled in, to prevent other threads from * using it when it is not ready. This is because we now do * fragment duplicate checking in parallel. * 2. We have two caches which need to be checked for the * presence of fragment blocks: the normal fragment cache * and a "reserve" cache. The reserve cache is used to * prevent an unnecessary pipeline stall when the fragment cache * is full of fragments waiting to be compressed. */ pthread_cleanup_push((void *) pthread_mutex_unlock, &dup_mutex); pthread_mutex_lock(&dup_mutex); again: buffer = cache_lookup_nowait(fragment_buffer, index, &locked); if(buffer) { pthread_mutex_unlock(&dup_mutex); if(locked) /* got a buffer being filled in. Wait for it */ cache_wait_unlock(buffer); goto finished; } /* not in fragment cache, is it in the reserve cache? */ buffer = cache_lookup_nowait(reserve_cache, index, &locked); if(buffer) { pthread_mutex_unlock(&dup_mutex); if(locked) /* got a buffer being filled in. Wait for it */ cache_wait_unlock(buffer); goto finished; } /* in neither cache, try to get it from the fragment cache */ buffer = cache_get_nowait(fragment_buffer, index); if(!buffer) { /* * no room, get it from the reserve cache, this is * dimensioned so it will always have space (no more than * processors + 1 can have an outstanding reserve buffer) */ buffer = cache_get_nowait(reserve_cache, index); if(!buffer) { /* failsafe */ ERROR("no space in reserve cache\n"); goto again; } } pthread_mutex_unlock(&dup_mutex); compressed_buffer = cache_lookup(fwriter_buffer, index); pthread_cleanup_push((void *) pthread_mutex_unlock, &fragment_mutex); pthread_mutex_lock(&fragment_mutex); disk_fragment = &fragment_table[index]; size = SQUASHFS_COMPRESSED_SIZE_BLOCK(disk_fragment->size); start_block = disk_fragment->start_block; pthread_cleanup_pop(1); if(SQUASHFS_COMPRESSED_BLOCK(disk_fragment->size)) { int error; char *data; if(compressed_buffer) data = compressed_buffer->data; else { res = read_filesystem(fd, start_block, size, data_buffer); if(res == 0) { ERROR("Failed to read fragment from output" " filesystem\n"); BAD_ERROR("Output filesystem corrupted?\n"); } data = data_buffer; } res = compressor_uncompress(comp, buffer->data, data, size, block_size, &error); if(res == -1) BAD_ERROR("%s uncompress failed with error code %d\n", comp->name, error); } else if(compressed_buffer) memcpy(buffer->data, compressed_buffer->data, size); else { res = read_filesystem(fd, start_block, size, buffer->data); if(res == 0) { ERROR("Failed to read fragment from output " "filesystem\n"); BAD_ERROR("Output filesystem corrupted?\n"); } } cache_unlock(buffer); cache_block_put(compressed_buffer); finished: pthread_cleanup_pop(0); return buffer; } struct file_buffer *get_fragment_cksum(struct file_info *file, char *data_buffer, int fd, unsigned short *checksum) { struct file_buffer *frag_buffer; struct append_file *append; int index = file->fragment->index; frag_buffer = get_fragment(file->fragment, data_buffer, fd); pthread_cleanup_push((void *) pthread_mutex_unlock, &dup_mutex); for(append = file_mapping[index]; append; append = append->next) { int offset = append->file->fragment->offset; int size = append->file->fragment->size; char *data = frag_buffer->data + offset; unsigned short cksum = get_checksum_mem(data, size); if(file == append->file) *checksum = cksum; pthread_mutex_lock(&dup_mutex); append->file->fragment_checksum = cksum; append->file->have_frag_checksum = TRUE; pthread_mutex_unlock(&dup_mutex); } pthread_cleanup_pop(0); return frag_buffer; } void *frag_thrd(void *destination_file) { sigset_t sigmask, old_mask; char *data_buffer; int fd; sigemptyset(&sigmask); sigaddset(&sigmask, SIGINT); sigaddset(&sigmask, SIGTERM); sigaddset(&sigmask, SIGUSR1); pthread_sigmask(SIG_BLOCK, &sigmask, &old_mask); fd = open(destination_file, O_RDONLY); if(fd == -1) BAD_ERROR("frag_thrd: can't open destination for reading\n"); data_buffer = malloc(SQUASHFS_FILE_MAX_SIZE); if(data_buffer == NULL) MEM_ERROR(); pthread_cleanup_push((void *) pthread_mutex_unlock, &dup_mutex); while(1) { struct file_buffer *file_buffer = queue_get(to_process_frag); struct file_buffer *buffer; int sparse = checksum_sparse(file_buffer); struct file_info *dupl_ptr; long long file_size; unsigned short checksum; char flag; int res; if(sparse_files && sparse) { file_buffer->c_byte = 0; file_buffer->fragment = FALSE; } else file_buffer->c_byte = file_buffer->size; /* * Specutively pull into the fragment cache any fragment blocks * which contain fragments which *this* fragment may be * be a duplicate. * * By ensuring the fragment block is in cache ahead of time * should eliminate the parallelisation stall when the * main thread needs to read the fragment block to do a * duplicate check on it. * * If this is a fragment belonging to a larger file * (with additional blocks) then ignore it. Here we're * interested in the "low hanging fruit" of files which * consist of only a fragment */ if(file_buffer->file_size != file_buffer->size) { seq_queue_put(to_main, file_buffer); continue; } file_size = file_buffer->file_size; pthread_mutex_lock(&dup_mutex); dupl_ptr = dupl[DUP_HASH(file_size)]; pthread_mutex_unlock(&dup_mutex); file_buffer->dupl_start = dupl_ptr; file_buffer->duplicate = FALSE; for(; dupl_ptr; dupl_ptr = dupl_ptr->next) { if(file_size != dupl_ptr->file_size || file_size != dupl_ptr->fragment->size) continue; pthread_mutex_lock(&dup_mutex); flag = dupl_ptr->have_frag_checksum; checksum = dupl_ptr->fragment_checksum; pthread_mutex_unlock(&dup_mutex); /* * If we have the checksum and it matches then * read in the fragment block. * * If we *don't* have the checksum, then we are * appending, and the fragment block is on the * "old" filesystem. Read it in and checksum * the entire fragment buffer */ if(!flag) { buffer = get_fragment_cksum(dupl_ptr, data_buffer, fd, &checksum); if(checksum != file_buffer->checksum) { cache_block_put(buffer); continue; } } else if(checksum == file_buffer->checksum) buffer = get_fragment(dupl_ptr->fragment, data_buffer, fd); else continue; res = memcmp(file_buffer->data, buffer->data + dupl_ptr->fragment->offset, file_size); cache_block_put(buffer); if(res == 0) { struct file_buffer *dup = malloc(sizeof(*dup)); if(dup == NULL) MEM_ERROR(); memcpy(dup, file_buffer, sizeof(*dup)); cache_block_put(file_buffer); dup->dupl_start = dupl_ptr; dup->duplicate = TRUE; file_buffer = dup; break; } } seq_queue_put(to_main, file_buffer); } pthread_cleanup_pop(0); } squashfs4.3/squashfs-tools/lzo_wrapper.h0000644000175000017500000000372012306776317020552 0ustar phillipphillip#ifndef LZO_WRAPPER_H #define LZO_WRAPPER_H /* * Squashfs * * Copyright (c) 2013 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * lzo_wrapper.h * */ #ifndef linux #define __BYTE_ORDER BYTE_ORDER #define __BIG_ENDIAN BIG_ENDIAN #define __LITTLE_ENDIAN LITTLE_ENDIAN #else #include #endif #if __BYTE_ORDER == __BIG_ENDIAN extern unsigned int inswap_le32(unsigned int); #define SQUASHFS_INSWAP_COMP_OPTS(s) { \ (s)->algorithm = inswap_le32((s)->algorithm); \ (s)->compression_level = inswap_le32((s)->compression_level); \ } #else #define SQUASHFS_INSWAP_COMP_OPTS(s) #endif /* Define the compression flags recognised. */ #define SQUASHFS_LZO1X_1 0 #define SQUASHFS_LZO1X_1_11 1 #define SQUASHFS_LZO1X_1_12 2 #define SQUASHFS_LZO1X_1_15 3 #define SQUASHFS_LZO1X_999 4 /* Default compression level used by SQUASHFS_LZO1X_999 */ #define SQUASHFS_LZO1X_999_COMP_DEFAULT 8 struct lzo_comp_opts { int algorithm; int compression_level; }; struct lzo_algorithm { char *name; int size; int (*compress) (const lzo_bytep, lzo_uint, lzo_bytep, lzo_uintp, lzo_voidp); }; struct lzo_stream { void *workspace; void *buffer; }; #define LZO_MAX_EXPANSION(size) (size + (size / 16) + 64 + 3) int lzo1x_999_wrapper(const lzo_bytep, lzo_uint, lzo_bytep, lzo_uintp, lzo_voidp); #endif squashfs4.3/squashfs-tools/caches-queues-lists.c0000644000175000017500000003673412333330365022070 0ustar phillipphillip/* * Create a squashfs filesystem. This is a highly compressed read only * filesystem. * * Copyright (c) 2013, 2014 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * caches-queues-lists.c */ #include #include #include #include #include "error.h" #include "caches-queues-lists.h" extern int add_overflow(int, int); extern int multiply_overflow(int, int); #define TRUE 1 #define FALSE 0 struct queue *queue_init(int size) { struct queue *queue = malloc(sizeof(struct queue)); if(queue == NULL) MEM_ERROR(); if(add_overflow(size, 1) || multiply_overflow(size + 1, sizeof(void *))) BAD_ERROR("Size too large in queue_init\n"); queue->data = malloc(sizeof(void *) * (size + 1)); if(queue->data == NULL) MEM_ERROR(); queue->size = size + 1; queue->readp = queue->writep = 0; pthread_mutex_init(&queue->mutex, NULL); pthread_cond_init(&queue->empty, NULL); pthread_cond_init(&queue->full, NULL); return queue; } void queue_put(struct queue *queue, void *data) { int nextp; pthread_cleanup_push((void *) pthread_mutex_unlock, &queue->mutex); pthread_mutex_lock(&queue->mutex); while((nextp = (queue->writep + 1) % queue->size) == queue->readp) pthread_cond_wait(&queue->full, &queue->mutex); queue->data[queue->writep] = data; queue->writep = nextp; pthread_cond_signal(&queue->empty); pthread_cleanup_pop(1); } void *queue_get(struct queue *queue) { void *data; pthread_cleanup_push((void *) pthread_mutex_unlock, &queue->mutex); pthread_mutex_lock(&queue->mutex); while(queue->readp == queue->writep) pthread_cond_wait(&queue->empty, &queue->mutex); data = queue->data[queue->readp]; queue->readp = (queue->readp + 1) % queue->size; pthread_cond_signal(&queue->full); pthread_cleanup_pop(1); return data; } int queue_empty(struct queue *queue) { int empty; pthread_cleanup_push((void *) pthread_mutex_unlock, &queue->mutex); pthread_mutex_lock(&queue->mutex); empty = queue->readp == queue->writep; pthread_cleanup_pop(1); return empty; } void queue_flush(struct queue *queue) { pthread_cleanup_push((void *) pthread_mutex_unlock, &queue->mutex); pthread_mutex_lock(&queue->mutex); queue->readp = queue->writep; pthread_cleanup_pop(1); } void dump_queue(struct queue *queue) { pthread_cleanup_push((void *) pthread_mutex_unlock, &queue->mutex); pthread_mutex_lock(&queue->mutex); printf("\tMax size %d, size %d%s\n", queue->size - 1, queue->readp <= queue->writep ? queue->writep - queue->readp : queue->size - queue->readp + queue->writep, queue->readp == queue->writep ? " (EMPTY)" : ((queue->writep + 1) % queue->size) == queue->readp ? " (FULL)" : ""); pthread_cleanup_pop(1); } /* define seq queue hash tables */ #define CALCULATE_SEQ_HASH(N) CALCULATE_HASH(N) /* Called with the seq queue mutex held */ INSERT_HASH_TABLE(seq, struct seq_queue, CALCULATE_SEQ_HASH, sequence, seq) /* Called with the cache mutex held */ REMOVE_HASH_TABLE(seq, struct seq_queue, CALCULATE_SEQ_HASH, sequence, seq); static unsigned int sequence = 0; struct seq_queue *seq_queue_init() { struct seq_queue *queue = malloc(sizeof(struct seq_queue)); if(queue == NULL) MEM_ERROR(); memset(queue, 0, sizeof(struct seq_queue)); pthread_mutex_init(&queue->mutex, NULL); pthread_cond_init(&queue->wait, NULL); return queue; } void seq_queue_put(struct seq_queue *queue, struct file_buffer *entry) { pthread_cleanup_push((void *) pthread_mutex_unlock, &queue->mutex); pthread_mutex_lock(&queue->mutex); insert_seq_hash_table(queue, entry); if(entry->fragment) queue->fragment_count ++; else queue->block_count ++; if(entry->sequence == sequence) pthread_cond_signal(&queue->wait); pthread_cleanup_pop(1); } struct file_buffer *seq_queue_get(struct seq_queue *queue) { /* * Look-up buffer matching sequence in the queue, if found return * it, otherwise wait until it arrives */ int hash = CALCULATE_SEQ_HASH(sequence); struct file_buffer *entry; pthread_cleanup_push((void *) pthread_mutex_unlock, &queue->mutex); pthread_mutex_lock(&queue->mutex); while(1) { for(entry = queue->hash_table[hash]; entry; entry = entry->seq_next) if(entry->sequence == sequence) break; if(entry) { /* * found the buffer in the queue, decrement the * appropriate count, and remove from hash list */ if(entry->fragment) queue->fragment_count --; else queue->block_count --; remove_seq_hash_table(queue, entry); sequence ++; break; } /* entry not found, wait for it to arrive */ pthread_cond_wait(&queue->wait, &queue->mutex); } pthread_cleanup_pop(1); return entry; } void seq_queue_flush(struct seq_queue *queue) { int i; pthread_cleanup_push((void *) pthread_mutex_unlock, &queue->mutex); pthread_mutex_lock(&queue->mutex); for(i = 0; i < HASH_SIZE; i++) queue->hash_table[i] = NULL; queue->fragment_count = queue->block_count = 0; pthread_cleanup_pop(1); } void dump_seq_queue(struct seq_queue *queue, int fragment_queue) { int size; pthread_cleanup_push((void *) pthread_mutex_unlock, &queue->mutex); pthread_mutex_lock(&queue->mutex); size = fragment_queue ? queue->fragment_count : queue->block_count; printf("\tMax size unlimited, size %d%s\n", size, size == 0 ? " (EMPTY)" : ""); pthread_cleanup_pop(1); } /* define cache hash tables */ #define CALCULATE_CACHE_HASH(N) CALCULATE_HASH(llabs(N)) /* Called with the cache mutex held */ INSERT_HASH_TABLE(cache, struct cache, CALCULATE_CACHE_HASH, index, hash) /* Called with the cache mutex held */ REMOVE_HASH_TABLE(cache, struct cache, CALCULATE_CACHE_HASH, index, hash); /* define cache free list */ /* Called with the cache mutex held */ INSERT_LIST(free, struct file_buffer) /* Called with the cache mutex held */ REMOVE_LIST(free, struct file_buffer) struct cache *cache_init(int buffer_size, int max_buffers, int noshrink_lookup, int first_freelist) { struct cache *cache = malloc(sizeof(struct cache)); if(cache == NULL) MEM_ERROR(); cache->max_buffers = max_buffers; cache->buffer_size = buffer_size; cache->count = 0; cache->used = 0; cache->free_list = NULL; /* * The cache will grow up to max_buffers in size in response to * an increase in readhead/number of buffers in flight. But * once the outstanding buffers gets returned, we can either elect * to shrink the cache, or to put the freed blocks onto a free list. * * For the caches where we want to do lookup (fragment/writer), * a don't shrink policy is best, for the reader cache it * makes no sense to keep buffers around longer than necessary as * we don't do any lookup on those blocks. */ cache->noshrink_lookup = noshrink_lookup; /* * The default use freelist before growing cache policy behaves * poorly with appending - with many duplicates the caches * do not grow due to the fact that large queues of outstanding * fragments/writer blocks do not occur, leading to small caches * and un-uncessary performance loss to frequent cache * replacement in the small caches. Therefore with appending * change the policy to grow the caches before reusing blocks * from the freelist */ cache->first_freelist = first_freelist; memset(cache->hash_table, 0, sizeof(struct file_buffer *) * 65536); pthread_mutex_init(&cache->mutex, NULL); pthread_cond_init(&cache->wait_for_free, NULL); pthread_cond_init(&cache->wait_for_unlock, NULL); return cache; } struct file_buffer *cache_lookup(struct cache *cache, long long index) { /* Lookup block in the cache, if found return with usage count * incremented, if not found return NULL */ int hash = CALCULATE_CACHE_HASH(index); struct file_buffer *entry; pthread_cleanup_push((void *) pthread_mutex_unlock, &cache->mutex); pthread_mutex_lock(&cache->mutex); for(entry = cache->hash_table[hash]; entry; entry = entry->hash_next) if(entry->index == index) break; if(entry) { /* found the block in the cache, increment used count and * if necessary remove from free list so it won't disappear */ if(entry->used == 0) { remove_free_list(&cache->free_list, entry); cache->used ++; } entry->used ++; } pthread_cleanup_pop(1); return entry; } static struct file_buffer *cache_freelist(struct cache *cache) { struct file_buffer *entry = cache->free_list; remove_free_list(&cache->free_list, entry); /* a block on the free_list is hashed */ remove_cache_hash_table(cache, entry); cache->used ++; return entry; } static struct file_buffer *cache_alloc(struct cache *cache) { struct file_buffer *entry = malloc(sizeof(struct file_buffer) + cache->buffer_size); if(entry == NULL) MEM_ERROR(); entry->cache = cache; entry->free_prev = entry->free_next = NULL; cache->count ++; return entry; } static struct file_buffer *_cache_get(struct cache *cache, long long index, int hash) { /* Get a free block out of the cache indexed on index. */ struct file_buffer *entry = NULL; pthread_cleanup_push((void *) pthread_mutex_unlock, &cache->mutex); pthread_mutex_lock(&cache->mutex); while(1) { if(cache->noshrink_lookup) { /* first try to get a block from the free list */ if(cache->first_freelist && cache->free_list) entry = cache_freelist(cache); else if(cache->count < cache->max_buffers) { entry = cache_alloc(cache); cache->used ++; } else if(!cache->first_freelist && cache->free_list) entry = cache_freelist(cache); } else { /* shrinking non-lookup cache */ if(cache->count < cache->max_buffers) { entry = cache_alloc(cache); if(cache->count > cache->max_count) cache->max_count = cache->count; } } if(entry) break; /* wait for a block */ pthread_cond_wait(&cache->wait_for_free, &cache->mutex); } /* initialise block and if hash is set insert into the hash table */ entry->used = 1; entry->locked = FALSE; entry->wait_on_unlock = FALSE; entry->error = FALSE; if(hash) { entry->index = index; insert_cache_hash_table(cache, entry); } pthread_cleanup_pop(1); return entry; } struct file_buffer *cache_get(struct cache *cache, long long index) { return _cache_get(cache, index, 1); } struct file_buffer *cache_get_nohash(struct cache *cache) { return _cache_get(cache, 0, 0); } void cache_hash(struct file_buffer *entry, long long index) { struct cache *cache = entry->cache; pthread_cleanup_push((void *) pthread_mutex_unlock, &cache->mutex); pthread_mutex_lock(&cache->mutex); entry->index = index; insert_cache_hash_table(cache, entry); pthread_cleanup_pop(1); } void cache_block_put(struct file_buffer *entry) { struct cache *cache; /* * Finished with this cache entry, once the usage count reaches zero it * can be reused. * * If noshrink_lookup is set, put the block onto the free list. * As blocks remain accessible via the hash table they can be found * getting a new lease of life before they are reused. * * if noshrink_lookup is not set then shrink the cache. */ if(entry == NULL) return; cache = entry->cache; pthread_cleanup_push((void *) pthread_mutex_unlock, &cache->mutex); pthread_mutex_lock(&cache->mutex); entry->used --; if(entry->used == 0) { if(cache->noshrink_lookup) { insert_free_list(&cache->free_list, entry); cache->used --; } else { free(entry); cache->count --; } /* One or more threads may be waiting on this block */ pthread_cond_signal(&cache->wait_for_free); } pthread_cleanup_pop(1); } void dump_cache(struct cache *cache) { pthread_cleanup_push((void *) pthread_mutex_unlock, &cache->mutex); pthread_mutex_lock(&cache->mutex); if(cache->noshrink_lookup) printf("\tMax buffers %d, Current size %d, Used %d, %s\n", cache->max_buffers, cache->count, cache->used, cache->free_list ? "Free buffers" : "No free buffers"); else printf("\tMax buffers %d, Current size %d, Maximum historical " "size %d\n", cache->max_buffers, cache->count, cache->max_count); pthread_cleanup_pop(1); } struct file_buffer *cache_get_nowait(struct cache *cache, long long index) { struct file_buffer *entry = NULL; /* * block doesn't exist, create it, but return it with the * locked flag set, so nothing tries to use it while it doesn't * contain data. * * If there's no space in the cache then return NULL. */ pthread_cleanup_push((void *) pthread_mutex_unlock, &cache->mutex); pthread_mutex_lock(&cache->mutex); /* first try to get a block from the free list */ if(cache->first_freelist && cache->free_list) entry = cache_freelist(cache); else if(cache->count < cache->max_buffers) { entry = cache_alloc(cache); cache->used ++; } else if(!cache->first_freelist && cache->free_list) entry = cache_freelist(cache); if(entry) { /* initialise block and insert into the hash table */ entry->used = 1; entry->locked = TRUE; entry->wait_on_unlock = FALSE; entry->error = FALSE; entry->index = index; insert_cache_hash_table(cache, entry); } pthread_cleanup_pop(1); return entry; } struct file_buffer *cache_lookup_nowait(struct cache *cache, long long index, char *locked) { /* * Lookup block in the cache, if found return it with the locked flag * indicating whether it is currently locked. In both cases increment * the used count. * * If it doesn't exist in the cache return NULL; */ int hash = CALCULATE_CACHE_HASH(index); struct file_buffer *entry; pthread_cleanup_push((void *) pthread_mutex_unlock, &cache->mutex); pthread_mutex_lock(&cache->mutex); /* first check if the entry already exists */ for(entry = cache->hash_table[hash]; entry; entry = entry->hash_next) if(entry->index == index) break; if(entry) { if(entry->used == 0) { remove_free_list(&cache->free_list, entry); cache->used ++; } entry->used ++; *locked = entry->locked; } pthread_cleanup_pop(1); return entry; } void cache_wait_unlock(struct file_buffer *buffer) { struct cache *cache = buffer->cache; pthread_cleanup_push((void *) pthread_mutex_unlock, &cache->mutex); pthread_mutex_lock(&cache->mutex); while(buffer->locked) { /* * another thread is filling this in, wait until it * becomes unlocked. Used has been incremented to ensure it * doesn't get reused. By definition a block can't be * locked and unused, and so we don't need to worry * about it being on the freelist now, but, it may * become unused when unlocked unless used is * incremented */ buffer->wait_on_unlock = TRUE; pthread_cond_wait(&cache->wait_for_unlock, &cache->mutex); } pthread_cleanup_pop(1); } void cache_unlock(struct file_buffer *entry) { struct cache *cache = entry->cache; /* * Unlock this locked cache entry. If anything is waiting for this * to become unlocked, wake it up. */ pthread_cleanup_push((void *) pthread_mutex_unlock, &cache->mutex); pthread_mutex_lock(&cache->mutex); entry->locked = FALSE; if(entry->wait_on_unlock) { entry->wait_on_unlock = FALSE; pthread_cond_broadcast(&cache->wait_for_unlock); } pthread_cleanup_pop(1); } squashfs4.3/squashfs-tools/pseudo.h0000644000175000017500000000316312333330365017473 0ustar phillipphillip#ifndef PSEUDO_H #define PSEUDO_H /* * Create a squashfs filesystem. This is a highly compressed read only * filesystem. * * Copyright (c) 2009, 2010, 2014 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * pseudo.h */ struct pseudo_dev { char type; unsigned int mode; unsigned int uid; unsigned int gid; unsigned int major; unsigned int minor; int pseudo_id; char *command; }; struct pseudo_entry { char *name; char *pathname; struct pseudo *pseudo; struct pseudo_dev *dev; }; struct pseudo { int names; int count; struct pseudo_entry *name; }; extern int read_pseudo_def(char *); extern int read_pseudo_file(char *); extern struct pseudo *pseudo_subdir(char *, struct pseudo *); extern struct pseudo_entry *pseudo_readdir(struct pseudo *); extern struct pseudo_dev *get_pseudo_file(int); extern int pseudo_exec_file(struct pseudo_dev *, int *); extern struct pseudo *get_pseudo(); extern void dump_pseudos(); #endif squashfs4.3/squashfs-tools/squashfs_swap.h0000644000175000017500000004775112333330365021076 0ustar phillipphillip#ifndef SQUASHFS_SWAP_H #define SQUASHFS_SWAP_H /* * Squashfs * * Copyright (c) 2008, 2009, 2010, 2013, 2014 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * squashfs_swap.h */ /* * macros to convert each stucture from big endian to little endian */ #if __BYTE_ORDER == __BIG_ENDIAN #include extern void swap_le16(void *, void *); extern void swap_le32(void *, void *); extern void swap_le64(void *, void *); extern void swap_le16_num(void *, void *, int); extern void swap_le32_num(void *, void *, int); extern void swap_le64_num(void *, void *, int); extern unsigned short inswap_le16(unsigned short); extern unsigned int inswap_le32(unsigned int); extern long long inswap_le64(long long); extern void inswap_le16_num(unsigned short *, int); extern void inswap_le32_num(unsigned int *, int); extern void inswap_le64_num(long long *, int); #define _SQUASHFS_SWAP_SUPER_BLOCK(s, d, SWAP_FUNC) {\ SWAP_FUNC(32, s, d, s_magic, struct squashfs_super_block);\ SWAP_FUNC(32, s, d, inodes, struct squashfs_super_block);\ SWAP_FUNC##S(32, s, d, mkfs_time, struct squashfs_super_block);\ SWAP_FUNC(32, s, d, block_size, struct squashfs_super_block);\ SWAP_FUNC(32, s, d, fragments, struct squashfs_super_block);\ SWAP_FUNC(16, s, d, compression, struct squashfs_super_block);\ SWAP_FUNC(16, s, d, block_log, struct squashfs_super_block);\ SWAP_FUNC(16, s, d, flags, struct squashfs_super_block);\ SWAP_FUNC(16, s, d, no_ids, struct squashfs_super_block);\ SWAP_FUNC(16, s, d, s_major, struct squashfs_super_block);\ SWAP_FUNC(16, s, d, s_minor, struct squashfs_super_block);\ SWAP_FUNC(64, s, d, root_inode, struct squashfs_super_block);\ SWAP_FUNC(64, s, d, bytes_used, struct squashfs_super_block);\ SWAP_FUNC(64, s, d, id_table_start, struct squashfs_super_block);\ SWAP_FUNC(64, s, d, xattr_id_table_start, struct squashfs_super_block);\ SWAP_FUNC(64, s, d, inode_table_start, struct squashfs_super_block);\ SWAP_FUNC(64, s, d, directory_table_start, struct squashfs_super_block);\ SWAP_FUNC(64, s, d, fragment_table_start, struct squashfs_super_block);\ SWAP_FUNC(64, s, d, lookup_table_start, struct squashfs_super_block);\ } #define _SQUASHFS_SWAP_DIR_INDEX(s, d, SWAP_FUNC) {\ SWAP_FUNC(32, s, d, index, struct squashfs_dir_index);\ SWAP_FUNC(32, s, d, start_block, struct squashfs_dir_index);\ SWAP_FUNC(32, s, d, size, struct squashfs_dir_index);\ } #define _SQUASHFS_SWAP_BASE_INODE_HEADER(s, d, SWAP_FUNC) {\ SWAP_FUNC(16, s, d, inode_type, struct squashfs_base_inode_header);\ SWAP_FUNC(16, s, d, mode, struct squashfs_base_inode_header);\ SWAP_FUNC(16, s, d, uid, struct squashfs_base_inode_header);\ SWAP_FUNC(16, s, d, guid, struct squashfs_base_inode_header);\ SWAP_FUNC##S(32, s, d, mtime, struct squashfs_base_inode_header);\ SWAP_FUNC(32, s, d, inode_number, struct squashfs_base_inode_header);\ } #define _SQUASHFS_SWAP_IPC_INODE_HEADER(s, d, SWAP_FUNC) {\ SWAP_FUNC(16, s, d, inode_type, struct squashfs_ipc_inode_header);\ SWAP_FUNC(16, s, d, mode, struct squashfs_ipc_inode_header);\ SWAP_FUNC(16, s, d, uid, struct squashfs_ipc_inode_header);\ SWAP_FUNC(16, s, d, guid, struct squashfs_ipc_inode_header);\ SWAP_FUNC##S(32, s, d, mtime, struct squashfs_ipc_inode_header);\ SWAP_FUNC(32, s, d, inode_number, struct squashfs_ipc_inode_header);\ SWAP_FUNC(32, s, d, nlink, struct squashfs_ipc_inode_header);\ } #define _SQUASHFS_SWAP_LIPC_INODE_HEADER(s, d, SWAP_FUNC) {\ SWAP_FUNC(16, s, d, inode_type, struct squashfs_lipc_inode_header);\ SWAP_FUNC(16, s, d, mode, struct squashfs_lipc_inode_header);\ SWAP_FUNC(16, s, d, uid, struct squashfs_lipc_inode_header);\ SWAP_FUNC(16, s, d, guid, struct squashfs_lipc_inode_header);\ SWAP_FUNC##S(32, s, d, mtime, struct squashfs_lipc_inode_header);\ SWAP_FUNC(32, s, d, inode_number, struct squashfs_lipc_inode_header);\ SWAP_FUNC(32, s, d, nlink, struct squashfs_lipc_inode_header);\ SWAP_FUNC(32, s, d, xattr, struct squashfs_lipc_inode_header);\ } #define _SQUASHFS_SWAP_DEV_INODE_HEADER(s, d, SWAP_FUNC) {\ SWAP_FUNC(16, s, d, inode_type, struct squashfs_dev_inode_header);\ SWAP_FUNC(16, s, d, mode, struct squashfs_dev_inode_header);\ SWAP_FUNC(16, s, d, uid, struct squashfs_dev_inode_header);\ SWAP_FUNC(16, s, d, guid, struct squashfs_dev_inode_header);\ SWAP_FUNC##S(32, s, d, mtime, struct squashfs_dev_inode_header);\ SWAP_FUNC(32, s, d, inode_number, struct squashfs_dev_inode_header);\ SWAP_FUNC(32, s, d, nlink, struct squashfs_dev_inode_header);\ SWAP_FUNC(32, s, d, rdev, struct squashfs_dev_inode_header);\ } #define _SQUASHFS_SWAP_LDEV_INODE_HEADER(s, d, SWAP_FUNC) {\ SWAP_FUNC(16, s, d, inode_type, struct squashfs_ldev_inode_header);\ SWAP_FUNC(16, s, d, mode, struct squashfs_ldev_inode_header);\ SWAP_FUNC(16, s, d, uid, struct squashfs_ldev_inode_header);\ SWAP_FUNC(16, s, d, guid, struct squashfs_ldev_inode_header);\ SWAP_FUNC##S(32, s, d, mtime, struct squashfs_ldev_inode_header);\ SWAP_FUNC(32, s, d, inode_number, struct squashfs_ldev_inode_header);\ SWAP_FUNC(32, s, d, nlink, struct squashfs_ldev_inode_header);\ SWAP_FUNC(32, s, d, rdev, struct squashfs_ldev_inode_header);\ SWAP_FUNC(32, s, d, xattr, struct squashfs_ldev_inode_header);\ } #define _SQUASHFS_SWAP_SYMLINK_INODE_HEADER(s, d, SWAP_FUNC) {\ SWAP_FUNC(16, s, d, inode_type, struct squashfs_symlink_inode_header);\ SWAP_FUNC(16, s, d, mode, struct squashfs_symlink_inode_header);\ SWAP_FUNC(16, s, d, uid, struct squashfs_symlink_inode_header);\ SWAP_FUNC(16, s, d, guid, struct squashfs_symlink_inode_header);\ SWAP_FUNC##S(32, s, d, mtime, struct squashfs_symlink_inode_header);\ SWAP_FUNC(32, s, d, inode_number, struct squashfs_symlink_inode_header);\ SWAP_FUNC(32, s, d, nlink, struct squashfs_symlink_inode_header);\ SWAP_FUNC(32, s, d, symlink_size, struct squashfs_symlink_inode_header);\ } #define _SQUASHFS_SWAP_REG_INODE_HEADER(s, d, SWAP_FUNC) {\ SWAP_FUNC(16, s, d, inode_type, struct squashfs_reg_inode_header);\ SWAP_FUNC(16, s, d, mode, struct squashfs_reg_inode_header);\ SWAP_FUNC(16, s, d, uid, struct squashfs_reg_inode_header);\ SWAP_FUNC(16, s, d, guid, struct squashfs_reg_inode_header);\ SWAP_FUNC##S(32, s, d, mtime, struct squashfs_reg_inode_header);\ SWAP_FUNC(32, s, d, inode_number, struct squashfs_reg_inode_header);\ SWAP_FUNC(32, s, d, start_block, struct squashfs_reg_inode_header);\ SWAP_FUNC(32, s, d, fragment, struct squashfs_reg_inode_header);\ SWAP_FUNC(32, s, d, offset, struct squashfs_reg_inode_header);\ SWAP_FUNC(32, s, d, file_size, struct squashfs_reg_inode_header);\ } #define _SQUASHFS_SWAP_LREG_INODE_HEADER(s, d, SWAP_FUNC) {\ SWAP_FUNC(16, s, d, inode_type, struct squashfs_lreg_inode_header);\ SWAP_FUNC(16, s, d, mode, struct squashfs_lreg_inode_header);\ SWAP_FUNC(16, s, d, uid, struct squashfs_lreg_inode_header);\ SWAP_FUNC(16, s, d, guid, struct squashfs_lreg_inode_header);\ SWAP_FUNC##S(32, s, d, mtime, struct squashfs_lreg_inode_header);\ SWAP_FUNC(32, s, d, inode_number, struct squashfs_lreg_inode_header);\ SWAP_FUNC(64, s, d, start_block, struct squashfs_lreg_inode_header);\ SWAP_FUNC(64, s, d, file_size, struct squashfs_lreg_inode_header);\ SWAP_FUNC(64, s, d, sparse, struct squashfs_lreg_inode_header);\ SWAP_FUNC(32, s, d, nlink, struct squashfs_lreg_inode_header);\ SWAP_FUNC(32, s, d, fragment, struct squashfs_lreg_inode_header);\ SWAP_FUNC(32, s, d, offset, struct squashfs_lreg_inode_header);\ SWAP_FUNC(32, s, d, xattr, struct squashfs_lreg_inode_header);\ } #define _SQUASHFS_SWAP_DIR_INODE_HEADER(s, d, SWAP_FUNC) {\ SWAP_FUNC(16, s, d, inode_type, struct squashfs_dir_inode_header);\ SWAP_FUNC(16, s, d, mode, struct squashfs_dir_inode_header);\ SWAP_FUNC(16, s, d, uid, struct squashfs_dir_inode_header);\ SWAP_FUNC(16, s, d, guid, struct squashfs_dir_inode_header);\ SWAP_FUNC##S(32, s, d, mtime, struct squashfs_dir_inode_header);\ SWAP_FUNC(32, s, d, inode_number, struct squashfs_dir_inode_header);\ SWAP_FUNC(32, s, d, start_block, struct squashfs_dir_inode_header);\ SWAP_FUNC(32, s, d, nlink, struct squashfs_dir_inode_header);\ SWAP_FUNC(16, s, d, file_size, struct squashfs_dir_inode_header);\ SWAP_FUNC(16, s, d, offset, struct squashfs_dir_inode_header);\ SWAP_FUNC(32, s, d, parent_inode, struct squashfs_dir_inode_header);\ } #define _SQUASHFS_SWAP_LDIR_INODE_HEADER(s, d, SWAP_FUNC) {\ SWAP_FUNC(16, s, d, inode_type, struct squashfs_ldir_inode_header);\ SWAP_FUNC(16, s, d, mode, struct squashfs_ldir_inode_header);\ SWAP_FUNC(16, s, d, uid, struct squashfs_ldir_inode_header);\ SWAP_FUNC(16, s, d, guid, struct squashfs_ldir_inode_header);\ SWAP_FUNC##S(32, s, d, mtime, struct squashfs_ldir_inode_header);\ SWAP_FUNC(32, s, d, inode_number, struct squashfs_ldir_inode_header);\ SWAP_FUNC(32, s, d, nlink, struct squashfs_ldir_inode_header);\ SWAP_FUNC(32, s, d, file_size, struct squashfs_ldir_inode_header);\ SWAP_FUNC(32, s, d, start_block, struct squashfs_ldir_inode_header);\ SWAP_FUNC(32, s, d, parent_inode, struct squashfs_ldir_inode_header);\ SWAP_FUNC(16, s, d, i_count, struct squashfs_ldir_inode_header);\ SWAP_FUNC(16, s, d, offset, struct squashfs_ldir_inode_header);\ SWAP_FUNC(32, s, d, xattr, struct squashfs_ldir_inode_header);\ } #define _SQUASHFS_SWAP_DIR_ENTRY(s, d, SWAP_FUNC) {\ SWAP_FUNC(16, s, d, offset, struct squashfs_dir_entry);\ SWAP_FUNC##S(16, s, d, inode_number, struct squashfs_dir_entry);\ SWAP_FUNC(16, s, d, type, struct squashfs_dir_entry);\ SWAP_FUNC(16, s, d, size, struct squashfs_dir_entry);\ } #define _SQUASHFS_SWAP_DIR_HEADER(s, d, SWAP_FUNC) {\ SWAP_FUNC(32, s, d, count, struct squashfs_dir_header);\ SWAP_FUNC(32, s, d, start_block, struct squashfs_dir_header);\ SWAP_FUNC(32, s, d, inode_number, struct squashfs_dir_header);\ } #define _SQUASHFS_SWAP_FRAGMENT_ENTRY(s, d, SWAP_FUNC) {\ SWAP_FUNC(64, s, d, start_block, struct squashfs_fragment_entry);\ SWAP_FUNC(32, s, d, size, struct squashfs_fragment_entry);\ } #define _SQUASHFS_SWAP_XATTR_ENTRY(s, d, SWAP_FUNC) {\ SWAP_FUNC(16, s, d, type, struct squashfs_xattr_entry);\ SWAP_FUNC(16, s, d, size, struct squashfs_xattr_entry);\ } #define _SQUASHFS_SWAP_XATTR_VAL(s, d, SWAP_FUNC) {\ SWAP_FUNC(32, s, d, vsize, struct squashfs_xattr_val);\ } #define _SQUASHFS_SWAP_XATTR_ID(s, d, SWAP_FUNC) {\ SWAP_FUNC(64, s, d, xattr, struct squashfs_xattr_id);\ SWAP_FUNC(32, s, d, count, struct squashfs_xattr_id);\ SWAP_FUNC(32, s, d, size, struct squashfs_xattr_id);\ } #define _SQUASHFS_SWAP_XATTR_TABLE(s, d, SWAP_FUNC) {\ SWAP_FUNC(64, s, d, xattr_table_start, struct squashfs_xattr_table);\ SWAP_FUNC(32, s, d, xattr_ids, struct squashfs_xattr_table);\ } /* big endian architecture copy and swap macros */ #define SQUASHFS_SWAP_SUPER_BLOCK(s, d) \ _SQUASHFS_SWAP_SUPER_BLOCK(s, d, SWAP_LE) #define SQUASHFS_SWAP_DIR_INDEX(s, d) \ _SQUASHFS_SWAP_DIR_INDEX(s, d, SWAP_LE) #define SQUASHFS_SWAP_BASE_INODE_HEADER(s, d) \ _SQUASHFS_SWAP_BASE_INODE_HEADER(s, d, SWAP_LE) #define SQUASHFS_SWAP_IPC_INODE_HEADER(s, d) \ _SQUASHFS_SWAP_IPC_INODE_HEADER(s, d, SWAP_LE) #define SQUASHFS_SWAP_LIPC_INODE_HEADER(s, d) \ _SQUASHFS_SWAP_LIPC_INODE_HEADER(s, d, SWAP_LE) #define SQUASHFS_SWAP_DEV_INODE_HEADER(s, d) \ _SQUASHFS_SWAP_DEV_INODE_HEADER(s, d, SWAP_LE) #define SQUASHFS_SWAP_LDEV_INODE_HEADER(s, d) \ _SQUASHFS_SWAP_LDEV_INODE_HEADER(s, d, SWAP_LE) #define SQUASHFS_SWAP_SYMLINK_INODE_HEADER(s, d) \ _SQUASHFS_SWAP_SYMLINK_INODE_HEADER(s, d, SWAP_LE) #define SQUASHFS_SWAP_REG_INODE_HEADER(s, d) \ _SQUASHFS_SWAP_REG_INODE_HEADER(s, d, SWAP_LE) #define SQUASHFS_SWAP_LREG_INODE_HEADER(s, d) \ _SQUASHFS_SWAP_LREG_INODE_HEADER(s, d, SWAP_LE) #define SQUASHFS_SWAP_DIR_INODE_HEADER(s, d) \ _SQUASHFS_SWAP_DIR_INODE_HEADER(s, d, SWAP_LE) #define SQUASHFS_SWAP_LDIR_INODE_HEADER(s, d) \ _SQUASHFS_SWAP_LDIR_INODE_HEADER(s, d, SWAP_LE) #define SQUASHFS_SWAP_DIR_ENTRY(s, d) \ _SQUASHFS_SWAP_DIR_ENTRY(s, d, SWAP_LE) #define SQUASHFS_SWAP_DIR_HEADER(s, d) \ _SQUASHFS_SWAP_DIR_HEADER(s, d, SWAP_LE) #define SQUASHFS_SWAP_FRAGMENT_ENTRY(s, d) \ _SQUASHFS_SWAP_FRAGMENT_ENTRY(s, d, SWAP_LE) #define SQUASHFS_SWAP_XATTR_ENTRY(s, d) \ _SQUASHFS_SWAP_XATTR_ENTRY(s, d, SWAP_LE) #define SQUASHFS_SWAP_XATTR_VAL(s, d) \ _SQUASHFS_SWAP_XATTR_VAL(s, d, SWAP_LE) #define SQUASHFS_SWAP_XATTR_ID(s, d) \ _SQUASHFS_SWAP_XATTR_ID(s, d, SWAP_LE) #define SQUASHFS_SWAP_XATTR_TABLE(s, d) \ _SQUASHFS_SWAP_XATTR_TABLE(s, d, SWAP_LE) #define SWAP_LE(bits, s, d, field, type) \ SWAP_LE##bits(((void *)(s)) + offsetof(type, field), \ ((void *)(d)) + offsetof(type, field)) #define SWAP_LES(bits, s, d, field, type) \ SWAP_LE(bits, s, d, field, type) #define SQUASHFS_SWAP_INODE_T(s, d) SQUASHFS_SWAP_LONG_LONGS(s, d, 1) #define SQUASHFS_SWAP_FRAGMENT_INDEXES(s, d, n) \ SQUASHFS_SWAP_LONG_LONGS(s, d, n) #define SQUASHFS_SWAP_LOOKUP_BLOCKS(s, d, n) SQUASHFS_SWAP_LONG_LONGS(s, d, n) #define SQUASHFS_SWAP_ID_BLOCKS(s, d, n) SQUASHFS_SWAP_LONG_LONGS(s, d, n) #define SQUASHFS_SWAP_SHORTS(s, d, n) swap_le16_num(s, d, n) #define SQUASHFS_SWAP_INTS(s, d, n) swap_le32_num(s, d, n) #define SQUASHFS_SWAP_LONG_LONGS(s, d, n) swap_le64_num(s, d, n) #define SWAP_LE16(s, d) swap_le16(s, d) #define SWAP_LE32(s, d) swap_le32(s, d) #define SWAP_LE64(s, d) swap_le64(s, d) /* big endian architecture swap in-place macros */ #define SQUASHFS_INSWAP_SUPER_BLOCK(s) \ _SQUASHFS_SWAP_SUPER_BLOCK(s, s, INSWAP_LE) #define SQUASHFS_INSWAP_DIR_INDEX(s) \ _SQUASHFS_SWAP_DIR_INDEX(s, s, INSWAP_LE) #define SQUASHFS_INSWAP_BASE_INODE_HEADER(s) \ _SQUASHFS_SWAP_BASE_INODE_HEADER(s, s, INSWAP_LE) #define SQUASHFS_INSWAP_IPC_INODE_HEADER(s) \ _SQUASHFS_SWAP_IPC_INODE_HEADER(s, s, INSWAP_LE) #define SQUASHFS_INSWAP_LIPC_INODE_HEADER(s) \ _SQUASHFS_SWAP_LIPC_INODE_HEADER(s, s, INSWAP_LE) #define SQUASHFS_INSWAP_DEV_INODE_HEADER(s) \ _SQUASHFS_SWAP_DEV_INODE_HEADER(s, s, INSWAP_LE) #define SQUASHFS_INSWAP_LDEV_INODE_HEADER(s) \ _SQUASHFS_SWAP_LDEV_INODE_HEADER(s, s, INSWAP_LE) #define SQUASHFS_INSWAP_SYMLINK_INODE_HEADER(s) \ _SQUASHFS_SWAP_SYMLINK_INODE_HEADER(s, s, INSWAP_LE) #define SQUASHFS_INSWAP_REG_INODE_HEADER(s) \ _SQUASHFS_SWAP_REG_INODE_HEADER(s, s, INSWAP_LE) #define SQUASHFS_INSWAP_LREG_INODE_HEADER(s) \ _SQUASHFS_SWAP_LREG_INODE_HEADER(s, s, INSWAP_LE) #define SQUASHFS_INSWAP_DIR_INODE_HEADER(s) \ _SQUASHFS_SWAP_DIR_INODE_HEADER(s, s, INSWAP_LE) #define SQUASHFS_INSWAP_LDIR_INODE_HEADER(s) \ _SQUASHFS_SWAP_LDIR_INODE_HEADER(s, s, INSWAP_LE) #define SQUASHFS_INSWAP_DIR_ENTRY(s) \ _SQUASHFS_SWAP_DIR_ENTRY(s, s, INSWAP_LE) #define SQUASHFS_INSWAP_DIR_HEADER(s) \ _SQUASHFS_SWAP_DIR_HEADER(s, s, INSWAP_LE) #define SQUASHFS_INSWAP_FRAGMENT_ENTRY(s) \ _SQUASHFS_SWAP_FRAGMENT_ENTRY(s, s, INSWAP_LE) #define SQUASHFS_INSWAP_XATTR_ENTRY(s) \ _SQUASHFS_SWAP_XATTR_ENTRY(s, s, INSWAP_LE) #define SQUASHFS_INSWAP_XATTR_VAL(s) \ _SQUASHFS_SWAP_XATTR_VAL(s, s, INSWAP_LE) #define SQUASHFS_INSWAP_XATTR_ID(s) \ _SQUASHFS_SWAP_XATTR_ID(s, s, INSWAP_LE) #define SQUASHFS_INSWAP_XATTR_TABLE(s) \ _SQUASHFS_SWAP_XATTR_TABLE(s, s, INSWAP_LE) #define INSWAP_LE(bits, s, d, field, type) \ (s)->field = inswap_le##bits((s)->field) #define INSWAP_LES(bits, s, d, field, type) \ (s)->field = INSWAP_LES##bits((s)->field) #define INSWAP_LES16(num) (short) inswap_le16((unsigned short) (num)) #define INSWAP_LES32(num) (int) inswap_le32((unsigned int) (num)) #define SQUASHFS_INSWAP_INODE_T(s) s = inswap_le64(s) #define SQUASHFS_INSWAP_FRAGMENT_INDEXES(s, n) inswap_le64_num(s, n) #define SQUASHFS_INSWAP_LOOKUP_BLOCKS(s, n) inswap_le64_num(s, n) #define SQUASHFS_INSWAP_ID_BLOCKS(s, n) inswap_le64_num(s, n) #define SQUASHFS_INSWAP_SHORTS(s, n) inswap_le16_num(s, n) #define SQUASHFS_INSWAP_INTS(s, n) inswap_le32_num(s, n) #define SQUASHFS_INSWAP_LONG_LONGS(s, n) inswap_le64_num(s, n) #else /* little endian architecture, just copy */ #define SQUASHFS_SWAP_SUPER_BLOCK(s, d) \ SQUASHFS_MEMCPY(s, d, sizeof(struct squashfs_super_block)) #define SQUASHFS_SWAP_DIR_INDEX(s, d) \ SQUASHFS_MEMCPY(s, d, sizeof(struct squashfs_dir_index)) #define SQUASHFS_SWAP_BASE_INODE_HEADER(s, d) \ SQUASHFS_MEMCPY(s, d, sizeof(struct squashfs_base_inode_header)) #define SQUASHFS_SWAP_IPC_INODE_HEADER(s, d) \ SQUASHFS_MEMCPY(s, d, sizeof(struct squashfs_ipc_inode_header)) #define SQUASHFS_SWAP_LIPC_INODE_HEADER(s, d) \ SQUASHFS_MEMCPY(s, d, sizeof(struct squashfs_lipc_inode_header)) #define SQUASHFS_SWAP_DEV_INODE_HEADER(s, d) \ SQUASHFS_MEMCPY(s, d, sizeof(struct squashfs_dev_inode_header)) #define SQUASHFS_SWAP_LDEV_INODE_HEADER(s, d) \ SQUASHFS_MEMCPY(s, d, sizeof(struct squashfs_ldev_inode_header)) #define SQUASHFS_SWAP_SYMLINK_INODE_HEADER(s, d) \ SQUASHFS_MEMCPY(s, d, sizeof(struct squashfs_symlink_inode_header)) #define SQUASHFS_SWAP_REG_INODE_HEADER(s, d) \ SQUASHFS_MEMCPY(s, d, sizeof(struct squashfs_reg_inode_header)) #define SQUASHFS_SWAP_LREG_INODE_HEADER(s, d) \ SQUASHFS_MEMCPY(s, d, sizeof(struct squashfs_lreg_inode_header)) #define SQUASHFS_SWAP_DIR_INODE_HEADER(s, d) \ SQUASHFS_MEMCPY(s, d, sizeof(struct squashfs_dir_inode_header)) #define SQUASHFS_SWAP_LDIR_INODE_HEADER(s, d) \ SQUASHFS_MEMCPY(s, d, sizeof(struct squashfs_ldir_inode_header)) #define SQUASHFS_SWAP_DIR_ENTRY(s, d) \ SQUASHFS_MEMCPY(s, d, sizeof(struct squashfs_dir_entry)) #define SQUASHFS_SWAP_DIR_HEADER(s, d) \ SQUASHFS_MEMCPY(s, d, sizeof(struct squashfs_dir_header)) #define SQUASHFS_SWAP_FRAGMENT_ENTRY(s, d) \ SQUASHFS_MEMCPY(s, d, sizeof(struct squashfs_fragment_entry)) #define SQUASHFS_SWAP_XATTR_ENTRY(s, d) \ SQUASHFS_MEMCPY(s, d, sizeof(struct squashfs_xattr_entry)) #define SQUASHFS_SWAP_XATTR_VAL(s, d) \ SQUASHFS_MEMCPY(s, d, sizeof(struct squashfs_xattr_val)) #define SQUASHFS_SWAP_XATTR_ID(s, d) \ SQUASHFS_MEMCPY(s, d, sizeof(struct squashfs_xattr_id)) #define SQUASHFS_SWAP_XATTR_TABLE(s, d) \ SQUASHFS_MEMCPY(s, d, sizeof(struct squashfs_xattr_table)) #define SQUASHFS_SWAP_INODE_T(s, d) SQUASHFS_SWAP_LONG_LONGS(s, d, 1) #define SQUASHFS_SWAP_FRAGMENT_INDEXES(s, d, n) \ SQUASHFS_SWAP_LONG_LONGS(s, d, n) #define SQUASHFS_SWAP_LOOKUP_BLOCKS(s, d, n) SQUASHFS_SWAP_LONG_LONGS(s, d, n) #define SQUASHFS_SWAP_ID_BLOCKS(s, d, n) SQUASHFS_SWAP_LONG_LONGS(s, d, n) #define SQUASHFS_MEMCPY(s, d, n) memcpy(d, s, n) #define SQUASHFS_SWAP_SHORTS(s, d, n) memcpy(d, s, n * sizeof(short)) #define SQUASHFS_SWAP_INTS(s, d, n) memcpy(d, s, n * sizeof(int)) #define SQUASHFS_SWAP_LONG_LONGS(s, d, n) \ memcpy(d, s, n * sizeof(long long)) /* little endian architecture, data already in place so do nothing */ #define SQUASHFS_INSWAP_SUPER_BLOCK(s) #define SQUASHFS_INSWAP_DIR_INDEX(s) #define SQUASHFS_INSWAP_BASE_INODE_HEADER(s) #define SQUASHFS_INSWAP_IPC_INODE_HEADER(s) #define SQUASHFS_INSWAP_LIPC_INODE_HEADER(s) #define SQUASHFS_INSWAP_DEV_INODE_HEADER(s) #define SQUASHFS_INSWAP_LDEV_INODE_HEADER(s) #define SQUASHFS_INSWAP_SYMLINK_INODE_HEADER(s) #define SQUASHFS_INSWAP_REG_INODE_HEADER(s) #define SQUASHFS_INSWAP_LREG_INODE_HEADER(s) #define SQUASHFS_INSWAP_DIR_INODE_HEADER(s) #define SQUASHFS_INSWAP_LDIR_INODE_HEADER(s) #define SQUASHFS_INSWAP_DIR_ENTRY(s) #define SQUASHFS_INSWAP_DIR_HEADER(s) #define SQUASHFS_INSWAP_FRAGMENT_ENTRY(s) #define SQUASHFS_INSWAP_XATTR_ENTRY(s) #define SQUASHFS_INSWAP_XATTR_VAL(s) #define SQUASHFS_INSWAP_XATTR_ID(s) #define SQUASHFS_INSWAP_XATTR_TABLE(s) #define SQUASHFS_INSWAP_INODE_T(s) #define SQUASHFS_INSWAP_FRAGMENT_INDEXES(s, n) #define SQUASHFS_INSWAP_LOOKUP_BLOCKS(s, n) #define SQUASHFS_INSWAP_ID_BLOCKS(s, n) #define SQUASHFS_INSWAP_SHORTS(s, n) #define SQUASHFS_INSWAP_INTS(s, n) #define SQUASHFS_INSWAP_LONG_LONGS(s, n) #endif #endif squashfs4.3/squashfs-tools/restore.h0000644000175000017500000000170712333330365017661 0ustar phillipphillip#ifndef RESTORE_H #define RESTORE_H /* * Create a squashfs filesystem. This is a highly compressed read only * filesystem. * * Copyright (c) 2013, 2014 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * restore.h */ extern pthread_t *init_restore_thread(); #endif squashfs4.3/squashfs-tools/mksquashfs.h0000644000175000017500000000762212333330365020365 0ustar phillipphillip#ifndef MKSQUASHFS_H #define MKSQUASHFS_H /* * Squashfs * * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011 * 2012, 2013, 2014 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * mksquashfs.h * */ struct dir_info { char *pathname; char *subpath; unsigned int count; unsigned int directory_count; int depth; unsigned int excluded; char dir_is_ldir; struct dir_ent *dir_ent; struct dir_ent *list; DIR *linuxdir; }; struct dir_ent { char *name; char *source_name; char *nonstandard_pathname; struct inode_info *inode; struct dir_info *dir; struct dir_info *our_dir; struct dir_ent *next; }; struct inode_info { struct stat buf; struct inode_info *next; squashfs_inode inode; unsigned int inode_number; unsigned int nlink; int pseudo_id; char type; char read; char root_entry; char pseudo_file; char no_fragments; char always_use_fragments; char noD; char noF; }; /* in memory file info */ struct file_info { long long file_size; long long bytes; long long start; unsigned int *block_list; struct file_info *next; struct fragment *fragment; unsigned short checksum; unsigned short fragment_checksum; char have_frag_checksum; char have_checksum; }; /* fragment block data structures */ struct fragment { unsigned int index; int offset; int size; }; /* in memory uid tables */ #define ID_ENTRIES 256 #define ID_HASH(id) (id & (ID_ENTRIES - 1)) #define ISA_UID 1 #define ISA_GID 2 struct id { unsigned int id; int index; char flags; struct id *next; }; /* fragment to file mapping used when appending */ struct append_file { struct file_info *file; struct append_file *next; }; #define PSEUDO_FILE_OTHER 1 #define PSEUDO_FILE_PROCESS 2 #define IS_PSEUDO(a) ((a)->pseudo_file) #define IS_PSEUDO_PROCESS(a) ((a)->pseudo_file & PSEUDO_FILE_PROCESS) #define IS_PSEUDO_OTHER(a) ((a)->pseudo_file & PSEUDO_FILE_OTHER) /* * Amount of physical memory to use by default, and the default queue * ratios */ #define SQUASHFS_TAKE 4 #define SQUASHFS_READQ_MEM 4 #define SQUASHFS_BWRITEQ_MEM 4 #define SQUASHFS_FWRITEQ_MEM 4 /* * Lowest amount of physical memory considered viable for Mksquashfs * to run in Mbytes */ #define SQUASHFS_LOWMEM 64 /* offset of data in compressed metadata blocks (allowing room for * compressed size */ #define BLOCK_OFFSET 2 extern struct cache *reader_buffer, *fragment_buffer, *reserve_cache; struct cache *bwriter_buffer, *fwriter_buffer; extern struct queue *to_reader, *to_deflate, *to_writer, *from_writer, *to_frag, *locked_fragment, *to_process_frag; extern struct append_file **file_mapping; extern struct seq_queue *to_main; extern pthread_mutex_t fragment_mutex, dup_mutex; extern struct squashfs_fragment_entry *fragment_table; extern struct compressor *comp; extern int block_size; extern struct file_info *dupl[]; extern int read_fs_bytes(int, long long, int, void *); extern void add_file(long long, long long, long long, unsigned int *, int, unsigned int, int, int); extern struct id *create_id(unsigned int); extern unsigned int get_uid(unsigned int); extern unsigned int get_guid(unsigned int); extern int read_bytes(int, void *, int); extern unsigned short get_checksum_mem(char *, int); #endif squashfs4.3/squashfs-tools/info.h0000644000175000017500000000176512333330365017135 0ustar phillipphillip#ifndef INFO_H #define INFO_H /* * Create a squashfs filesystem. This is a highly compressed read only * filesystem. * * Copyright (c) 2013, 2014 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * info.h */ extern void disable_info(); extern void update_info(struct dir_ent *); extern void init_info(); #endif squashfs4.3/squashfs-tools/compressor.h0000644000175000017500000000651712333330365020376 0ustar phillipphillip#ifndef COMPRESSOR_H #define COMPRESSOR_H /* * * Copyright (c) 2009, 2010, 2011, 2012, 2013, 2014 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * compressor.h */ struct compressor { int id; char *name; int supported; int (*init)(void **, int, int); int (*compress)(void *, void *, void *, int, int, int *); int (*uncompress)(void *, void *, int, int, int *); int (*options)(char **, int); int (*options_post)(int); void *(*dump_options)(int, int *); int (*extract_options)(int, void *, int); int (*check_options)(int, void *, int); void (*display_options)(void *, int); void (*usage)(); }; extern struct compressor *lookup_compressor(char *); extern struct compressor *lookup_compressor_id(int); extern void display_compressors(char *, char *); extern void display_compressor_usage(char *); static inline int compressor_init(struct compressor *comp, void **stream, int block_size, int datablock) { if(comp->init == NULL) return 0; return comp->init(stream, block_size, datablock); } static inline int compressor_compress(struct compressor *comp, void *strm, void *dest, void *src, int size, int block_size, int *error) { return comp->compress(strm, dest, src, size, block_size, error); } static inline int compressor_uncompress(struct compressor *comp, void *dest, void *src, int size, int block_size, int *error) { return comp->uncompress(dest, src, size, block_size, error); } /* * For the following functions please see the lzo, lz4 or xz * compressors for commented examples of how they are used. */ static inline int compressor_options(struct compressor *comp, char *argv[], int argc) { if(comp->options == NULL) return -1; return comp->options(argv, argc); } static inline int compressor_options_post(struct compressor *comp, int block_size) { if(comp->options_post == NULL) return 0; return comp->options_post(block_size); } static inline void *compressor_dump_options(struct compressor *comp, int block_size, int *size) { if(comp->dump_options == NULL) return NULL; return comp->dump_options(block_size, size); } static inline int compressor_extract_options(struct compressor *comp, int block_size, void *buffer, int size) { if(comp->extract_options == NULL) return size ? -1 : 0; return comp->extract_options(block_size, buffer, size); } static inline int compressor_check_options(struct compressor *comp, int block_size, void *buffer, int size) { if(comp->check_options == NULL) return 0; return comp->check_options(block_size, buffer, size); } static inline void compressor_display_options(struct compressor *comp, void *buffer, int size) { if(comp->display_options != NULL) comp->display_options(buffer, size); } #endif squashfs4.3/squashfs-tools/sort.c0000644000175000017500000002245412333330365017162 0ustar phillipphillip/* * Create a squashfs filesystem. This is a highly compressed read only * filesystem. * * Copyright (c) 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2012, * 2013, 2014 * Phillip Lougher * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License * as published by the Free Software Foundation; either version 2, * or (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. * * sort.c */ #define TRUE 1 #define FALSE 0 #define MAX_LINE 16384 #include #include #include #include #include #include #include #include #include #include #include "squashfs_fs.h" #include "mksquashfs.h" #include "sort.h" #include "error.h" #include "progressbar.h" int mkisofs_style = -1; struct sort_info { dev_t st_dev; ino_t st_ino; int priority; struct sort_info *next; }; struct sort_info *sort_info_list[65536]; struct priority_entry *priority_list[65536]; extern int silent; extern void write_file(squashfs_inode *inode, struct dir_ent *dir_ent, int *c_size); extern char *pathname(struct dir_ent *dir_ent); void add_priority_list(struct dir_ent *dir, int priority) { struct priority_entry *new_priority_entry; priority += 32768; new_priority_entry = malloc(sizeof(struct priority_entry)); if(new_priority_entry == NULL) MEM_ERROR(); new_priority_entry->dir = dir;; new_priority_entry->next = priority_list[priority]; priority_list[priority] = new_priority_entry; } int get_priority(char *filename, struct stat *buf, int priority) { int hash = buf->st_ino & 0xffff; struct sort_info *s; for(s = sort_info_list[hash]; s; s = s->next) if((s->st_dev == buf->st_dev) && (s->st_ino == buf->st_ino)) { TRACE("returning priority %d (%s)\n", s->priority, filename); return s->priority; } TRACE("returning priority %d (%s)\n", priority, filename); return priority; } #define ADD_ENTRY(buf, priority) {\ int hash = buf.st_ino & 0xffff;\ struct sort_info *s;\ if((s = malloc(sizeof(struct sort_info))) == NULL) \ MEM_ERROR(); \ s->st_dev = buf.st_dev;\ s->st_ino = buf.st_ino;\ s->priority = priority;\ s->next = sort_info_list[hash];\ sort_info_list[hash] = s;\ } int add_sort_list(char *path, int priority, int source, char *source_path[]) { int i, n; struct stat buf; TRACE("add_sort_list: filename %s, priority %d\n", path, priority); if(strlen(path) > 1 && strcmp(path + strlen(path) - 2, "/*") == 0) path[strlen(path) - 2] = '\0'; TRACE("add_sort_list: filename %s, priority %d\n", path, priority); re_read: if(path[0] == '/' || strncmp(path, "./", 2) == 0 || strncmp(path, "../", 3) == 0 || mkisofs_style == 1) { if(lstat(path, &buf) == -1) goto error; TRACE("adding filename %s, priority %d, st_dev %d, st_ino " "%lld\n", path, priority, (int) buf.st_dev, (long long) buf.st_ino); ADD_ENTRY(buf, priority); return TRUE; } for(i = 0, n = 0; i < source; i++) { char *filename; int res = asprintf(&filename, "%s/%s", source_path[i], path); if(res == -1) BAD_ERROR("asprintf failed in add_sort_list\n"); res = lstat(filename, &buf); free(filename); if(res == -1) { if(!(errno == ENOENT || errno == ENOTDIR)) goto error; continue; } ADD_ENTRY(buf, priority); n ++; } if(n == 0 && mkisofs_style == -1 && lstat(path, &buf) != -1) { ERROR("WARNING: Mkisofs style sortlist detected! This is " "supported but please\n"); ERROR("convert to mksquashfs style sortlist! A sortlist entry"); ERROR(" should be\neither absolute (starting with "); ERROR("'/') start with './' or '../' (taken to be\nrelative to " "$PWD), otherwise it "); ERROR("is assumed the entry is relative to one\nof the source " "directories, i.e. with "); ERROR("\"mksquashfs test test.sqsh\",\nthe sortlist "); ERROR("entry \"file\" is assumed to be inside the directory " "test.\n\n"); mkisofs_style = 1; goto re_read; } mkisofs_style = 0; if(n == 1) return TRUE; if(n > 1) { ERROR(" Ambiguous sortlist entry \"%s\"\n\nIt maps to more " "than one source entry! Please use an absolute path." "\n", path); return FALSE; } error: ERROR_START("Cannot stat sortlist entry \"%s\"\n", path); ERROR("This is probably because you're using the wrong file\n"); ERROR("path relative to the source directories."); ERROR_EXIT(" Ignoring"); /* * Historical note * Failure to stat a sortlist entry is deliberately ignored, even * though it is an error. Squashfs release 2.2 changed the behaviour * to treat it as a fatal error, but it was changed back to * the original behaviour to ignore it in release 2.2-r2 following * feedback from users at the time. */ return TRUE; } void generate_file_priorities(struct dir_info *dir, int priority, struct stat *buf) { struct dir_ent *dir_ent = dir->list; priority = get_priority(dir->pathname, buf, priority); for(; dir_ent; dir_ent = dir_ent->next) { struct stat *buf = &dir_ent->inode->buf; if(dir_ent->inode->root_entry) continue; switch(buf->st_mode & S_IFMT) { case S_IFREG: add_priority_list(dir_ent, get_priority(pathname(dir_ent), buf, priority)); break; case S_IFDIR: generate_file_priorities(dir_ent->dir, priority, buf); break; } } } int read_sort_file(char *filename, int source, char *source_path[]) { FILE *fd; char line_buffer[MAX_LINE + 1]; /* overflow safe */ char sort_filename[MAX_LINE + 1]; /* overflow safe */ char *line, *name; int n, priority, res; if((fd = fopen(filename, "r")) == NULL) { ERROR("Failed to open sort file \"%s\" because %s\n", filename, strerror(errno)); return FALSE; } while(fgets(line = line_buffer, MAX_LINE + 1, fd) != NULL) { int len = strlen(line); if(len == MAX_LINE && line[len - 1] != '\n') { /* line too large */ ERROR("Line too long when reading " "sort file \"%s\", larger than %d " "bytes\n", filename, MAX_LINE); goto failed; } /* * Remove '\n' terminator if it exists (the last line * in the file may not be '\n' terminated) */ if(len && line[len - 1] == '\n') line[len - 1] = '\0'; /* Skip any leading whitespace */ while(isspace(*line)) line ++; /* if comment line, skip */ if(*line == '#') continue; /* * Scan for filename, don't use sscanf() and "%s" because * that can't handle filenames with spaces */ for(name = sort_filename; !isspace(*line) && *line != '\0';) { if(*line == '\\') { line ++; if (*line == '\0') break; } *name ++ = *line ++; } *name = '\0'; /* * if filename empty, then line was empty of anything but * whitespace or a backslash character. Skip empy lines */ if(sort_filename[0] == '\0') continue; /* * Scan the rest of the line, we expect a decimal number * which is the filename priority */ errno = 0; res = sscanf(line, "%d%n", &priority, &n); if((res < 1 || errno) && errno != ERANGE) { if(errno == 0) /* No error, assume EOL or match failure */ ERROR("Sort file \"%s\", can't find priority " "in entry \"%s\", EOL or match " "failure\n", filename, line_buffer); else /* Some other failure not ERANGE */ ERROR("Sscanf failed reading sort file \"%s\" " "because %s\n", filename, strerror(errno)); goto failed; } else if((errno == ERANGE) || (priority < -32768 || priority > 32767)) { ERROR("Sort file \"%s\", entry \"%s\" has priority " "outside range of -32767:32768.\n", filename, line_buffer); goto failed; } /* Skip any trailing whitespace */ line += n; while(isspace(*line)) line ++; if(*line != '\0') { ERROR("Sort file \"%s\", trailing characters after " "priority in entry \"%s\"\n", filename, line_buffer); goto failed; } res = add_sort_list(sort_filename, priority, source, source_path); if(res == FALSE) goto failed; } if(ferror(fd)) { ERROR("Reading sort file \"%s\" failed because %s\n", filename, strerror(errno)); goto failed; } fclose(fd); return TRUE; failed: fclose(fd); return FALSE; } void sort_files_and_write(struct dir_info *dir) { int i; struct priority_entry *entry; squashfs_inode inode; int duplicate_file; for(i = 65535; i >= 0; i--) for(entry = priority_list[i]; entry; entry = entry->next) { TRACE("%d: %s\n", i - 32768, pathname(entry->dir)); if(entry->dir->inode->inode == SQUASHFS_INVALID_BLK) { write_file(&inode, entry->dir, &duplicate_file); INFO("file %s, uncompressed size %lld bytes %s" "\n", pathname(entry->dir), (long long) entry->dir->inode->buf.st_size, duplicate_file ? "DUPLICATE" : ""); entry->dir->inode->inode = inode; entry->dir->inode->type = SQUASHFS_FILE_TYPE; } else INFO("file %s, uncompressed size %lld bytes " "LINK\n", pathname(entry->dir), (long long) entry->dir->inode->buf.st_size); } } squashfs4.3/CHANGES0000644000175000017500000006527312334230024014025 0ustar phillipphillip SQUASHFS CHANGE LOG 4.3 12 MAY 2014 New compressor options, new Mksquashfs/Unsquashfs functionality, duplicate checking optimisations, stability improvements (option/file parsing, buffer/memory overflow checks, filesystem hardening on corrupted filesystems), CVE fixes. Too many changes to do the traditional custom changelog. But, this is now unnecessary, so instead list most significant 15% of commits from git changelog in chronological order. - unsquashfs: add checks for corrupted data in opendir functions - unsquashfs: completely empty filesystems incorrectly generate an error - unsquashfs: fix open file limit - mksquashfs: Use linked list to store directory entries rather - mksquashfs: Remove qsort and add a bottom up linked list merge sort - mksquashfs: optimise lookup_inode2() for dirs - pseudo: fix handling of modify pseudo files - pseudo: fix handling of directory pseudo files - xattr: Fix ERROR() so that it is synchronised with the progress bar - mksquashfs/sort: Fix INFO() so that it is synced with the progress bar - mksquashfs: Add -progress to force progress bar when using -info - error.h: consolidate the various error macros into one header file - mksquashfs: fix stack overflow in write_fragment_table() - mksquashfs: move list allocation from off the stack - unsquashfs: fix oversight in directory permission setting - mksquashfs: dynamically allocate recovery_file - mksquashfs: dynamically allocate buffer in subpathname() - mksquashfs: dynamically allocate buffer in pathname() - unsquashfs: fix CVE-2012-4024 - unsquashfs: fix CVE-2012-4025 - mksquashfs: fix potential stack overflow in get_component() - mksquashfs: add parse_number() helper for numeric command line options - mksquasfs: check return value of fstat() in reader_read_file() - mksquashfs: dynamically allocate filename in old_add_exclude() - unsquashfs: dynamically allocate pathname in dir_scan() - unsquashfs: dynamically allocate pathname in pre_scan() - sort: dynamically allocate filename in add_sort_list() - mksquashfs: fix dir_scan() exit if lstat of source directory fails - pseudo: fix memory leak in read_pseudo_def() if exec_file() fails - pseudo: dynamically allocate path in dump_pseudo() - mksquashfs: dynamically allocate path in display_path2() - mksquashfs: dynamically allocate b_buffer in getbase() - pseudo: fix potential stack overflow in get_component() - pseudo: avoid buffer overflow in read_pseudo_def() using sscanf() - pseudo: dynamically allocate filename in exec_file() - pseudo: avoid buffer overflow in read_sort_file() using fscanf() - sort: tighten up sort file parsing - unsquashfs: fix name under-allocation in process_extract_files() - unsquashfs: avoid buffer overflow in print_filename() using sprintf() - Fix some limits in the file parsing routines - pseudo: Rewrite pseudo file processing - read_fs: fix small memory leaks in read_filesystem() - mksquashfs: fix fclose leak in reader_read_file() on I/O error - mksquashfs: fix frag struct leak in write_file_{process|blocks|frag} - unsquashfs_xattr: fix memory leak in write_xattr() - read_xattrs: fix xattr free in get_xattr() in error path - unsquashfs: add -user-xattrs option to only extract user.xxx xattrs - unsquashfs: add code to only print "not superuser" error message once - unsquashfs: check for integer overflow in user input - mksquashfs: check for integer overflow in user input - mksquashfs: fix "new" variable leak in dir_scan1() - read_fs: prevent buffer {over|under}flow in read_block() with corrupted filesystems - read_fs: check metadata blocks are expected size in scan_inode_table() - read_fs: check the root inode block is found in scan_inode_table() - read_fs: Further harden scan_inode_table() against corrupted filesystems - unsquashfs: prevent buffer {over|under}flow in read_block() with corrupted filesystems - read_xattrs: harden xattr data reading against corrupted filesystems - unsquash-[23]: harden frag table reading against corrupted filesystems - unsquash-4.c: harden uid/gid & frag table reading against corruption - unsquashfs: harden inode/directory table reading against corruption - mksquashfs: improve out of space in output filesystem handling - mksquashfs: flag lseek error in writer as probable out of space - mksquashfs: flag lseek error in write_destination as probable out of space - mksquashfs: print file being squashed when ^\ (SIGQUIT) typed - mksquashfs: make EXIT_MKSQUASHFS() etc restore via new restore thread - mksquashfs: fix recursive restore failure check - info: dump queue and cache status if ^\ hit twice within one second - mksquashfs: fix rare race condition in "locked fragment" queueing - lz4: add experimental support for lz4 compression - lz4: add support for lz4 "high compression" - lzo_wrapper: new implementation with compression options - gzip_wrapper: add compression options - mksquashfs: redo -comp parsing - mksquashfs: display compressor options when -X option isn't recognised - mksquashfs: add -Xhelp option - mksquashfs/unsquashfs: fix mtime signedness - Mksquashfs: optimise duplicate checking when appending - Mksquashfs: introduce additional per CPU fragment process threads - Mksquashfs: significantly optimise fragment duplicate checking - read_fs: scan_inode_table(), fix memory leak on filesystem corruption - pseudo: add_pseudo(), fix use of freed variable - mksquashfs/unsquashfs: exclude/extract/pseudo files, fix handling of leaf name - mksquashfs: rewrite default queue size so it's based on physical mem - mksquashfs: add a new -mem option - mksquashfs: fix limit on the number of dynamic pseudo files - mksquashfs: make -mem take a normal byte value, optionally with a K, M or G 4.2 28 FEB 2011 XZ compression, and compression options support 1. Filesystem improvements: 1.1 Added XZ compression 1.2 Added compression options support 2. Miscellaneous improvements/bug fixes 1.1 Add missing NO_XATTR filesystem flag to indicate no-xattrs option was specified and no xattrs should be stored when appending. 1.2 Add suppport in Unquashfs -stat option for displaying NO_XATTR flag. 1.3 Remove checkdata entry from Unsquashfs -stat option if a 4.0 filesystem - checkdata is no longer supported. 1.4 Fix appending bug when appending to an empty filesystem - this would be incorrectly treated as an error. 1.5 Use glibc sys/xattr.h include rather than using attr/xattr.h which isn't present by default on some distributions. 1.6 Unsquashfs, fix block calculation error with regular files when file size is between 2^32-block_size+1 and 2^32-1. 1.7 Unsquashfs, fix sparse file writing when holes are larger than 2^31-1. 1.8 Add external CFLAGS and LDFLAGS support to Makefile, and allow build options to be specified on command line. Also don't over-write passed in CFLAGS definition. 4.1 19 SEPT 2010 Major filesystem and tools improvements 1. Filesystem improvements: 1.1 Extended attribute support 1.2 New compression framework 1.3 Support for LZO compression 1.4 Support for LZMA compression (not yet in mainline) 2. Mksquashfs improvements: 1.1 Enhanced pseudo file support 1.2 New options for choosing compression algorithm used 1.3 New options for controlling extended attributes 1.4 Fix misalignment issues with memcpy etc. seen on ARM 1.5 Fix floating point error in progress_bar when max == 0 1.6 Removed use of get_nproc() call unavailable in ulibc 1.7 Reorganised help text 3. Unsquashfs improvements: 1.1 New options for controlling extended attributes 1.2 Fix misalignment issues with memcpy etc. seen on ARM 1.3 Fix floating point error in progress_bar when max == 0 1.4 Removed use of get_nproc() call unavailable in ulibc 4.0 5 APR 2009 Major filesystems improvements 1. Kernel code improvements: 1.1 Fixed little endian layout adopted. All swapping macros removed, and in-line swapping added for big-endian architectures. 1.2 Kernel code substantially improved and restructured. 1.3 Kernel code split into separate files along functional lines. 1.4 Vmalloc usage removed, and code changed to use separately allocated 4K buffers 2. Unsquashfs improvements: 2.1 Support for 4.0 filesystems added. 2.2 Swapping macros rewritten. 2.3 Unsquashfs code restructured and split into separate files. 3. Mksquashfs improvements: 3.1 Swapping macros rewritten. Fixed little-endian layout allows code to be optimised and only added at compile time for big endian systems. 3.2 Support for pseudo files added. 3.4 26 AUG 2008 Performance improvements to Unsquashfs, Mksquashfs and the kernel code. Plus many small bug fixes. 1. Kernel code improvements: 1.1 Internal Squashfs kernel metadata and fragment cache implementations have been merged and optimised. Spinlocks are now used, locks are held for smaller periods and wakeups have been minimised. Small race condition fixed where if two or more processes tried to read the same cache block simultaneously they would both read and decompress it. 10-20%+ speed improvement has been seen on tests. 1.2 NFS export code rewritten following VFS changes in linux-2.6.24. 1.3 New patches for linux-2.6.25, linux-2.6.26, and linux-2.6.27. Fixed patch for linux-2.6.24. 1.4 Fixed small buffer_head leak in squashfs_read_data when handling badly corrupted filesystems. 1.5 Fixed bug in get_dir_index_using_offset. 2. Unsquashfs improvements: 2.1 Unsquashfs has been parallelised. Filesystem reading, writing and decompression is now multi-threaded. Up to 40% speed improvement seen on tests. 2.2 Unsquashfs now has a progress bar. Use -no-progress to disable it. 2.3 Fixed small bug where unistd.h wasn't being included on some distributions, leading to lseek being used rather than lseek64 - which meant on these distributions Unsquashfs couldn't unsquash filesystems larger than 4GB. 3. Mksquashfs improvements: 3.1 Removed some small remaining parallelisation bottlenecks. Depending on source filesystem, up to 10%+ speed improvement. 3.2 Progress bar improved, and moved to separate thread. 3.3 Sparse file handling bug in Mksquashfs 3.3 fixed. 3.4 Two rare appending restore bugs fixed (when ^C hit twice). 3.3 1 NOV 2007 Increase in block size, sparse file support, Mksquashfs and Unsquashfs extended to use pattern matching in exclude/extract files, plus many more improvements and bug fixes. 1. Filesystem improvements: 1.1. Maximum block size has been increased to 1Mbyte, and the default block size has been increased to 128 Kbytes. This improves compression. 1.2. Sparse files are now supported. Sparse files are files which have large areas of unallocated data commonly called holes. These files are now detected by Squashfs and stored more efficiently. This improves compression and read performance for sparse files. 2. Mksquashfs improvements: 2.1. Exclude files have been extended to use wildcard pattern matching and regular expressions. Support has also been added for non-anchored excludes, which means it is now possible to specify excludes which match anywhere in the filesystem (i.e. leaf files), rather than always having to specify exclude files starting from the root directory (anchored excludes). 2.2. Recovery files are now created when appending to existing Squashfs filesystems. This allows the original filesystem to be recovered if Mksquashfs aborts unexpectedly (i.e. power failure). 3. Unsquashfs improvements: 3.1. Multiple extract files can now be specified on the command line, and the files/directories to be extracted can now also be given in a file. 3.2. Extract files have been extended to use wildcard pattern matching and regular expressions. 3.3. Filename printing has been enhanced and Unquashfs can now display filenames with file attributes ('ls -l' style output). 3.4. A -stat option has been added which displays the filesystem superblock information. 3.5. Unsquashfs now supports 1.x filesystems. 4. Miscellaneous improvements/bug fixes: 4.1. Squashfs kernel code improved to use SetPageError in squashfs_readpage() if I/O error occurs. 4.2. Fixed Squashfs kernel code bug preventing file seeking beyond 2GB. 4.3. Mksquashfs now detects file size changes between first phase directory scan and second phase filesystem create. It also deals better with file I/O errors. 3.2-r2 15 JAN 2007 Kernel patch update and progress bar bug fix 1. Kernel patches 2.6.19/2.6.20 have been updated to use const structures and mutexes rather than older semaphores. 2. Minor SMP bug fixes. 3. Progress bar broken on x86-64. Fixed. 3.2 2 JAN 2007 NFS support, improvements to the Squashfs-tools, major bug fixes, lots of small improvements/bug fixes, and new kernel patches. Improvements: 1. Squashfs filesystems can now be exported via NFS. 2. Unsquashfs now supports 2.x filesystems. 3. Mksquashfs now displays a progress bar. 4. Squashfs kernel code has been hardened against accidently or maliciously corrupted Squashfs filesystems. Bug fixes: 5. Race condition occurring on S390 in readpage() fixed. 6. Odd behaviour of MIPS memcpy in read_data() routine worked-around. 7. Missing cache_flush in Squashfs symlink_readpage() added. 3.1-r2 30 AUG 2006 Mksquashfs -sort bug fix A code optimisation after testing unfortunately broke sorting in Mksquashfs. This has been fixed. 3.1 19 AUG 2006 This release has some major improvements to the squashfs-tools, a couple of major bug fixes, lots of small improvements/bug fixes, and new kernel patches. 1. Mksquashfs has been rewritten to be multi-threaded. It has the following improvements 1.1. Parallel compression. By default as many compression and fragment compression threads are created as there are available processors. This significantly speeds up performance on SMP systems. 1.2. File input and filesystem output is peformed in parallel on separate threads to maximise I/O performance. Even on single processor systems this speeds up performance by at least 10%. 1.3. Appending has been significantly improved, and files within the filesystem being appended to are no longer scanned and checksummed. This significantly improves append time for large filesystems. 1.4. File duplicate checking has been optimised, and split into two separate phases. Only files which are considered possible duplicates after the first phase are checksummed and cached in memory. 1.5 The use of swap memory was found to significantly impact performance. The amount of memory used to cache files is now a command line option, by default this is 512 Mbytes. 2. Unsquashfs has the following improvements 2.1 Unsquashfs now allows you to specify the filename or the directory within the Squashfs filesystem that is to be extracted, rather than always extracting the entire filesystem. 2.2 A new -force option has been added which forces Unsquashfs to output to the destination directory even if files and directories already exist in the destination directory. This allows you to update an already existing directory tree, or to Unsquashfs to a partially filled directory tree. Without the -force option Unsquashfs will refuse to output. 3. The following major bug fixes have been made 3.1 A fragment table rounding bug has been fixed in Mksquashfs. Previously if the number of fragments in the filesystem were a multiple of 512, Mksquashfs would generate an incorrect filesystem. 3.2 A rare SMP bug which occurred when simultaneously acccessing multiply mounted Squashfs filesystems has been fixed. 4. Miscellaneous improvements/bug fixes 4.1 Kernel code stack usage has been reduced. This is to ensure Squashfs works with 4K stacks. 4.2 Readdir (Squashfs kernel code) has been fixed to always return 0, rather than the number of directories read. Squashfs should now interact better with NFS. 4.3 Lseek bug in Mksquashfs when appending to larger than 4GB filesystems fixed. 4.4 Squashfs 2.x initrds can now been mounted. 4.5 Unsquashfs exit status fixed. 4.6 New patches for linux-2.6.18 and linux-2.4.33. 3.0 15 MAR 2006 Major filesystem improvements 1. Filesystems are no longer limited to 4 GB. In theory 2^64 or 4 exabytes is now supported. 2. Files are no longer limited to 4 GB. In theory the maximum file size is 4 exabytes. 3. Metadata (inode table and directory tables) are no longer restricted to 16 Mbytes. 4. Hardlinks are now suppported. 5. Nlink counts are now supported. 6. Readdir now returns '.' and '..' entries. 7. Special support for files larger than 256 MB has been added to the Squashfs kernel code for faster read access. 8. Inode numbers are now stored within the inode rather than being computed from inode location on disk (this is not so much an improvement, but a change forced by the previously listed improvements). 2.2-r2 8 SEPT 2005 Second release of 2.2, this release fixes a couple of small bugs, a couple of small documentation mistakes, and adds a patch for kernel 2.6.13. 1. Mksquashfs now deletes the output filesystem image file if an error occurs whilst generating the filesystem. Previously on error the image file was left empty or partially written. 2. Updated mksquashfs so that it doesn't allow you to generate filesystems with block sizes smaller than 4K. Squashfs hasn't supported block sizes less than 4K since 2.0-alpha. 3. Mksquashfs now ignores missing files/directories in sort files. This was the original behaviour before 2.2. 4. Fixed small mistake in fs/Kconfig where the version was still listed as 2.0. 5. Updated ACKNOWLEDGEMENTS file. 2.2 3 JUL 2005 This release has some small improvements, bug fixes and patches for new kernels. 1. Sort routine re-worked and debugged from release 2.1. It now allows you to give Mkisofs style sort files and checks for filenames that don't match anything. Sort priority has also been changed to conform to Mkisofs usage, highest priority files are now placed at the start of the filesystem (this means they will be on the inside of a CD or DVD). 2. New Configure options for embedded systems (memory constrained systems). See INSTALL file for further details. 3. Directory index bug fixed where chars were treated as signed on some architectures. A file would not be found in the rare case that the filename started with a chracter greater than 127. 4. Bug introduced into the read_data() routine when sped up to use data block queueing fixed. If the second or later block resulted in an I/O error this was not checked. 5. Append bug introduced in 2.1 fixed. The code to compute the new compressed and uncompressed directory parts after appending was wrong. 6. Metadata block length read routine altered to not perform a misaligned short read. This was to fix reading on an ARM7 running uCLinux without a misaligned read interrupt handler. 7. Checkdata bug introduced in 2.1 fixed. 2.1-r2 15 DEC 2004 Code changed so it can be compiled with gcc 2.x 1. In some of the code added for release 2.1 I unknowingly used some gcc extensions only supported by 3.x compilers. I have received a couple of reports that the 2.1 release doesn't build on 2.x and so people are clearly still using gcc 2.x. The code has been rewritten to remove these extensions. 2.1 10 DEC 2004 Significantly improved directory handling plus numerous other smaller improvements 1. Fast indexed directories implemented. These speed up directory operations (ls, file lookup etc.) significantly for directories larger than 8 KB. 2. All directories are now sorted in alphabetical order. This again speeds up directory operations, and in some cases it also results in a small compression improvement (greater data similarity between files with alphabetically similar names). 3. Maximum directory size increased from 512 KB to 128 MB. 4. Duplicate fragment checking and appending optimised in mksquashfs, depending on filesystem, this is now up to 25% faster. 5. Mksquashfs help information reformatted and reorganised. 6. The Squashfs version and release date is now printed at kernel boot-time or module insertion. This addition will hopefully help to reduce the growing problem where the Squashfs version supported by a kernel is unknown and the kernel source is unavailable. 7. New PERFORMANCE.README file. 8. New -2.0 mksquashfs option. 9. CHANGES file reorganised. 10. README file reorganised, clarified and updated to include the 2.0 mksquashfs options. 11. New patch for Linux 2.6.9. 12. New patch for Linux 2.4.28. 2.0r2 29 AUG 2004 Workaround for kernel bug in kernels 2.6.8 and newer added 1. New patch for kernel 2.6.8.1. This includes a workaround for a kernel bug introduced in 2.6.7bk14, which is present in all later versions of the kernel. If you're using a 2.6.8 kernel or later then you must use this 2.6.8.1 patch. If you've experienced hangs or oopses using Squashfs with a 2.6.8 or later kernel then you've hit this bug, and this patch will fix it. It is worth mentioning that this kernel bug potentially affects other filesystems. If you receive odd results with other filesystems you may be experiencing this bug with that filesystem. I submitted a patch but this has not yet gone into the kernel, hopefully the bug will be fixed in later kernels. 2.0 13 JULY 2004 A couple of new options, and some bug fixes 1. New mksquashfs -all-root, -root-owned, -force-uid, and -force-gid options. These allow the uids/gids of files in the generated filesystem to be specified, overriding the uids/gids in the source filesystem. 2. Initrds are now supported for kernels 2.6.x. 3. amd64 bug fixes. If you use an amd64, please read the README-AMD64 file. 4. Check-data and gid bug fixes. With 2.0-alpha when mounting 1.x filesystems in certain cases file gids were corrupted. 5. New patch for Linux 2.6.7. 2.0-ALPHA 21 MAY 2004 Filesystem changes and compression improvements 1. Squashfs 2.0 has added the concept of fragment blocks. Files smaller than the file block size and optionally the remainder of files that do not fit fully into a block (i.e. the last 32K in a 96K file) are packed into shared fragments and compressed together. This achieves on average 5 - 20% better compression than Squashfs 1.x. 2. The maximum block size has been increased to 64K (in the ALPHA version of Squashfs 2.0). 3. The maximum number of UIDs has been increased to 256 (from 48 in 1.x). 4. The maximum number of GIDs has been increased to 256 (from 15 in 1.x). 5. Removal of sleep_on() function call in 2.6.x patch, to allow Squashfs to work on the Fedora rc2 kernel. 6. Numerous small bug fixes have been made. 1.3r3 18 JAN 2004 Third release of 1.3, this adds a new mksquashfs option, some bug fixes, and extra patches for new kernels 1. New mksquashfs -ef exclude option. This option reads the exclude dirs/files from an exclude file, one exclude dir/file per line. This avoids the command line size limit when using the -e exclude option, 2. When appending to existing filesystems, if mksquashfs experiences a fatal error (e.g. out of space when adding to the destination), the original filesystem is restored, 3. Mksquashfs now builds standalone, without the kernel needing to be patched. 4. Bug fix in the kernel squashfs filesystem, where the pages being filled were not kmapped. This seems to only have caused problems on an Apple G5, 5. New patch for Linux 2.4.24, 6. New patch for Linux 2.6.1, this replaces the patch for 2.6.0-test7. 1.3r2 14 OCT 2003 Second release of 1.3, bug fixes and extra patches for new kernels 1. Bug fix in routine that adds files to the filesystem being generated in mksquashfs. This bug was introduced in 1.3 (not enough testing...) when I rewrote it to handle files larger than available memory. This bug caused a SEGV, so if you've ever got that, it is now fixed, 2. Long running bug where ls -s and du reported wrong block size fixed. I'm pretty sure this used to work many kernel versions ago (2.4.7) but it broke somewhere along the line since then, 3. New patch for Linux 2.4.22, 4. New patch for 2.6.0-test7, this replaces the patch for 2.6.0-test1. 1.3 29 JUL 2003 FIFO/Socket support added plus optimisations and improvements 1. FIFOs and Socket inodes are now supported, 2. Mksquashfs can now compress files larger than available memory, 3. File duplicate check routine optimised, 4. Exit codes fixed in Mksquashfs, 5. Patch for Linux 2.4.21, 6. Patch for Linux 2.6.0-test1. Hopefully, this will work for the next few releases of 2.6.0-testx, otherwise, I'll be releasing a lot of updates to the 2.6.0 patch... 1.2 13 MAR 2003 Append feature and new mksquashfs options added Mksquashfs can now add to existing squashfs filesystems. Three extra options "-noappend", "-keep-as-directory", and "root-becomes" have been added. The append option with file duplicate detection, means squashfs can be used as a simple versioning archiving filesystem. A squashfs filesystem can be created with for example the linux-2.4.19 source. Appending the linux-2.4.20 source will create a filesystem with the two source trees, but only the changed files will take extra room, the unchanged files will be detected as duplicates. See the README file for usage changes. 1.1b 16 JAN 2003 Bug fix release Fixed readpage deadlock bug. This was a rare deadlock bug that happened when pushing pages into the page cache when using greater than 4K blocks. I never got this bug when I tested the filesystem, but two people emailed me on the same day about the problem! I fixed it by using a page cache function that wasn't there when I originally did the work, which was nice :-) 1.1 8 JAN 2003 Added features 1. Kernel squashfs can now mount different byte order filesystems. 2. Additional features added to mksquashfs. Mksquashfs now supports exclude files and multiple source files/directories can be specified. A nopad option has also been added, which informs mksquashfs not to pad filesystems to a multiple of 4K. See README for mksquashfs usage changes. 3. Greater than 2GB filesystems bug fix. Filesystems greater than 2GB can now be created. 1.0c 14 NOV 2002 Bug fix release Fixed bugs with initrds and device nodes 1.0 23 OCT 2002 Initial release