com.android.tools.build.apkzlib.zip

Class ZFile

java.lang.Object

com.android.tools.build.apkzlib.zip.ZFile

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable

public class ZFile
extends java.lang.Object
implements java.io.Closeable

The ZFile provides the main interface for interacting with zip files. A ZFile can be created on a new file or in an existing file. Once created, files can be added or removed from the zip file.

Changes in the zip file are always deferred. Any change requested is made in memory and written to disk only when update() or close() is invoked.

Zip files are open initially in read-only mode and will switch to read-write when needed. This is done automatically. Because modifications to the file are done in-memory, the zip file can be manipulated when closed. When invoking update() or close() the zip file will be reopen and changes will be written. However, the zip file cannot be modified outside the control of ZFile. So, if a ZFile is closed, modified outside and then a file is added or removed from the zip file, when reopening the zip file, ZFile will detect the outside modification and will fail.

In memory manipulation means that files added to the zip file are kept in memory until written to disk. This provides much faster operation and allows better zip file allocation (see below). It may, however, increase the memory footprint of the application. When adding large files, if memory consumption is a concern, a call to update() will actually write the file to disk and discard the memory buffer. Information about allocation can be obtained from a ByteTracker that can be given to the file on creation.

ZFile keeps track of allocation inside of the zip file. If a file is deleted, its space is marked as freed and will be reused for an added file if it fits in the space. Allocation of files to empty areas is done using a best fit algorithm. When adding a file, if it doesn't fit in any free area, the zip file will be extended.

ZFile provides a fast way to merge data from another zip file (see mergeFrom(ZFile, Predicate)) avoiding recompression and copying of equal files. When merging, patterns of files may be provided that are ignored. This allows handling special files in the merging process, such as files in META-INF.

When adding files to the zip file, unless files are explicitly required to be stored, files will be deflated. However, deflating will not occur if the deflated file is larger then the stored file, e.g. if compression would yield a bigger file. See Compressor for details on how compression works.

Because ZFile was designed to be used in a build system and not as general-purpose zip utility, it is very strict (and unforgiving) about the zip format and unsupported features.

ZFile supports alignment. Alignment means that file data (not entries -- the local header must be discounted) must start at offsets that are multiple of a number -- the alignment. Alignment is defined by an alignment rules (AlignmentRule in the ZFileOptions object used to create the ZFile.

When a file is added to the zip, the alignment rules will be checked and alignment will be honored when positioning the file in the zip. This means that unused spaces in the zip may be generated as a result. However, alignment of existing entries will not be changed.

Entries can be realigned individually (see StoredEntry.realign() or the full zip file may be realigned (see realign()). When realigning the full zip entries that are already aligned will not be affected.

Because realignment may cause files to move in the zip, realignment is done in-memory meaning that files that need to change location will moved to memory and will only be flushed when either update() or close() are called.

Alignment only applies to filed that are forced to be uncompressed. This is because alignment is used to allow mapping files in the archive directly into memory and compressing defeats the purpose of alignment.

Manipulating zip files with ZFile may yield zip files with empty spaces between files. This happens in two situations: (1) if alignment is required, files may be shifted to conform to the request alignment leaving an empty space before the previous file, and (2) if a file is removed or replaced with a file that does not fit the space it was in. By default, ZFile does not do any special processing in these situations. Files are indexed by their offsets from the central directory and empty spaces can exist in the zip file.

However, it is possible to tell ZFile to use the extra field in the local header to do cover the empty spaces. This is done by setting ZFileOptions.setCoverEmptySpaceUsingExtraField(boolean) to true. This has the advantage of leaving no gaps between entries in the zip, as required by some tools like Oracle's {code jar} tool. However, setting this option will destroy the contents of the file's extra field.

Activating ZFileOptions.setCoverEmptySpaceUsingExtraField(boolean) may lead to virtual files being added to the zip file. Since extra field is limited to 64k, it is not possible to cover any space bigger than that using the extra field. In those cases, virtual files are added to the file. A virtual file is a file that exists in the actual zip data, but is not referenced from the central directory. A zip-compliant utility should ignore these files. However, zip utilities that expect the zip to be a stream, such as Oracle's jar, will find these files instead of considering the zip to be corrupt.

ZFile support sorting zip files. Sorting (done through the sortZipContents() method) is a process by which all files are re-read into memory, if not already in memory, removed from the zip and re-added in alphabetical order, respecting alignment rules. So, in general, file b will come after file a unless file a is subject to alignment that forces an empty space before that can be occupied by b. Sorting can be used to minimize the changes between two zips.

Sorting in ZFile can be done manually or automatically. Manual sorting is done by invoking sortZipContents(). Automatic sorting is done by setting the ZFileOptions.getAutoSortFiles() option when creating the ZFile. Automatic sorting invokes sortZipContents() immediately when doing an update() after all extensions have processed the ZFileExtension.beforeUpdate(). This has the guarantee that files added by extensions will be sorted, something that does not happen if the invocation is sequential, i.e., sortZipContents() called before update(). The drawback of automatic sorting is that sorting will happen every time update() is called and the file is dirty having a possible penalty in performance.

To allow whole-apk signing, the ZFile allows the central directory location to be offset by a fixed amount. This amount can be set using the setExtraDirectoryOffset(long) method. Setting a non-zero value will add extra (unused) space in the zip file before the central directory. This value can be changed at any time and it will force the central directory rewritten when the file is updated or closed.

ZFile provides an extension mechanism to allow objects to register with the file and be notified when changes to the file happen. This should be used to add extra features to the zip file while providing strong decoupling. See ZFileExtension, addZFileExtension(ZFileExtension) and removeZFileExtension(ZFileExtension).

This class is not thread-safe. Neither are any of the classes associated with it in this package, except when otherwise noticed.

Field Summary

Fields

Modifier and Type	Field and Description
static char	SEPARATOR The file separator in paths in the zip file.

Constructor Summary

Constructors

Constructor and Description
ZFile(java.io.File file) Creates a new zip file.
ZFile(java.io.File file, ZFileOptions options) Creates a new zip file.
ZFile(java.io.File file, ZFileOptions options, boolean readOnly) Creates a new zip file.

Method Summary

Modifier and Type	Method and Description
void	add(java.lang.String name, java.io.InputStream stream) Equivalent to call add(String, InputStream, boolean) using true as mayCompress.
void	add(java.lang.String name, java.io.InputStream stream, boolean mayCompress) Adds a file to the archive.
void	addAllRecursively(java.io.File file) Adds all files and directories recursively.
void	addAllRecursively(java.io.File file, java.util.function.Function<? super java.io.File,java.lang.Boolean> mayCompress) Adds all files and directories recursively.
void	addZFileExtension(ZFileExtension extension) Adds an extension to this zip file.
boolean	areTimestampsIgnored() Obtains whether this ZFile is ignoring timestamps.
void	close() Updates the file and closes it.
void	directFullyRead(long offset, byte[] data) Reads exactly data.length bytes of data, failing if it was not possible to read all the requested data.
void	directFullyRead(long offset, java.nio.ByteBuffer dest) Reads exactly dest.remaining() bytes of data, failing if it was not possible to read all the requested data.
java.io.InputStream	directOpen(long start, long end) Opens a portion of the zip for reading.
int	directRead(long offset, byte[] data) Same as directRead(offset, data, 0, data.length).
int	directRead(long offset, byte[] data, int start, int count) Directly reads data from the zip file.
int	directRead(long offset, java.nio.ByteBuffer dest) Directly reads data from the zip file.
long	directSize() Returns the current size (in bytes) of the underlying file.
void	directWrite(long offset, byte[] data) Same as directWrite(offset, data, 0, data.length).
void	directWrite(long offset, byte[] data, int start, int count) Directly writes data in the zip file.
java.util.Set<StoredEntry>	entries() Obtains all entries in the file.
void	finishAllBackgroundTasks() Wait for any background tasks to finish and report any errors.
StoredEntry	get(java.lang.String path) Obtains an entry at a given path in the zip.
byte[]	getCentralDirectoryBytes() Obtains the byte array representation of the central directory.
long	getCentralDirectoryOffset() Obtains the offset at which the central directory exists, or at which it will be written if the zip file were to be flushed immediately.
long	getCentralDirectorySize() Obtains the size of the central directory, if the central directory is written in the zip file.
byte[]	getEocdBytes() Obtains the byte array representation of the EOCD.
byte[]	getEocdComment() Obtains the comment in the EOCD.
long	getEocdOffset() Obtains the offset of the EOCD record, if the EOCD has been written to the file.
long	getEocdSize() Obtains the size of the EOCD record, if the EOCD has been written to the file.
long	getExtraDirectoryOffset() Obtains the extra offset for the central directory.
java.io.File	getFile() Obtains the filesystem path to the zip file.
boolean	hasPendingChangesWithWait() Are there in-memory changes that have not been written to the zip file?
void	mergeFrom(ZFile src, java.util.function.Predicate<java.lang.String> ignoreFilter) Adds all files from another zip file, maintaining their compression.
void	openReadOnly() If the zip file is closed, opens it in read-only mode.
boolean	realign() Realigns all entries in the zip.
void	removeZFileExtension(ZFileExtension extension) Removes an extension from this zip file.
void	setEocdComment(byte[] comment) Sets the comment in the EOCD.
void	setExtraDirectoryOffset(long offset) Sets an extra offset for the central directory.
void	sortZipContents() Sorts all files in the zip.
void	touch() Forcibly marks this zip file as touched, forcing it to be updated when update() or close() are invoked.
void	update() Updates the file writing new entries and removing deleted entries.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SEPARATOR

public static final char SEPARATOR

The file separator in paths in the zip file. This is fixed by the zip specification (section 4.4.17).

See Also:

Constant Field Values

Constructor Detail

ZFile

public ZFile(@Nonnull
             java.io.File file)
      throws java.io.IOException

Creates a new zip file. If the zip file does not exist, then no file is created at this point and ZFile will contain an empty structure. However, an (empty) zip file will be created if either update() or close() are used. If a zip file exists, it will be parsed and read.

Parameters:

file - the zip file

Throws:

java.io.IOException - some file exists but could not be read

ZFile

public ZFile(@Nonnull
             java.io.File file,
             @Nonnull
             ZFileOptions options)
      throws java.io.IOException

Creates a new zip file. If the zip file does not exist, then no file is created at this point and ZFile will contain an empty structure. However, an (empty) zip file will be created if either update() or close() are used. If a zip file exists, it will be parsed and read.

Parameters:

file - the zip file

options - configuration options

Throws:

java.io.IOException - some file exists but could not be read

ZFile

public ZFile(@Nonnull
             java.io.File file,
             @Nonnull
             ZFileOptions options,
             boolean readOnly)
      throws java.io.IOException

Creates a new zip file. If the zip file does not exist, then no file is created at this point and ZFile will contain an empty structure. However, an (empty) zip file will be created if either update() or close() are used. If a zip file exists, it will be parsed and read.

Parameters:

file - the zip file

options - configuration options

readOnly - should the file be open in read-only mode? If true then the file must exist and no methods can be invoked that could potentially change the file

Throws:

java.io.IOException - some file exists but could not be read

Method Detail

entries

@Nonnull
public java.util.Set<StoredEntry> entries()

Obtains all entries in the file. Entries themselves may be or not written in disk. However, all of them can be open for reading.

Returns:

all entries in the zip

get

@Nullable
public StoredEntry get(@Nonnull
                                 java.lang.String path)

Obtains an entry at a given path in the zip.

Parameters:

path - the path

Returns:

the entry at the path or null if none exists

directOpen

@Nonnull
public java.io.InputStream directOpen(long start,
                                               long end)
                                        throws java.io.IOException

Opens a portion of the zip for reading. The zip must be open for this method to be invoked. Note that if the zip has not been updated, the individual zip entries may not have been written yet.

Parameters:

start - the index within the zip file to start reading

end - the index within the zip file to end reading (the actual byte pointed by end will not be read)

Returns:

a stream that will read the portion of the file; no decompression is done, data is returned as is

Throws:

java.io.IOException - failed to open the zip file

update

public void update()
            throws java.io.IOException

Updates the file writing new entries and removing deleted entries. This will force reopening the file as read/write if the file wasn't open in read/write mode.

Throws:

java.io.IOException - failed to update the file; this exception may have been thrown by the compressor but only reported here

close

public void close()
           throws java.io.IOException

Updates the file and closes it.

Specified by:

close in interface java.io.Closeable

Specified by:

close in interface java.lang.AutoCloseable

Throws:

java.io.IOException

getCentralDirectoryBytes

@Nonnull
public byte[] getCentralDirectoryBytes()
                                         throws java.io.IOException

Obtains the byte array representation of the central directory. The central directory must have been already computed. If there are no entries in the zip, the central directory will be empty.

Returns:

the byte representation, or an empty array if there are no entries in the zip

Throws:

java.io.IOException - failed to compute the central directory byte representation

getEocdBytes

@Nonnull
public byte[] getEocdBytes()
                             throws java.io.IOException

Obtains the byte array representation of the EOCD. The EOCD must have already been computed for this method to be invoked.

Returns:

the byte representation of the EOCD

Throws:

java.io.IOException - failed to obtain the byte representation of the EOCD

openReadOnly

public void openReadOnly()
                  throws java.io.IOException

If the zip file is closed, opens it in read-only mode. If it is already open, does nothing. In general, it is not necessary to directly invoke this method. However, if directly reading the zip file using, for example directRead(long, byte[]), then this method needs to be called.

Throws:

java.io.IOException - failed to open the file

add

public void add(@Nonnull
                java.lang.String name,
                @Nonnull
                java.io.InputStream stream)
         throws java.io.IOException

Equivalent to call add(String, InputStream, boolean) using true as mayCompress.

Parameters:

name - the file name (i.e., path); paths should be defined using slashes and the name should not end in slash

stream - the source for the file's data

Throws:

java.io.IOException - failed to read the source data

java.lang.IllegalStateException - if the file is in read-only mode

add

public void add(@Nonnull
                java.lang.String name,
                @Nonnull
                java.io.InputStream stream,
                boolean mayCompress)
         throws java.io.IOException

Adds a file to the archive.

Adding the file will not update the archive immediately. Updating will only happen when the update() method is invoked.

Adding a file with the same name as an existing file will replace that file in the archive.

Parameters:

name - the file name (i.e., path); paths should be defined using slashes and the name should not end in slash

stream - the source for the file's data

mayCompress - can the file be compressed? This flag will be ignored if the alignment rules force the file to be aligned, in which case the file will not be compressed.

Throws:

java.io.IOException - failed to read the source data

java.lang.IllegalStateException - if the file is in read-only mode

mergeFrom

public void mergeFrom(@Nonnull
                      ZFile src,
                      @Nonnull
                      java.util.function.Predicate<java.lang.String> ignoreFilter)
               throws java.io.IOException

Adds all files from another zip file, maintaining their compression. Files specified in src that are already on this file will replace the ones in this file. However, if their sizes and checksums are equal, they will be ignored.

This method will not perform any changes in itself, it will only update in-memory data structures. To actually write the zip file, invoke either update() or close().

Parameters:

src - the source archive

ignoreFilter - predicate that, if true, identifies files in src that should be ignored by merging; merging will behave as if these files were not there

Throws:

java.io.IOException - failed to read from src or write on the output

java.lang.IllegalStateException - if the file is in read-only mode

touch

public void touch()

Forcibly marks this zip file as touched, forcing it to be updated when update() or close() are invoked.

Throws:

java.lang.IllegalStateException - if the file is in read-only mode

finishAllBackgroundTasks

public void finishAllBackgroundTasks()
                              throws java.io.IOException

Wait for any background tasks to finish and report any errors. In general this method does not need to be invoked directly as errors from background tasks are reported during add(String, InputStream, boolean), update() and close(). However, if required for some purposes, e.g., ensuring all notifications have been done to extensions, then this method may be called. It will wait for all background tasks to complete.

Throws:

java.io.IOException - some background work failed

realign

public boolean realign()
                throws java.io.IOException

Realigns all entries in the zip. This is equivalent to call StoredEntry.realign() for all entries in the zip file.

Returns:

has any entry been changed? Note that for entries that have not yet been written on the file, realignment does not count as a change as nothing needs to be updated in the file; entries that have been updated may have been recreated and the existing references outside of ZFile may refer to StoredEntrys that are no longer valid

Throws:

java.io.IOException - failed to realign the zip; some entries in the zip may have been lost due to the I/O error

java.lang.IllegalStateException - if the file is in read-only mode

addZFileExtension

public void addZFileExtension(@Nonnull
                              ZFileExtension extension)

Adds an extension to this zip file.

Parameters:

extension - the listener to add

Throws:

java.lang.IllegalStateException - if the file is in read-only mode

removeZFileExtension

public void removeZFileExtension(@Nonnull
                                 ZFileExtension extension)

Removes an extension from this zip file.

Parameters:

extension - the listener to remove

Throws:

java.lang.IllegalStateException - if the file is in read-only mode

directWrite

public void directWrite(long offset,
                        @Nonnull
                        byte[] data,
                        int start,
                        int count)
                 throws java.io.IOException

Directly writes data in the zip file. Incorrect use of this method may corrupt the zip file. Invoking this method may force the zip to be reopened in read/write mode.

Parameters:

offset - the offset at which data should be written

data - the data to write, may be an empty array

start - start offset in data where data to write is located

count - number of bytes of data to write

Throws:

java.io.IOException - failed to write the data

java.lang.IllegalStateException - if the file is in read-only mode

directWrite

public void directWrite(long offset,
                        @Nonnull
                        byte[] data)
                 throws java.io.IOException

Same as directWrite(offset, data, 0, data.length).

Parameters:

offset - the offset at which data should be written

data - the data to write, may be an empty array

Throws:

java.io.IOException - failed to write the data

java.lang.IllegalStateException - if the file is in read-only mode

directSize

public long directSize()
                throws java.io.IOException

Returns the current size (in bytes) of the underlying file.

Throws:

java.io.IOException - if an I/O error occurs

directRead

public int directRead(long offset,
                      @Nonnull
                      byte[] data,
                      int start,
                      int count)
               throws java.io.IOException

Directly reads data from the zip file. Invoking this method may force the zip to be reopened in read/write mode.

Parameters:

offset - the offset at which data should be written

data - the array where read data should be stored

start - start position in the array where to write data to

count - how many bytes of data can be written

Returns:

how many bytes of data have been written or -1 if there are no more bytes to be read

Throws:

java.io.IOException - failed to write the data

directRead

public int directRead(long offset,
                      @Nonnull
                      java.nio.ByteBuffer dest)
               throws java.io.IOException

Directly reads data from the zip file. Invoking this method may force the zip to be reopened in read/write mode.

Parameters:

offset - the offset from which data should be read

dest - the output buffer to fill with data from the offset.

Returns:

how many bytes of data have been written or -1 if there are no more bytes to be read

Throws:

java.io.IOException - failed to write the data

directRead

public int directRead(long offset,
                      @Nonnull
                      byte[] data)
               throws java.io.IOException

Same as directRead(offset, data, 0, data.length).

Parameters:

offset - the offset at which data should be read

data - receives the read data, may be an empty array

Throws:

java.io.IOException - failed to read the data

directFullyRead

public void directFullyRead(long offset,
                            @Nonnull
                            byte[] data)
                     throws java.io.IOException

Reads exactly data.length bytes of data, failing if it was not possible to read all the requested data.

Parameters:

offset - the offset at which to start reading

data - the array that receives the data read

Throws:

java.io.IOException - failed to read some data or there is not enough data to read

directFullyRead

public void directFullyRead(long offset,
                            @Nonnull
                            java.nio.ByteBuffer dest)
                     throws java.io.IOException

Reads exactly dest.remaining() bytes of data, failing if it was not possible to read all the requested data.

Parameters:

offset - the offset at which to start reading

dest - the output buffer to fill with data

Throws:

java.io.IOException - failed to read some data or there is not enough data to read

addAllRecursively

public void addAllRecursively(@Nonnull
                              java.io.File file)
                       throws java.io.IOException

Adds all files and directories recursively.

Equivalent to calling addAllRecursively(File, Function) using a function that always returns true

Parameters:

file - a file or directory; if it is a directory, all files and directories will be added recursively

Throws:

java.io.IOException - failed to some (or all ) of the files

java.lang.IllegalStateException - if the file is in read-only mode

addAllRecursively

public void addAllRecursively(@Nonnull
                              java.io.File file,
                              @Nonnull
                              java.util.function.Function<? super java.io.File,java.lang.Boolean> mayCompress)
                       throws java.io.IOException

Adds all files and directories recursively.

Parameters:

file - a file or directory; if it is a directory, all files and directories will be added recursively

mayCompress - a function that decides whether files may be compressed

Throws:

java.io.IOException - failed to some (or all ) of the files

java.lang.IllegalStateException - if the file is in read-only mode

getCentralDirectoryOffset

public long getCentralDirectoryOffset()

Obtains the offset at which the central directory exists, or at which it will be written if the zip file were to be flushed immediately.

Returns:

the offset, in bytes, where the central directory is or will be written; this value includes any extra offset for the central directory

getCentralDirectorySize

public long getCentralDirectorySize()

Obtains the size of the central directory, if the central directory is written in the zip file.

Returns:

the size of the central directory or -1 if the central directory has not been computed

getEocdOffset

public long getEocdOffset()

Obtains the offset of the EOCD record, if the EOCD has been written to the file.

Returns:

the offset of the EOCD or -1 if none exists yet

getEocdSize

public long getEocdSize()

Obtains the size of the EOCD record, if the EOCD has been written to the file.

Returns:

the size of the EOCD of -1 it none exists yet

getEocdComment

@Nonnull
public byte[] getEocdComment()

Obtains the comment in the EOCD.

Returns:

the comment exactly as it was encoded in the EOCD, no encoding conversion is done

setEocdComment

public void setEocdComment(@Nonnull
                           byte[] comment)

Sets the comment in the EOCD.

Parameters:

comment - the new comment; no conversion is done, these exact bytes will be placed in the EOCD comment

Throws:

java.lang.IllegalStateException - if file is in read-only mode

setExtraDirectoryOffset

public void setExtraDirectoryOffset(long offset)

Sets an extra offset for the central directory. See class description for details. Changing this value will mark the file as dirty and force a rewrite of the central directory when updated.

Parameters:

offset - the offset or 0 to write the central directory at its current location

Throws:

java.lang.IllegalStateException - if file is in read-only mode

getExtraDirectoryOffset

public long getExtraDirectoryOffset()

Obtains the extra offset for the central directory. See class description for details.

Returns:

the offset or 0 if no offset is set

areTimestampsIgnored

public boolean areTimestampsIgnored()

Obtains whether this ZFile is ignoring timestamps.

Returns:

are the timestamps being ignored?

sortZipContents

public void sortZipContents()
                     throws java.io.IOException

Sorts all files in the zip. This will force all files to be loaded and will wait for all background tasks to complete. Sorting files is never done implicitly and will operate in memory only (maybe reading files from the zip disk into memory, if needed). It will leave the zip in dirty state, requiring a call to update() to force the entries to be written to disk.

Throws:

java.io.IOException - failed to load or move a file in the zip

java.lang.IllegalStateException - if file is in read-only mode

getFile

@Nonnull
public java.io.File getFile()

Obtains the filesystem path to the zip file.

Returns:

the file that may or may not exist (depending on whether something existed there before the zip was created and on whether the zip has been updated or not)

hasPendingChangesWithWait

public boolean hasPendingChangesWithWait()
                                  throws java.io.IOException

Are there in-memory changes that have not been written to the zip file?

Waits for all pending processing which may make changes.

Throws:

java.io.IOException