Package net.algart.arrays

Class AbstractDataFileModel

java.lang.Object
net.algart.arrays.AbstractDataFileModel
All Implemented Interfaces:
DataFileModel<File>
Direct Known Subclasses:
DefaultDataFileModel, StandardIODataFileModel
public abstract class AbstractDataFileModel extends Object implements DataFileModel<File>

A skeletal implementation of the DataFileModel interface to minimize the effort required to implement this interface for processing usual disk files.

Author:
Daniel Alievsky

  • Field Details

    • allTemporaryFiles

      protected final Set<DataFile> allTemporaryFiles
      The internal synchronized set of all non-deleted temporary files: allTemporaryFiles() method returns its clone. createTemporary(boolean) method adds a path to this set, delete(DataFile) method removes the path from it.

    • tempPath

      protected final File tempPath
      The path where new temporary files will be created by createTemporaryFile(boolean) method or null if the default temporary-file directory is to be used. If it is not null, it can refer to: 1) an existing directory, 2) existing file or 3) non-existing file/directory. In 1st case, it is considered as the temporary-file directory, in which new temporary files will be created. In 2nd and 3rd case, it is considered as a file name: the name of new temporary file will always equal to this (so, it will be possible to create only one temporary file). It is equal to the first argument of AbstractDataFileModel(File, long) constructor.
      See Also:

    • prefixSize

      protected final long prefixSize
      The value returned by recommendedPrefixSize() method. It is equal to the second argument of AbstractDataFileModel(File, long) constructor.

  • Constructor Details

    • AbstractDataFileModel

      protected AbstractDataFileModel()
      Equivalent to AbstractDataFileModel(null, 0).

    • AbstractDataFileModel

      protected AbstractDataFileModel(File tempPath, long prefixSize)
      Creates a new instance with the specified temporary-file path and the starting gap size in all temporary files.

      The tempPath argument may refer both to a directory and to a file. The following 4 cases are possible.

      1. tempPath is an existing directory or another non-file disk resource; more precisely, tempPath.exists() && !tempPath.isFile(). In this case, new temporary files, created by createTemporaryFile(boolean) method, will be placed in this directory with unique names.
      2. tempPath is null. In this case, new temporary files, created by createTemporaryFile(boolean) method, will be placed in the default system-dependent temporary-file directory.
      3. tempPath is an existing file (tempPath.isFile()) or
      4. tempPath does not exists (!tempPath.exists()). These are special cases! The createTemporaryFile(boolean) will create the file at this position (tempPath.createNewFile()) and return tempPath. In other words, this data file model will not be able to create more than one temporary file, and all further temporary files, if will be requested, will overwrite the same file.

      The last variants 3 and 4 can be useful for creating a single array mapped to some required file. For example, the following code will create new array, containing 1024 int values, in the file "myfile.dat": 

       DefaultDataFileModel dfm = new DefaultDataFileModel("myfile.dat");
 LargeMemoryModel<File> mm = LargeMemoryModel.getInstance(dfm);
 UpdatableIntArray dest = mm.newUnresizableIntArray(1024);
 LargeMemoryModel.setTemporary(dest, false);

      Please be careful: if there is an existing subdirectory with the same "myfile.dat" name, the behavior of the data file model will be another (the variant #1).

      The prefixSize defines the starting gap in all temporary files, returned by recommendedPrefixSize() implementation in this class. Usually you may pass 0 here. The only case when it makes sense to pass positive value is if you are planning to clear temporary status of the created array via LargeMemoryModel.setTemporary(Array, boolean) method and save some additional meta-information in the file prefix — element type, array length, etc. A good example of using prefix is the tool MatrixInfo.

      Parameters:
      tempPath - the path where new temporary files will be created by createTemporaryFile(boolean) method or null if the default temporary-file directory is to be used.
      prefixSize - the value returned by recommendedPrefixSize() implementation in this class.

  • Method Details

    • pathClass

      public Class<File> pathClass()
      This implementation returns File.class.
      Specified by:
      pathClass in interface DataFileModel<File>
      Returns:
      File.class

    • getDataFile

      public abstract DataFile getDataFile(File path, ByteOrder byteOrder)
      Description copied from interface: DataFileModel
      Returns a new instance of DataFile object corresponding to the given path. This path will be returned by DataFileModel.getPath(DataFile) method for the returned object.

      The passed byte order will be used for mapping this file: the DataFile.map(DataFile.Range, boolean) method of the data file will return ByteBuffer with this byte order.

      The physical object (for example, disk file), described by path string, should already exist. This method does not attempt to create physical file; it only creates new Java object associated with an existing file.

      This method never throws java.io.IOError.

      Specified by:
      getDataFile in interface DataFileModel<File>
      Parameters:
      path - the path describing unique position of the existing data file.
      byteOrder - the byte order that will be always used for mapping this file.
      Returns:
      new instance of DataFile object.

    • getPath

      public abstract File getPath(DataFile dataFile)
      Description copied from interface: DataFileModel
      Returns the path describing unique position of the data file (usually the absolute path to the disk file).

      This method never throws java.io.IOError.

      Specified by:
      getPath in interface DataFileModel<File>
      Parameters:
      dataFile - the data file.
      Returns:
      the path describing unique position of the data file.

    • createTemporary

      public DataFile createTemporary(boolean unresizable)
      This implementation creates a temporary file by calling createTemporaryFile(boolean) protected method and returns a result of getDataFile(pathToTemporaryFile, ByteOrder.nativeOrder()). You may override createTemporaryFile(boolean) method to change a location or names of temporary files.

      The byte order in the created file will be equal to the result of byteOrderInTemporaryFiles() protected method. You may override it to specify custom byte order in temporary files.

      This method adds the created instance into allTemporaryFiles, if isAutoDeletionRequested() method returns true.

      Specified by:
      createTemporary in interface DataFileModel<File>
      Parameters:
      unresizable - the argument passed to createTemporaryFile(boolean).
      Returns:
      new instance of DataFile object corresponding newly created temporary data file.
      Throws:
      IOError - if createTemporaryFile(boolean) throws IOException.

    • delete

      public boolean delete(DataFile dataFile)
      This implementation removes the file by the call of standard java.io.File.delete() method: getPath(dataFile).delete() (if java.io.File.exists() method returns true). After deletion, this implementation removes the passed instance from allTemporaryFiles.

      This implementation is fully synchronized on the internal set returned by allTemporaryFiles() method.

      You should override this method if the data files in this model do not correspond to usual disk files.

      Specified by:
      delete in interface DataFileModel<File>
      Parameters:
      dataFile - the data file that should be deleted.
      Returns:
      true if and only if the data file existed and was successfully deleted, false if the data file does not exist (maybe was deleted already).
      Throws:
      IOError - in a case of any problems while file deletion.
      NullPointerException - if the passed data file is null.

    • finalizationNotify

      public void finalizationNotify(File dataFilePath, boolean isApplicationShutdown)
      This implementation does nothing.
      Specified by:
      finalizationNotify in interface DataFileModel<File>
      Parameters:
      dataFilePath - the path describing unique position of the data file.
      isApplicationShutdown - true if this method is called by the cleanup procedure, performed by this package, while finishing the application; false if it is called from the garbage collector.

    • allTemporaryFiles

      public Set<DataFile> allTemporaryFiles()
      Description copied from interface: DataFileModel
      Returns the set of all data files, that are temporary and should be automatically deleted while system shutdown. The returned set is an immutable view or a newly allocated copy of an internal set stored in this instance. The returned instance must not be null (but may be the empty set).

      Usually this method returns the set of temporary files that were created by DataFileModel.createTemporary(boolean) method by this instance of this factory, but not were successfully deleted by DataFileModel.delete(DataFile) method yet.

      This package includes automatic cleanup procedure, that is performed in the internal shutdown hook and calls DataFileModel.delete(DataFile) for all data files, returned by this method for all instances of this class, which were used since the application start and returned true as a result of DataFileModel.isAutoDeletionRequested() method. You may install additional cleanup procedures, that will be called before or after this, via Arrays.addShutdownTask(Runnable, Arrays.TaskExecutionOrder) method.

      Specified by:
      allTemporaryFiles in interface DataFileModel<File>
      Returns:
      the set of the paths of all created temporary files.

    • setTemporary

      public void setTemporary(DataFile dataFile, boolean value)
      Description copied from interface: DataFileModel
      If value is true, adds the passed data file instance into the internal collection returned by DataFileModel.allTemporaryFiles() method; if value is false, removes it from that collection.

      This method does nothing if DataFileModel.isAutoDeletionRequested() returns false.

      This method is called in LargeMemoryModel.setTemporary(net.algart.arrays.Array, boolean) method only.

      In some data file models (implemented in another packages) this method may do nothing or throw an exception.

      Specified by:
      setTemporary in interface DataFileModel<File>
      Parameters:
      dataFile - the data file.
      value - specifies whether the data file should be included into or excluded from the internal collection of temporary files.

    • isAutoDeletionRequested

      public abstract boolean isAutoDeletionRequested()
      Description copied from interface: DataFileModel
      Returns true if the standard cleanup procedure, that deletes all temporary files (as described in comments to DataFileModel.allTemporaryFiles() method), is necessary for this file model. In this case, the instance of this model is automatically registered, while creating any LargeMemoryModel instance with this model, in the internal static collection that can be retrieved by LargeMemoryModel.allUsedDataFileModelsWithAutoDeletion() method.

      This method returns true for all implementations from this package. If you implemented own cleanup procedure in your implementation of this class, you may return false in this method. If this method returns false, the implementations of DataFileModel.createTemporary(boolean) method in this package do not add the file name into the internal collection, returned by DataFileModel.allTemporaryFiles() method.

      If this method returns false, it does not mean that temporary files will not be deleted automatically. It only means that this data file model instance will not be registered in the internal static collection (available via LargeMemoryModel.allUsedDataFileModelsWithAutoDeletion() method) and that the DataFileModel.createTemporary(boolean) method will not register the file name in DataFileModel.allTemporaryFiles() collection. To avoid automatic file deletion, you must call LargeMemoryModel.setTemporary(Array, boolean) method with false second argument.

      Specified by:
      isAutoDeletionRequested in interface DataFileModel<File>
      Returns:
      true if the temporary data files, created by this model, should be automatically deleted by the standard cleanup procedure.

    • recommendedNumberOfBanks

      public abstract int recommendedNumberOfBanks()
      Description copied from interface: DataFileModel
      The number of memory banks, recommended for data files created by this factory. AlgART arrays, based on data file mapping, allocate this number of memory banks and load there portions of large data file.

      The returned number of banks must not be less than 2. In other case, an attempt to create Array instance will throw an exception. Usual values are 8-16.

      Please note that many algorithms, on multiprocessor or multi-core systems, use several parallel threads for processing arrays: see Arrays.ParallelExecutor. So, the number of banks should be enough for parallel using by all CPU units, to avoid frequently bank swapping. There should be at least 2 banks per each CPU unit, better 3-4 banks (for complex random-access algorithms).

      Specified by:
      recommendedNumberOfBanks in interface DataFileModel<File>
      Returns:
      the recommended number of memory banks.

    • recommendedBankSize

      public abstract int recommendedBankSize(boolean unresizable)
      Description copied from interface: DataFileModel
      The size of every memory bank in bytes, recommended for data files created by this factory. AlgART arrays, based on data file mapping, allocate memory banks with this size and load there portions of large data file.

      The unresizable flag specifies whether this bank size will be used for data file, which stores unresizable arrays only. In this case, this method may return greater value than if unresizable is false. The reason is that the data files, containing resizable arrays, may grow per blocks, which size is equal to the bank size. If bank size is 8 MB, then any resizable array, created by MemoryModel.newIntArray(long) or similar method, will occupy at least 8 MB of disk space, even its length is only several int values. For unresizable arrays, created by MemoryModel.newUnresizableIntArray(long) and similar methods, the file size is usually fixed while its creation and bank size information is not used.

      This returned size must be the power of two (2k) and must not be less than 256. In other case, an attempt to create Array instance will throw an exception.

      We recommend use large banks to reduce bank swapping. But do not specify too large values here: every opened data file use DataFileModel.recommendedNumberOfBanks()*DataFileModel.recommendedBankSize(boolean) bytes of the address space, which is limited by ~1.0-1.5 GB under 32-bit OS. Typical value is 2-8 MB for unresizable arrays (when the argument is true) and 64-256 KB for resizable ones (when the argument is false).

      Specified by:
      recommendedBankSize in interface DataFileModel<File>
      Parameters:
      unresizable - true if this bank size will be used for unresizable arrays only.
      Returns:
      the recommended size of every memory bank in bytes.
      See Also:

    • recommendedSingleMappingLimit

      public int recommendedSingleMappingLimit()

      This implementation returns 0.

      Specified by:
      recommendedSingleMappingLimit in interface DataFileModel<File>
      Returns:
      0.

    • recommendedPrefixSize

      public long recommendedPrefixSize()

      This implementation returns prefixSize — the long value passed to the constructor, or 0 if the constructor without long argument was used.

      Specified by:
      recommendedPrefixSize in interface DataFileModel<File>
      Returns:
      prefixSize.
      See Also:

    • autoResizingOnMapping

      public boolean autoResizingOnMapping()

      This implementation returns false.

      Specified by:
      autoResizingOnMapping in interface DataFileModel<File>
      Returns:
      false.

    • tempPath

      public final File tempPath()
      Returns the value of tempPath protected field, in other words, the first argument of AbstractDataFileModel(File, long) constructor.
      Returns:
      the value of tempPath protected field.

    • isConcreteFile

      public final boolean isConcreteFile()
      Returns true if the tempPath field is not null and corresponds to existing file or non-existing file/directory: tempPath != null && (!tempPath.exists() || tempPath.isFile()) In such situations, this data file model always works with the only one constant file, specified in tempPath field.

      This method returns true is the cases #3 and #4, described in the comments to the constructor.

      Returns:
      whether this data file model always works with the fixed file tempPath.

    • temporaryFilePrefix

      public String temporaryFilePrefix()
      Returns the prefix, used by the current implementation of createTemporary(boolean) method while creating temporary files — see comments to that method. This prefix helps to indicate a concrete data file model class, which was used while creating temporaty files.

      The result of this method must be at least three characters long. In other case, its result will be automatically appended with "_" to fulfil the requirements of the standard File.createTempFile method.

      This method must not return null.

      This implementation returns "lmm". The implementation from DefaultDataFileModel class returns "mapmm". The implementation from StandardIODataFileModel class returns "stdmm".

      Returns:
      the prefix (3 characters minimum), which will be added to the name of the temporary files, created by createTemporary(boolean) method.

    • createTemporaryFile

      protected File createTemporaryFile(boolean unresizable) throws IOException
      Creates a new temporary empty disk file and returns the absolute (unique for the file system) path to it. This method is used by createTemporary(boolean) method, if it was not overridden, in particular, in both DefaultDataFileModel and StandardIODataFileModel classes.

      If isConcreteFile() method returns false (i.e. if tempPath is null or an existing directory: tempPath.exists() && !tempPath.isFile()), this implementation returns the following result: File.createTempFile(prefix,suffix,tempPath).getAbsolutePath(), where tempPath is the constructor argument (null for constructor without File argument), prefix is the result of temporaryFilePrefix() method (appened to the length 3 with "_" character if its length <3), suffix is calculated as

       String suffix = (unresizable ? ".uarray." : ".marray.")
 + (byteOrderInTemporaryFiles() == ByteOrder.BIG_ENDIAN ? "be.tmp" : "le.tmp");

      If isConcreteFile() method returns true (i.e. if tempPath is not null, and it is a file or does not exists: !tempPath.exists() || tempPath.isFile()), this implementation calls tempPath.createNewFile() and returns tempPath.

      You may override this method to change this behavior.

      This method does not try to use File.deleteOnExit() method for created files. Instead, the temporary files will deleted via built-in cleanup procedure (described in comments to allTemporaryFiles() method), if isAutoDeletionRequested() method returns true. The reasons are the Sun's bugs #6359560: "(fs) File.deleteOnExit() doesn't work when MappedByteBuffer exists (win)" and #4171239 (Java 1.5 and 1.6): "java.io.File.deleteOnExit does not work on open files (win32)".

      You need to keep in mind that some of these files will not be deleted, if all finalization code will not be performed until system exit. You may call Arrays.gcAndAwaitFinalization(int) method before system exit to reduce the probability of appearing non-deleted files.

      Parameters:
      unresizable - true if this file will be used for unresizable arrays only. It is an information flag: for example, it may be used for choosing file name or directory. If this flag is set, it does not mean that DataFile.length(long) method will not be called to change the file length; it will be called at least once.
      Returns:
      the absolute path to new created temporary file.
      Throws:
      IOException - in a case of any disk errors.

    • byteOrderInTemporaryFiles

      protected ByteOrder byteOrderInTemporaryFiles()
      Returns byte order that is used for new temporary files by createTemporary(boolean) method in this class. Never returns null.

      This implementation returns ByteOrder.nativeOrder(): in most situations, the best choice for temporary data, that will be automatically deleted before application exit and, probably, will never be transferred to another computers. You may override this method to specify another byte order; it be useful if you are planning to clear temporary status for some arrays via LargeMemoryModel.setTemporary(Array, boolean) method.

      Returns:
      byte order that is used for new temporary files.