Package net.algart.arrays

Class DefaultDataFileModel

java.lang.Object
net.algart.arrays.AbstractDataFileModel
net.algart.arrays.DefaultDataFileModel
All Implemented Interfaces:
DataFileModel<File>
public class DefaultDataFileModel extends AbstractDataFileModel implements DataFileModel<File>

Default implementation of DataFileModel that creates usual Java files, which are mapped via standard Java technique (FileChannel.map method).

The data files, returned by this class, creates buffer holders with the unmap(boolean) method which does not perform actual unmapping: Java NIO package does not support unmapping. File mapping will be released automatically by the built-in finalizers.

The close() method in data files, returned by this class, perform closing via RandomAccessFile.close() method, but it may not completely close the disk file! The disk file will be completely closed and all connected system resources will be freed only while the following garbage collection at the unspecified moment, or while exiting JVM. So, if you need to process a very large number of AlgART arrays (tens of thousands or millions), we recommend to use the StandardIODataFileModel and call Array.freeResources(ArrayContext) method in time.

See comments to AbstractDataFileModel.createTemporary(boolean) method for information about temporary files created by this class.

Warning: under Sun Java versions before 1.7 (i.e. 1.5 and 1.6), this data file model is not stable, due to the Sun's bug "(fc) "Cleaner terminated abnormally" error in simple mapping test". More precisely, for Java prior to 1.7:

  • In lazy-writing mode, this model is unstable at all: processing large arrays can lead to internal error while garbage collection, that will lead to immediate abnormal JVM termination. Due to this reason, the default lazy-writing mode is false in Java versions prior to 1.7, but true in Java 1.7+.
  • In usual mode, this model can occasionally lead to unexpected IOError while processing large arrays. Unlike an internal error in the lazy-writing mode, this exception can be normally caught and shown to the user in GUI applications. It can occur with large bank size (32 MB or more), if an array occupies several banks. This probability of this situation is not too high for unresizable arrays when single mapping is used.

This class is immutable and thread-safe: there are no ways to modify settings of the created instance.

Author:
Daniel Alievsky
See Also:

  • Constructor Details

    • DefaultDataFileModel

      public DefaultDataFileModel()
      Equivalent to new DefaultDataFileModel(null, 0, defaultLazyWriting()).

    • DefaultDataFileModel

      public DefaultDataFileModel(boolean lazyWriting)
      Equivalent to new DefaultDataFileModel(null, 0, lazyWriting).
      Parameters:
      lazyWriting - it true, lazy-writing mode will be used.

    • DefaultDataFileModel

      public DefaultDataFileModel(File tempPath)
      Equivalent to new DefaultDataFileModel(tempPath, 0, defaultLazyWriting()).
      Parameters:
      tempPath - the path where new temporary files will be created by AbstractDataFileModel.createTemporaryFile(boolean) method or null if the default temporary-file directory is to be used.

    • DefaultDataFileModel

      public DefaultDataFileModel(File tempPath, boolean lazyWriting)
      Equivalent to new DefaultDataFileModel(tempPath, 0, lazyWriting).
      Parameters:
      tempPath - the path where new temporary files will be created by AbstractDataFileModel.createTemporaryFile(boolean) method or null if the default temporary-file directory is to be used.
      lazyWriting - it true, lazy-writing mode will be used.

    • DefaultDataFileModel

      public DefaultDataFileModel(File tempPath, long prefixSize, boolean lazyWriting)
      Creates new instance with specified lazy-writing mode.

      Please see AbstractDataFileModel(File, long) about tempPath and prefixSize arguments.

      The lazyWriting argument specifies whether the data files will use lazy writing mode. Namely, if this flag is set, flushing or unmapping the mapped regions via DataFile.BufferHolder.flush(false), DataFile.BufferHolder.unmap(false) calls will not lead to immediate writing data to the disk file: this method will not do anything. Instead, the data will be really written by garbage collector. If this flag is not set, any call of DataFile.BufferHolder.flush(boolean) or DataFile.BufferHolder.unmap(boolean) method forces writing data to the disk by force() method of MappedByteBuffer class.

      By default, if you use constructors without lazyWriting argument, this flag is retrieved from the system property "net.algart.arrays.DefaultDataFileModel.lazyWriting" or, if there is no such property, is set to true in Java 1.7+ or false in Java 1.5 and 1.6. Please see defaultLazyWriting().

      Usually, you should set lazyWriting flag to true. It can essentially increase the performance, if you create and modify many large AlgART arrays, because OS will store the new data in the cache and will not physically write data to the disk. Even in this case, this class periodically flushs the unsaved data, when the summary amount of mapped buffers exceeds the limit returned by maxMappedMemory() method.

      The false value of this flag may be recommended if the stable behavior of your application is more important than the speed. If lazy writing is disabled, the application will use less RAM and the risk of swapping will be much less, because each new data will be immediately saved to the disk and will not be cached in RAM.

      Unfortunately, lazy-writing mode leads to internal Sun's bug in Java 1.5 and 1.6: we recommend never set it to true in these Java versions. The detailed description of this bug is here: "(fc) "Cleaner terminated abnormally" error in simple mapping test".

      Parameters:
      tempPath - the path where new temporary files will be created by AbstractDataFileModel.createTemporaryFile(boolean) method or null if the default temporary-file directory is to be used.
      prefixSize - the value returned by AbstractDataFileModel.recommendedPrefixSize() implementation in this class.
      lazyWriting - it true, lazy-writing mode will be used.
      See Also:

  • Method Details

    • defaultLazyWriting

      public static boolean defaultLazyWriting()
      Default lazy-writing mode, used when this this class is instantiated by a constructor without lazyWriting argument. More precisely, if there is the system property "net.algart.arrays.DefaultDataFileModel.lazyWriting", containing "true" or "false" string (in lower case), this method returns true if this property contains "true" and false if this property contains "false". If there is no such property, or if it contains illegal string (not "true" or "false"), or if some exception occurred while calling System.getProperty, this method returns true in Java 1.7 or higher Java version and false in Java 1.5 and Java 1.6. The value of this system property is loaded and checked only once while initializing DefaultDataFileModel class.
      Returns:
      default lazy-writing mode

    • maxMappedMemory

      public static long maxMappedMemory()
      The maximal amount of RAM (in bytes), allowed for simultaneous mapping by FileChannel.map method without flushing data to the disk by MappedByteBuffer.force() method.

      This value is important while using lazy-writing mode. In this case, a lot of mapping requests (calls of FileChannel.map), with modifications of the mapped data and without further MappedByteBuffer.force(), will use RAM for storing the changed data in the system cache. When all (or almost all) available RAM will be spent, it may lead to intensive disk swapping. The reason is that the mapped memory is not controlled by Java garbage collector: it is possible to map much more disk memory than Runtime.maxMemory(). The result may be extremely slowing down of all other applications, working on the computer, and even practical impossibility of any work: all RAM will be occupied by your application.

      To avoid this behavior, this class controls the total amount of mapped memory (summary size of all mapped buffers in all files), and when it exceeds the limit, returned by this method, calls MappedByteBuffer.force() for all currently mapped buffers and, so, flushs the data to the disk and frees the system memory.

      This value, returned by this method, is retrieved from the system property "net.algart.arrays.maxMappedMemory", if it exists and contains a valid integer number. If this property contains zero or negative integer, this method returns 0, and it means that the amount of RAM for simultaneous mapping is not limited at all: the application will try to use all available system memory. If this property contains an integer greater than the limit 256~7.2*1016, this limit is used instead: it guarantees that using this value will not lead to integer overflow. If there is no such property, or if it contains not a number, or if some exception occurred while calling Long.getLong, this method returns the default value 536870912 (512 MB). The value of this system property is loaded and checked only once while initializing DefaultDataFileModel class.

      We recommend to always set this system property in serious applications, working with large AlgART arrays. The suitable value is about 50-100% of the current RAM installed on the computer. The default value 512 MB works well if you don't need to process larger amounts of data frequently.

      Returns:
      the maximal amount of RAM (in bytes), allowed for simultaneous mapping by this class.

    • isLazyWriting

      public final boolean isLazyWriting()
      Returns the lazyWriting argument, passed to the constructor while creating this instance.
      Returns:
      the lazyWriting flag, passed to the constructor.

    • getDataFile

      public DataFile getDataFile(File path, ByteOrder byteOrder)
      This implementation returns the data file corresponding to usual Java file new java.io.File(path) with DataFile.map method based on standard Java mapping.

      This method never throws java.io.IOError.

      Specified by:
      getDataFile in interface DataFileModel<File>
      Specified by:
      getDataFile in class AbstractDataFileModel
      Parameters:
      path - the path to disk file (as the argument of new java.io.File(path)).
      byteOrder - the byte order that will be always used for mapping this file.
      Returns:
      new instance of DataFile object.
      Throws:
      NullPointerException - if one of the passed arguments is null.

    • getPath

      public File getPath(DataFile dataFile)
      Returns the absolute path to the disk file (java.io.File.getAbsoluteFile()). The argument may be created by this data file model or by StandardIODataFileModel.

      This method never throws java.io.IOError.

      Specified by:
      getPath in interface DataFileModel<File>
      Specified by:
      getPath in class AbstractDataFileModel
      Parameters:
      dataFile - the data file.
      Returns:
      the absolute path to the disk file.
      Throws:
      NullPointerException - if the argument is null.
      ClassCastException - if the data file was created by data file model, other than DefaultDataFileModel or StandardIODataFileModel.

    • isAutoDeletionRequested

      public boolean isAutoDeletionRequested()

      This implementation returns true.

      Specified by:
      isAutoDeletionRequested in interface DataFileModel<File>
      Specified by:
      isAutoDeletionRequested in class AbstractDataFileModel
      Returns:
      true.

    • recommendedNumberOfBanks

      public int recommendedNumberOfBanks()

      This implementation returns the value Integer.getInteger("net.algart.arrays.DefaultDataFileModel.numberOfBanksPerCPU", 3) * Arrays.SystemSettings.availableProcessors(), stored while initializing this DefaultDataFileModel class, or default value 3 * Arrays.SystemSettings.availableProcessors(), if some exception occurred while calling Integer.getInteger. If this value is less than 2, returns 2. If "net.algart.arrays.DefaultDataFileModel.numberOfBanksPerCPU" property contains negative or zero integer, returns 2.

      Please note that many algorithms, on multiprocessor or multi-core systems, use several parallel threads for processing arrays: see Arrays.ParallelExecutor. So, the number of banks should be enough for parallel using by all CPU units, to avoid frequently bank swapping. There should be at least 2 banks per each CPU unit, better 3-4 banks (for complex random-access algorithms).

      Specified by:
      recommendedNumberOfBanks in interface DataFileModel<File>
      Specified by:
      recommendedNumberOfBanks in class AbstractDataFileModel
      Returns:
      the recommended number of memory banks.

    • recommendedBankSize

      public int recommendedBankSize(boolean unresizable)

      This implementation returns the value Integer.getInteger("net.algart.arrays.DefaultDataFileModel.bankSize",16777216) (16 MB) when the argument is true and Integer.getInteger("net.algart.arrays.DefaultDataFileModel.resizableBankSize",4194304) (4 MB) when the argument is false on 64-bit Java machines. On 32-bit JVM, this method returns Integer.getInteger("net.algart.arrays.DefaultDataFileModel.bankSize32",4194304) (4 MB) when the argument is true and Integer.getInteger("net.algart.arrays.DefaultDataFileModel.resizableBankSize32",2097152) (2 MB) when the argument is false. These values are stored while initializing DefaultDataFileModel class. If some exceptions occur while calling Integer.getInteger, the default values 16777216 / 4194304 (for 64-bit Java) or 4194304 / 2097152 (for 32-bit Java) are returned. If this property contains invalid value (for example, not a power of two), this value is automatically corrected to the nearest valid one.

      This method distinguishes between 32-bit and 64-bit Java via Arrays.SystemSettings.isJava32() method. Please remember that the result of that method is not 100% robust; so, please not specify too high values if you are not quite sure that your JVM is not 32-bit and has no 32-bit limitations for the address space.

      Specified by:
      recommendedBankSize in interface DataFileModel<File>
      Specified by:
      recommendedBankSize in class AbstractDataFileModel
      Parameters:
      unresizable - true if this bank size will be used for unresizable arrays only.
      Returns:
      the recommended size of every memory bank in bytes.
      See Also:

    • recommendedSingleMappingLimit

      public int recommendedSingleMappingLimit()

      This implementation returns the value Integer.getInteger("net.algart.arrays.DefaultDataFileModel.singleMappingLimit",268435456) (256 MB) on 64-bit Java machines. On 32-bit JVM, this method returns Integer.getInteger("net.algart.arrays.DefaultDataFileModel.singleMappingLimit32",4194304) (4 MB). This value is stored while initializing DefaultDataFileModel class. If some exceptions occur while calling Integer.getInteger, the default value 268435456 (or 4194304 for 32-bit Java) is returned.

      This method distinguishes between 32-bit and 64-bit Java via Arrays.SystemSettings.isJava32() method. Please remember that the result of that method is not 100% robust; so, please not specify too high values if you are not quite sure that your JVM is not 32-bit and has no 32-bit limitations for the address space.

      Specified by:
      recommendedSingleMappingLimit in interface DataFileModel<File>
      Overrides:
      recommendedSingleMappingLimit in class AbstractDataFileModel
      Returns:
      the recommended limit for file size, in bytes, so that less files, if they are unresizable, should be mapped only once by single call of DataFile.map(net.algart.arrays.DataFile.Range, boolean) method.

    • autoResizingOnMapping

      public boolean autoResizingOnMapping()

      This implementation returns the value Boolean.getBoolean("net.algart.arrays.DefaultDataFileModel.autoResizingOnMapping"), stored while initializing DefaultDataFileModel class, or false if there is no such system property or some exception occurred while calling Boolean.getBoolean.

      Specified by:
      autoResizingOnMapping in interface DataFileModel<File>
      Overrides:
      autoResizingOnMapping in class AbstractDataFileModel
      Returns:
      true if mapping outside the file length automatically increase the length.

    • temporaryFilePrefix

      public String temporaryFilePrefix()
      This implementation returns "mapmm";
      Overrides:
      temporaryFilePrefix in class AbstractDataFileModel
      Returns:
      "mapmm".

    • toString

      public String toString()
      Returns a brief string description of this class.

      The result of this method may depend on implementation.

      Overrides:
      toString in class Object
      Returns:
      a brief string description of this object.