- unfortunately this changes our INDEX format
  - on Ubuntu <16.04 hashdeep seems to be not available
  - on Ubuntu >14.04 sha1deep seems to be not available
  - for a long term archival program this situation is problematic, we
    have to make sure the indexer/hasher is available for 10 years,
    can't rely on linux distros here.
  - metadata test is turned off for now until I find a good solution for
    the hashdeep/sha1deep problem

  - do not depend on external tool awk
  - no need to handle pipestatus

  - most likely compressable data will be already compressed
  - will make archive generation faster
  - usually, target systems (like in the case of RZG) will do
    compression already

  - useful for versioning

It seems that the order of the files in "tar t" mode is not
predictable. Therefore sort the test data file and run sort
on the test output before comparison.

by adding tons of commentary