ZTree.com  | ZEN  | About...  

 Index   Back

Or have a hash comparison option? An idea I like even more!   [Discuss]

By: John Leslie       
Date: Jun 17,2018 at 12:41
In Response to: Do buffers mess with verification? (Ron Metzger)

> > > My guess is that Syncback Pro or Qnap are initiating a File
> Compare
> > > independently of the copy process, Correct?
> >
> > Yes, and I'd like that option for ZTreeeWin for copying important
> stuff
> >
> > >
> > > If that is not correct, consider using ZTree CLP /APIT instead of
> using
> > > ZTreeWin's native copy process. (Or if using /APIT, use
> ZTreeWin's
> > > native copy process.)
> >
> > I always use that, partly as I came up with the idea originally :-)
> > http://www.ztw3.com/forum/forum_entry.php?id=119982
>
> In order to have ZTreeWin do the file compare, you would have to give
> up the use of /API or /APIT. This would impact performance significantly
> as has been shown since ZTreeWin would be have to WAIT for the cache to
> commit, then read the resulting file stream.
>
> This would not give you the Reason for the failure, just when it
> happens. Of course, you would need to be At the console when it happens.
> I usually start a large copy/move process and let it rip while I do other
> things.
>
> I Still think that Alt-Compare would provide the solution. Perhaps,
> logging comparison errors, like SyncBack Pro or Qnap does, would help.
>
> This problem reminds me of the Verify process on Tape Backup systems 30
> or so years ago. The Verify process use to write to tape, then shoeshine
> the tape back and re-read what it wrote, and compare against the source.
> It put so much wear and tear on the tape drive that the manufacturers
> eventually disallowed that form of verify. (Drives were failing so fast
> that something had to be done.) Instead they calculate (for x-amount of
> data) a CRC at the source, write the x-amount of data and store the CRC
> of that data on tape, later checking the Source CRC to the tape CRC. Even
> that was not very effective. Still produces a lot of shoe shining of the
> tape.
>
> Modern Tape drives now use error detection and log Soft and Hard read
> or write errors on the tape as they happen. This is done at the hardware
> level and logged in the software (which is fairly easy to do on
> sequential data streams like tape). When a Error occurs, it skips that
> part of the tape and tries again on another part of the tape, logging
> that part of the tape as potentially bad or actually bad.
>
> Today's hard drives and flash drives, use a similar approach for error
> recovery. NAS' have a lot more moving parts, such as the network
> (ethernet?), unknown/untested hardware (caches) used for storage, and a
> NAS OS interaction with the hardware. Each by themselves might be highly
> reliable, but combined may create failures not easily detected.
> Diagnosing the failure is a daunting process. Was it a bad packet, a soft
> error on drive, did the NAS OS not wait long enough for the cache to
> clear, etc. etc.
>
> Ultimately, I feel that ZTreeWin should not spend too much effort
> correcting or handling hardware errors, in my humble opinion.
>
> Thanks for reading my diatribe.
> Ron Metzger

I understand it would be slower and I only suggest it as an option for when it's critical to make sure files are transferred correctly, but it happens often enough doing a post-copy verify would be a pain, especially when just a handful of files in a tree structure are updated. Plus just a suggestion for debate, I found a real-World problem (I've copied a LOT of files to a NAS and I'm pretty confident about 1 in 10k-50k got corrupted.)

I suppose an alternative would be to have an improved md5deep-like behaviour, where ZTreeWin can calculate hashes for tagged files and store them in a text file (user-chosen name, like ZLog). Then another option to compare hashes of tagged files with what's stored in a selected hash-text file (matched by identical name and size so they could be somewhere else). Okay I like that a lot! What do you think? Helps with bit-rot on disks too.

217 views      
Thread locked
 

Messages in this Thread

 
94,424 Postings in 11,918 Threads, 348 registered users, 8 users online (0 registered, 8 guests)
Index | Admin contact |   Forum Time: Sep 16, 2019 - 8:52 am EDT  |  Hits:28,643,043  (716 Today )
RSS Feed