fsck parallelism

[michaepr]

I'm curious if Greyhole copies data from the samba shares to the storage drives in some parallel way or if it happens sequentially. For example, let's assume I configured there to be two copies of each file on a specific share. During the fsck, is the file first copied to the first drive and only when that succeeds to the second? Or is the writing parallelized? How about parallelization for copying multiple files at once?

Also, in what order are files processed? Maybe there's an iteration through adjacent disk locations?

Just to clarify, I'm not having any issues with Greyhole. I think it works very well. These questions are just for my education and so I can figure out how important sequential vs random read and write speed of my disks is for optimal performance.

Thanks for your work!

[gboudreau]

Greyhole works on each file operation (executed on Samba) in sequence, using a queue system (operations get queued in /var/spool/greyhole first, then are moved into the tasks table in MySQL). So if you create fileA, then fileB, then rename fileA to fileB, it will work through those operations in the same order. It will never work on multiple file operations in parallel, because the order is very important.
fsck is different because it doesn't know about operations, it only knows about which files exist now. So if it finds a file that isn't where it should be, it will handle it as best as possible.

For fsck runs, the order in which folders & files are checked is undefined. I think it uses PHP's glob() function to walk through folders & files, which I think returns results sorted alphabetically.

For file copies specifically, there is a config named parallel_copying, which is true by default, which will use some cat | tee sorcery to create multiple files in parallel. This is used for normal processing by the daemon, and during fsck (as needed).
You can see parallel copying being used in the logs, which will appear as INFO write: Copying X.XXX KB file to: /mnt/hdd5/gh/ShareName/Folder/SomeFile, /mnt/hdd3/gh/ShareName/Folder/SomeFile, which lists all file copies being created simultaneously. (Of note: parallel copying logs the temporary filenames used to transfer; once the transfer completes, it will rename those temporary files using the real filename.)

[michaepr]

Ah ok, I see. I may not have waited long enough for the daemon to notice the new files.

I do have a follow up question. What exactly happens during the file system check? Is it mostly about checking if the files and symlinks are in order? Or is there also a check to see if all the copies of a file are still consistent with each other?

For example, let's assume a share saves data on two drives and there is one file on the share (so there are two copies of it, one on each drive). One of the two copies suffers from data corruption (e.g. some bits flipped), the other one is ok. Will the fsck be able to detect (and correct) that?

[gboudreau]

To verify the files' checksums, you'd simply add the --checksums option when launching your fsck:

  -k, --checksums       Read ALL files in your storage pool, and check that file
                        copies are identical. This will identify any problem you
                        might have with your file-systems.
                        NOTE: this can take a LONG time to complete, since it
                        will read everything from all your drives!

Running fsck without this option will NOT check files' checksums. It would only fix problems with metadata, or missing file copies or symlinks, etc.

[michaepr]

Ah I didn't see that one, sorry. When I ran the command a couple of hours ago, the file /usr/share/greyhole/fsck_checksums.log was not created. I'm assuming that means that no errors were found (?). What happens when there are errors? Does GH automatically replace the file whose checksum doesn't match with another copy of the file where checksums do match?

[gboudreau]

Yes, fsck_checksums.log will only contain mismatches.
If you have log_level at DEBUG, you'll see in the greyhole.log file entries like those, for each checked file:

All copies have the same MD5 checksum: $md5

If it can fix it, it will. You need another file copy with the correct checksum, and the correct checksum needs to be known. (The file checksums are calculated when the first file copies are created, and they are saved in your DB, unless you disable this in greyhole.conf.)

[michaepr]

Ok great, thanks!