Checksumming: reading of the file happen quickly afer writing or triggered later by user?

[slrslr]

https://btrfs.readthedocs.io/en/latest/Checksumming.html says:

checksum is calculated before write and verifed after reading the blocks from devices.

this needs to be improved IMO.

For example, when there is an error completely writing the file or the writing is completed, but the data are different from the source non-BTRFS drive (due to maybe bad drive controller, port, cable?), is BTRFS automatically reading the file after copying it in order to compare its checksum with the checksum that it calculated out of the source file which is located outside of BTRFS filesystem? Or instead it is NOT automatically reading the BTRFS file after writing it, but it is done later after i need to read this file in future (for example because file on non-BTRFS drive is lost, and my attempt copying the file out of BTRFS fails, because the file checksum is now found to be different from the checksum made from original file that was located on a non-BTRFSD drive?)

Thank you for explaining in advance and sorry for hard to understand english.

FIX: if it can be a bit cleared in linked documentation + there is a typo in word "verifed".

UPDATE: it seems that btrfs is not made to verify that it written the data correctly, per the @Zygo comment:

The checksum is written at the same time as the data, but the checksum is not verified at write time. Verification occurs only when an application reads the data, or a maintenance scrub is run. ... you will have to run an application or script which re-reads and verifies the data

he mentions "sysctl vm.drop_caches=3" may be needed. Regarding improving the docummentation, he also suggested how to do it, even it unfortunately does not explain that the csum verification/data reading is NOT done by btrfs right after it complete writing.

[adam900710]

If you're talking about error handling:

Firstly, if the error happens at data write (can be an immediate error upon bio submission) then the corresponding pages will be marked error, and their csum will not be inserted at all.

And the error will normally be reported to user space, some operations like fsync() or direct IO can properly report the error, but some may not (like sync()).

If initial bio submission is successful, but later the disk/controller screw up the cache sync (syncing the write cache to disk), then it will cause error at barrier time (FUA/FLUSH) command.
In that case, btrfs will forced into RO mode preventing further corruption.

Then I didn't get what you're saying related to the following sentences:

but the data are different from the source non-BTRFS drive

Why btrfs would read some data from non-BTRFS driver? If one device doesn't belong to the btrfs you're reading from, how could that be read by btrfs?

Are you talking about copying data from a non-btrfs fs? In that case, please refer to the source fs code to see how they handle the read.
They may or may not do verification depending on the storage stack, it's really nothing related to btrfs.

Please make yourself clear, or we can not help at all.

[slrslr]

Are you talking about copying data from a non-btrfs fs?

Yes, ofcourse, i am talking as someone who does not understand BTRFS and Linux well. When i talk about "source non-BTRFS drive" i mean system is copying from one drive that is not BTRFS to a BTRFS drive and i was expecting that during the task, BTRFS will at earliest count the csum of source file and then csum of destination BTRFS file after writing it to make sure it is a clone (considering possible drive/cable/port failure or malformed data during the process). The text from documentation i have quoted in 1st post does not make it clear when the csum reading happen (after writing the data) and it is important to know for the reasons i have described in 1st post. And there is that typo in word "verifed". Thank you

[adam900710]

OK, then it becomes much easier to explain, as there is a hidden (and very common basic) knowledge which could apply to all operation systems.

Every modern file system only handles:

1. Read from the filesystem
   The source is the filesystem, the destination is either the user space program (Direct IO), or some kind of cache provided by the OS, then the OS copy the cache to user space program (cached IO).
   And in Linux world, the cache is called page cache.

2. Do extra checks before dirtying the page cache
   The check includes the common ones like ownership/privilege checks, and filesystem specific ones like free space check.

3. Copy the data into page cache and record them for later writeback

4. Write the cache into the disks

(Some small misc operations are skipped intentionally, but it doesn't affect the points I want to make here)

By above potints, you should see why your previous question doesn't make sense to most Linux guys.

If you're copying data from, say a EXT4 fs, to a btrfs fs.

Then the cp command is mostly doing the following thing:

1. Read data from the EXT4 fs
   The destination is the user space buffer of cp program.
   How the data is read is completely determined by EXT4 and its underlying storage stack.
   Btrfs doesn't care nor it should care.

2. Write the newly read data into the Btrfs fs
   The source is the user space buffer of cp program.
   In that case, btrfs firstly checks things like if there is enough space,
   then just copy the data provided by cp program into page cache.

In this case, btrfs is just handling a bunch of data provided by user space.
Btrfs doesn't want to know how the data come from, nor it needs to know, it's just data, nothing more.

So to answer your initial question, if one is reading data from a non-btrfs fs, then it's out of the realm of btrfs.
And obviously all the btrfs documents only need to explain things inside btrfs.

For the typo, thanks for pointing it out, I'll send a patch for it.
Feel free to open an issue for all typos you can find (one issue for all typos) so we can fix it in a batch.

[slrslr]

Thanks,

all the btrfs documents only need to explain things inside btrfs.

yes, I think that this is a thing of a btrfs, because when you read what i have quoted in my 1st post:

checksum is calculated before write and verifed after reading the blocks from devices.

that checksum reading is done by btrfs on btrfs fs, but that sentence is missing information on how long it takes after file is writen to do this reading check of a csum (and if it is actually done automatically). And as i understand it, it is comparing the csum with the one obtained based on original data:

btrfs is just handling a bunch of data provided by user space.

Again, "it is important to know the highlighted information for the reasons i have described in 1st post.", i quote it here:

Or instead it is NOT automatically reading the BTRFS file after writing it, but it is done later after i need to read this file in future (for example because file on non-BTRFS drive is lost, and my attempt copying the file out of BTRFS fails, because the file checksum is now found to be different from the checksum made from original file that was located on a non-BTRFSD drive?)

my understanding: csum is made before write (i assume csum of original data), then verified after writing by reading the file (not mentioned when / if at all happen automatically or triggered after for example one year by user trying to read the file to copy it elsewhere finding the file has different csum than original data had before 1 year)

that sentence does not make it clear and it should so one knows how csum process ensure data against error

[adam900710]

checksum is calculated before write and verifed after reading the blocks from devices.

I still believe this sentence is enough for end uesrs. End users should only need to know that, if they write some data into btrfs, by default the data will be checksumed, and later it will be verified when read from disks.

The timing is not important, and it's indeed a technical detail.

E.g. the data is still in cache, then why you need to bother wasting IO to read the data again and do verification?
In that, that's the whole purpose of Linux page cache, to avoid unnecessary IO.

Furthermore, the original sentence is already containing some detail, "after reading the blocks from devices.".
That's a completely btrfs/VFS internal detail, end users should not bother at all.

[adam900710]

my understanding: csum is made before write (i assume csum of original data), then verified after writing by reading the file (not mentioned when / if at all happen automatically or triggered after for example one year by user trying to read the file to copy it elsewhere finding the file has different csum than original data had before 1 year)

The checksum verification is done automatically, and more importantly, it's completely transparent to the end user.

(I'd be way more surprised otherwise, how do you expect it to be done non-automatically? If non-automatically it shouldn't even be called a feature at all, you can always generate a checksum file by commands like md5sum)

They just read the file as all other filesystems. If the content is correct, then it is no different than any filesystem.
If the content is corrupted, btrfs just throws a -EIO.
End users don't even need to bother anything, it's completely an internal mechanism.

Thus the initial quota is very correct, "verifed after reading the blocks from devices.", which is already an internal tech detail for those who had some basic ideas on Linux and filesystems.

[slrslr]

The checksum verification is done automatically

i guess you mean checksum which is done after writing, aimed to verify that the file is the same as the data btrfs obtained from non-btfs drive "bunch of data provided by user space". But this still does not give idea when this happen. ASAP or after year when i try to read the file.

The timing is not important

It is, because if it is done after year (when i try to read the file myself), it may be too late to be informed that the file csum is not same as original csum done 1 year back before writing.

data is still in cache, then why you need to bother wasting IO to read the data again and do verification?

because of what i have mentioned in 1st post regarding difference between the data obtained from source non-btrfs drive/cache and the data written to btrfs drive. What if that "data are different from the source non-BTRFS drive (due to maybe bad drive controller, port, cable?)" being malformed. User may want to know the drive failed to clone the data as soon as possible and it is not clear from the documentation sentence mentioned.

[Zygo]

Checksums are verified during reads. If you read the file (from disk, not from memory cache), the csums will be verified at that time. Scrubs will also read and verify csums. The checksum is written at the same time as the data, but the checksum is not verified at write time. Verification occurs only when an application reads the data, or a maintenance scrub is run.

Maybe we can break up that first comma-spliced sentence in the docs. I suggest:

Data and metadata are checksummed by default. Checksums are generated from data blocks in memory as the blocks are written to the storage device. Checksums are verified each time the blocks are read into memory from the storage device.

If you are concerned about the integrity of data immediately after making copies, you will have to run an application or script which re-reads and verifies the data immediately after making the copy. A simple version of this would be to run sysctl vm.drop_caches=3, then use cmp or diff to reread the original file and the copy, and compare them with each other to see if any differences were introduced, or if btrfs detects a csum error. Remember that data loss can be introduced at any point in the copying system: 1) the original storage device might fail intermittently and corrupt the data as it is read, especially if the original source filesystem does not have csums, 2) RAM might fail and corrupt data in transit during the copy, or 3) data can be corrupted on the destination storage device at any moment after it is written. btrfs csums can only detect failures in the 3rd location, so you'll need your copying software to check the 1st and 2nd locations.

If you are concerned about media degrading over time, you will need to schedule periodic scrubs to read and verify the data, e.g. with systemd timers or using the btrfsmaintenance package. You will need to monitor the results or set up an alert so that you are informed when devices are failing and data has been lost. You will also need to keep backup copies of the data in case of unrecoverable device failures, failing host hardware, software bugs, fires, theft, ransomware, and so forth (some of those are outside the scope of btrfs).

[slrslr]

@Zygo Yours suggested revision of the https://btrfs.readthedocs.io/en/latest/Checksumming.html seems more detailed, i like it, even it does not explain if btrfs verify checksum right after writing. Per your explanation it does not. I have updated OP with this information. Thank you

[Zygo]

I wrote "Checksums are verified each time the blocks are read into memory from the storage device." That's the complete list of times when checksums are verified. There is no verification after writing.