Repository navigation
Link or copy consistent target files #390
Description
Activity
Working on it right now.
I have solved this problem. But there seems be other bugs which I'm checking now.
Make sure you pull the latest changes from the "develop" branch.
Yes, I have pulled. Thanks for your reminding.
Report:
I've successfully hard link the newest<digest>.file.txtto thefile.txtnow. However, once testing, there is a problem. Since generating and writing target files is not the same as meta files. When we handle with meta files, we generate the metadata(signable) firstly and then create the temp files and finally taking care of "hard_link" or "copy" for the consistent things. But, I go through the documentations and the demo docs again, I notice there's no specific description of how a repository maintainer change the content of a target file, so I guess he/she should be able to access the target file on the server and change the content directly. That will be a problem according to our design right now. Suppose we set the"Consistent_method"as "hard_link" now, the<digest>.file.txtwill point tofile.txt. Then, the maintainer changes the content offile.txtwhich will also change the content of<digest>.file.txt. After this change, there will be four<digest> filesin the target directory (because we have two hash algorithms), three of them will have the content of the changedfile.txt, while there should only be two of them have.
My suggestion is we should not support hard_link for target files. That will increase the memory usage but avoid the problem. Actually, memory should not be a problem on server side.What if the repository tool first moves
foo.tar.gzto<digest>.foo.tar.gz? By doing so, it can create a hard link fromfoo.tar.gz-><digest>.foo.tar.gz. Iffoo.tar.gzever needs to be replaced by a repository maintainer, its consistent version need not be affected (because the consistent file is the "original"). After a repository maintainer manually replaces a target file, they would use the repository tool again to create a new consistent file and update the target file's entry in metadata.`foo.tar.gz` <-- latest version of `foo.tar.gz` that currently points to `456.foo.tar.gz` `123.foo.tar.gz` <-- previous version of `foo.tar.gz`. `456.foo.tar.gz`The documentation section on target files contains all the information needed to add and replace target files in TUF metadata. A repository maintainer will normally only need to call repository.add_target(), repository.add_targets(), or repository.remove_target() to update metadata. They would also need to manually add the actual target files to the repository (without a digest prepended). Please note that the repository tool does not create the actual target files stored in the
targets/directory.This is what I thought at first. However, by doing so, you have to make sure every repository maintainer must remove
foo.tar.gzfirst and then change the content of the file and then addfoo.tar.gzback to the repository. They cannot modify the content offoo.tar.gzdirectly. Because, it will always change the<digest>.foo.tar.gzeven this hard link is fromfoo.tar.gzto<digest>.tar.gz.Oh, of course. By modifying
foo.tar.gzthe second time, the repository maintainer would also modify the consistent file (see step (3) in the snippet below). I guess just creating a copy for the consistent file is good enough. I don't think we should mandate that a repository maintainer first delete a target file before editing it.import os import shutil with open('foo.txt', 'wb') as file_object: file_object.write('hello') # Let's (1) move 'foo.txt' to '123.foo.txt' (2) create a hard link from # 'foo.txt' -> '123.foo.txt' (3) modify 'foo.txt' again and (4) create a new # hard link from 'foo.txt' -> '456.foo.txt'. # step (1) shutil.move('foo.txt', '123.foo.txt') # step (2) # os.link(source, link_name) os.link('123.foo.txt', 'foo.txt') # step (3) with open('foo.txt', 'wb') as file_object: file_object.write('hello world') # step (4) shutil.move('foo.txt', '456.foo.txt') os.link('456.foo.txt', 'foo.txt') # show contents of '123.foo.txt', '456.foo.txt', and 'foo.txt' print('would be nice if this actually prints "hello": ' + repr(open('123.foo.txt').read())) print('this should print "hello world": ' + repr(open('456.foo.txt').read())) print('this should print "hello world": ' + repr(open('foo.txt').read()))
Yes, that's exactly what I tested. So the conclusion is we only support "copy" for consistent target files?
If
consistent_snapshot= True , I think we should create a copy of the target file and save it to<digest>.foo.tar.gz(for example). The code currently tries to create a hard link. We'd also not provide the option to specify whether a hard link or copy is made.- added a commit that references this issue
on Nov 8, 2016 Issue has now been fixed!
write_metadata_file() was modifed in PRs #379 and #383 to correctly link or copy consistent metadata. However, consistent target files are not presently written in the same manner. We should verify that generate_targets_metadata() matches the approach used in write_metadata_file().
See the following lines that must be updated: https://github.com/theupdateframework/tuf/blob/develop/tuf/repository_lib.py#L1655-L1666
@FelixWang1994 would be a good person to work on this issue.