sync first lists, then excludes when it should be the other way around

[sauloal]

Hello,

As the script is now, it first list all the files in the local folder then apply the exclude rules.

I think it should be the other way around. My reasoning is: .gvfs, /mnt and other mounting points.

With the current setting s3cmd will read all the mountings (and to my davs, sshfs and ftp, this is a real pain). If I want to exclude, I don't want it to be read.

Kind regards, thanks and great program.

[mludvig]

Hi,

On 15/08/11 12:37, sauloal wrote:

As the script is now, it first list all the files in the local folder then apply the exclude rules.

I think it should be the other way around. My reasoning is: .gvfs, /mnt and other mounting points.

With the current setting s3cmd will read all the mountings (and to my davs, sshfs and ftp, this is a real pain). If I want to exclude, I don't want it to be read.

That's quite a valid argument indeed. At present the --exclude list is
meant to be a "don't upload" list not a "don't read" list.

I'll keep the request in my TODO list.

Thanks

Michal

[sauloal]

as a counterpoint,
it would be more interesting to add a --skip-folder option.
--exclude has the option of some files being put back by --include, this is interesting also

regards,

[mdomsch]

I wrote a nasty hack today to accomplish this. I'll see about cleaning it up some. Basically, I want to take the exclude list (I'm not using regexps or globs, just plain text at this point), and because my list is all directories, simply filter out of the dirs list returned by the _fswalk() call filelist, as it walks. Here's the ugly patch:

diff --git a/S3/FileLists.py b/S3/FileLists.py
index 20e0227..896fd8a 100644
--- a/S3/FileLists.py
+++ b/S3/FileLists.py
@@ -81,6 +81,62 @@ def filter_exclude_include(src_list):
debug(u"PASS: %s" % (file))
return src_list, exclude_list

+always_exclude = [
+u'/.snapshot',
+u'/source',
+u'/SRPMS',
+u'/debug',
+u'/beta',
+u'/ppc',
+u'/ppc64',
+u'/repoview',
+u'/Fedora',
+u'/Live',
+u'/isolinux',
+u'/images',
+u'/EFI',
+u'/drpms',
+u'/core',
+u'/extras',
+u'/LiveOS',
+u'/releases/test'
+u'/releases/8',
+u'/releases/9',
+u'/releases/10',
+u'/releases/11',
+u'/releases/12',
+u'/releases/13',
+u'/releases/14',
+u'/releases/15',
+u'/updates/8',
+u'/updates/9',
+u'/updates/10',
+u'/updates/11',
+u'/updates/12',
+u'/updates/13',
+u'/updates/14',
+u'/updates/15',
+u'/updates/testing/8',
+u'/updates/testing/9',
+u'/updates/testing/10',
+u'/updates/testing/11',
+u'/updates/testing/12',
+u'/updates/testing/13',
+u'/updates/testing/14',
+u'/updates/testing/15',
+]
+
+import copy
+def exclude_dirs(root, dirs):

• copydirs = copy.copy(dirs)
• for d in copydirs:

•    testdir = os.path.join(root, d)
  

•   for a in always_exclude:
  

•       if a in testdir:
  

•           print "Excluding %s by pattern %s" % (testdir, a)
  

•           dirs.remove(d)
  

•           break
  

  def fetch_local_list(args, recursive = None):
  def _get_filelist_local(local_uri):
  info(u"Compiling list of local files...")
  @@ -96,6 +152,7 @@ def fetch_local_list(args, recursive = None):
  single_file = True
  loc_list = SortedDict(ignore_case = False)
  for root, dirs, files in filelist:

•       exclude_dirs(root, dirs)
       rel_root = root.replace(local_path, local_base, 1)
       for f in files:
           full_name = os.path.join(root, f)
  

[mdomsch]

FWIW, this results is a >20x speedup for me, reducing the number of files to consider transferring from ~700k down to ~35k by pruning the os.walk() at higher points in the tree. Saves tens of minutes and a ton of disk I/O, all those extra readdir() and stat() calls, and conversion of filenames to unicode...

[mludvig]

That makes sense indeed. Just the patch in its current form is not ready for the prime time yet ;)

[mdomsch]

See pull request I filed.

[mdomsch]

This is in master branch now, version 1.5.0-alpha1 and later.