kpi exports cannot be opened, says they are corrupt and need to be repaired

[emilykferguson]

https://www.flowdock.com/app/kobotoolbox/support/threads/9wS-xZ9IFAz82NS77EqzlW7Mkqe

[jnm]

(happens on large forms, e.g. the form discussed on flowdock has 3850 submissions and 608 questions)

[jnm]

S3 storage must be used to see the problem. Probably 🔑 output from unzip on the troublesome xlsx:

warning [../../2-IN-Survey-(v3.1.1)-PROD - Swahili - 2018-02-11-05-08.xlsx]:  31457280 extra bytes at beginning or within zipfile

Running diff on the unzipped contents of a file that opens flawlessly in Excel and one that gets rejected due to corruption reveals only a trivial difference in the date:

    john@scrappy:/tmp/comp/unz$ mkdir 2n1
    john@scrappy:/tmp/comp/unz$ mkdir manual
    john@scrappy:/tmp/comp/unz$ cd 2n1/
    john@scrappy:/tmp/comp/unz/2n1$ unzip ../../2-IN-Survey-\(v3.1.1\)-PROD\ -\ Swahili\ -\ 2018-02-11-05-08.xlsx 
    Archive:  ../../2-IN-Survey-(v3.1.1)-PROD - Swahili - 2018-02-11-05-08.xlsx
    warning [../../2-IN-Survey-(v3.1.1)-PROD - Swahili - 2018-02-11-05-08.xlsx]:  31457280 extra bytes at beginning or within zipfile
      (attempting to process anyway)
      inflating: _rels/.rels             
      inflating: docProps/app.xml        
      inflating: docProps/core.xml       
      inflating: xl/theme/theme1.xml     
      inflating: xl/worksheets/sheet1.xml  
      inflating: xl/sharedStrings.xml    
      inflating: xl/styles.xml           
      inflating: xl/workbook.xml         
      inflating: xl/_rels/workbook.xml.rels  
      inflating: [Content_Types].xml     
    john@scrappy:/tmp/comp/unz/2n1$ cd ../manual/
    john@scrappy:/tmp/comp/unz/manual$ unzip ../../manually\ generated.xlsx 
    Archive:  ../../manually generated.xlsx
      inflating: _rels/.rels             
      inflating: docProps/app.xml        
      inflating: docProps/core.xml       
      inflating: xl/theme/theme1.xml     
      inflating: xl/worksheets/sheet1.xml  
      inflating: xl/sharedStrings.xml    
      inflating: xl/styles.xml           
      inflating: xl/workbook.xml         
      inflating: xl/_rels/workbook.xml.rels  
      inflating: [Content_Types].xml     
    john@scrappy:/tmp/comp/unz/manual$ cd ..
    john@scrappy:/tmp/comp/unz$ diff --recursive 2n1/ manual/
    diff --recursive 2n1/docProps/core.xml manual/docProps/core.xml
    1c1
    < <cp:coreProperties xmlns:cp="http://schemas.openxmlformats.org/package/2006/metadata/core-properties"><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">openpyxl</dc:creator><dcterms:created xmlns:dcterms="http://purl.org/dc/terms/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="dcterms:W3CDTF">2018-02-11T04:41:59Z</dcterms:created><dcterms:modified xmlns:dcterms="http://purl.org/dc/terms/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="dcterms:W3CDTF">2018-02-11T04:41:59Z</dcterms:modified></cp:coreProperties>
    \ No newline at end of file
    ---
    > <cp:coreProperties xmlns:cp="http://schemas.openxmlformats.org/package/2006/metadata/core-properties"><dc:creator xmlns:dc="http://purl.org/dc/elements/1.1/">openpyxl</dc:creator><dcterms:created xmlns:dcterms="http://purl.org/dc/terms/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="dcterms:W3CDTF">2018-02-11T04:58:06Z</dcterms:created><dcterms:modified xmlns:dcterms="http://purl.org/dc/terms/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="dcterms:W3CDTF">2018-02-11T04:58:06Z</dcterms:modified></cp:coreProperties>
    \ No newline at end of file
    john@scrappy:/tmp/comp/unz$ 

[jnm]

Possibly caused by jschneier/django-storages#449