Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fwrite gzip #3288

Merged
merged 65 commits into from
Apr 23, 2019
Merged
Changes from 1 commit
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
aa848a7
Add gzip support to fwrite
Jan 12, 2019
92b9ca4
Add compress= option in fwrite documentation
Jan 12, 2019
a2d6969
Add tests for fwrite with compress="gzip" option
Jan 12, 2019
6010131
Rewrite test 1658.12
Jan 12, 2019
9f16e31
Add default option in compress
Jan 13, 2019
1ca3454
Adapt fwrite compress option documentation
Jan 13, 2019
70fdf8b
Replace 'default' by 'auto' in fwrite compress option
Jan 13, 2019
a279dae
Tests for gzip compression in fwrite and restore tests 1658.11,12
Jan 13, 2019
c8858e4
\#endif was in wrong place
Jan 13, 2019
42b4964
Remove realloc sections
Jan 13, 2019
81cc49b
Header line is written in buff
Jan 13, 2019
9b575db
Header compression OK
Jan 13, 2019
1375a34
Tmp : fwrite csv doesn't end
Jan 13, 2019
38764d9
self-or for failed
Jan 13, 2019
7586e2f
Add if fail
Jan 13, 2019
72c70ae
Remove #endif
Jan 13, 2019
6dd28c8
Move compress buffer at the right place
Jan 13, 2019
7346875
Compress in thread, only write is ordered
Jan 13, 2019
98d90dc
Remove comment
Jan 13, 2019
9202717
Move bool is_gzip in fwrite.h
Jan 14, 2019
2564197
Add cast and remove unused variables
Jan 14, 2019
9e9de37
Test buffer size et adapt error messages
Jan 14, 2019
ffa26cb
Test buffer size after each line
Jan 14, 2019
8423996
Remove old comment
Jan 14, 2019
c161828
Use strlen to compute maxLineLen in fwrite
Jan 15, 2019
646535b
Compute mexHeaderLen and maxLineLen
Jan 15, 2019
ea68fd0
Compute output length for writeListe with buffer
Jan 15, 2019
7464a9a
Reinsert fwrite progress bar
Jan 15, 2019
e62b77e
Introduce buffLimit and buffSecure in fwrite
Jan 15, 2019
af44e9a
TRUE instead of T in tests.Rraw
Jan 15, 2019
dc2b825
Typo correction
Jan 15, 2019
49ee223
Alloc zbuff only if args.gzip
Jan 15, 2019
9200e8e
Test if string/factor is NA is line header length detection
Jan 15, 2019
5b93d91
Correct zbuff allocation
Jan 15, 2019
7ccc3e6
Add missing is_gzip test
Jan 15, 2019
47d2210
Use directly maxLineLen ; don't divide by 2
Jan 15, 2019
01f3db8
Initialize buffer pointer to NULL
Jan 16, 2019
e2a97ff
Add news entry for gzip support in fwrite
Jan 16, 2019
c24c154
Move enum WF in fwrite.h
Jan 16, 2019
cd8768b
Replace integer by WF enum
Jan 16, 2019
33ad794
Test size before writing in thread
Jan 17, 2019
739ac7e
Replace strlen by strnlen
Jan 20, 2019
f7f37e4
Use void* and size_t instead of Bytef* and uLongf
Jan 24, 2019
fa7d6c0
Merge branch 'master' into fwrite_gzip
philippechataignon Feb 9, 2019
07c0f6f
Merge branch 'master' into fwrite_gzip
philippechataignon Feb 22, 2019
3d37bad
Merge branch 'master' into fwrite_gzip
mattdowle Apr 16, 2019
46fd237
attempt to pass on travis/appveyor: added -lz to PKG_LIBS
mattdowle Apr 16, 2019
0e10df9
fixed fwrite.Rd warning (treated as fail by CI)
mattdowle Apr 16, 2019
031ebd0
news tidy
mattdowle Apr 17, 2019
c32249a
nocov and 2-space indentation
mattdowle Apr 17, 2019
1189568
removed not-used writer_len item in fwriteMainArgs
mattdowle Apr 17, 2019
25ffdb0
extra 1 width for sign of negative ints
mattdowle Apr 17, 2019
3c097ae
declare variables close to first usage, and added comment as to why u…
mattdowle Apr 18, 2019
c2efc8e
reduce diff; free(NULL) is no-op ok (removed if)
mattdowle Apr 18, 2019
01833e2
more diff reduction
mattdowle Apr 19, 2019
6a7d87b
free's were missing on STOP('compress gzip error'). Moved zbuffUsed d…
mattdowle Apr 19, 2019
42711f2
interm: no sample for maxLineLen, reorganized
mattdowle Apr 20, 2019
034149b
hot loop hot again; passes all tests
mattdowle Apr 23, 2019
96d6230
revert unrelated indentation fix to reduce diff (will do post-merge)
mattdowle Apr 23, 2019
a593abd
coverage
mattdowle Apr 23, 2019
8482bd1
compress tests made cross-platform, and example added to news item
mattdowle Apr 23, 2019
42ceb61
simpler example in news item
mattdowle Apr 23, 2019
ee3e595
tidy
mattdowle Apr 23, 2019
989740c
news item for bug fixes with links
mattdowle Apr 23, 2019
7665a8d
coverage
mattdowle Apr 23, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Test buffer size after each line
  • Loading branch information
Philippe Chataignon committed Jan 14, 2019
commit ffa26cb2400750d17156d4391266372944af4794
12 changes: 5 additions & 7 deletions src/fwrite.c
Original file line number Diff line number Diff line change
Expand Up @@ -818,15 +818,13 @@ void fwriteMain(fwriteMainArgs args)
}
// Hot loop
for (int j=0; j<args.ncol; j++) {
//printf("j=%d args.ncol=%d myBuff='%.*s' ch=%p\n", j, args.ncol, 20, myBuff, ch);
(args.funs[args.whichFun[j]])(args.columns[j], i, &ch);
//printf(" j=%d args.ncol=%d myBuff='%.*s' ch=%p\n", j, args.ncol, 20, myBuff, ch);
*ch++ = sep;
//printf(" j=%d args.ncol=%d myBuff='%.*s' ch=%p\n", j, args.ncol, 20, myBuff, ch);
}
// Test if buffer to low
if ( (int)(ch - myBuff) >= buffSize ) {
failed = -1;
// Test if buffer to low
if ( (int)(ch - myBuff) >= buffSize ) {
failed = -1;
break; // stop writing
}
}
// Tepid again (once at the end of each line)
ch--; // backup onto the last sep after the last column. ncol>=1 because 0-columns was caught earlier.
Expand Down