Tokens with "pads" can't be converted to dfm

[chainsawriot]

While working #27

require(quanteda); require(quanteda.proximity)
#> Loading required package: quanteda
#> Package version: 3.3.1
#> Unicode version: 14.0
#> ICU version: 70.1
#> Parallel computing: 8 of 8 threads used.
#> See https://quanteda.io for tutorials and examples.
#> Loading required package: quanteda.proximity
toks <- tokens(c("a b c", "A B C D")) |>
    tokens_remove("b", padding = TRUE)
toks
#> Tokens consisting of 2 documents.
#> text1 :
#> [1] "a" ""  "c"
#> 
#> text2 :
#> [1] "A" ""  "C" "D"
toks %>% tokens_proximity("a") %>% dfm()
#> Error in Matrix::sparseMatrix(j = index, p = cumsum(c(1L, lengths(x))) - : 'i' and 'j' must be positive

^{Created on 2023-11-22 with reprex v2.0.2}

[chainsawriot]

pads are assigned position zero, which errs Matrix::sparseMatrix(). In quanteda, it's handled by the C++ code.

require(quanteda); require(quanteda.proximity)
#> Loading required package: quanteda
#> Package version: 3.3.1
#> Unicode version: 14.0
#> ICU version: 70.1
#> Parallel computing: 8 of 8 threads used.
#> See https://quanteda.io for tutorials and examples.
#> Loading required package: quanteda.proximity
toks <- tokens(c("a b c", "A B C D")) |>
    tokens_remove("b", padding = TRUE)
toks
#> Tokens consisting of 2 documents.
#> text1 :
#> [1] "a" ""  "c"
#> 
#> text2 :
#> [1] "A" ""  "C" "D"
toks %>% tokens_proximity("a") -> temp
unlist(unclass(temp), use.names = FALSE)
#> [1] 1 0 2 1 0 2 3
attr(temp, "types")
#> [1] "a" "c" "d"

^{Created on 2023-11-22 with reprex v2.0.2}

A hacky solution is to check whether there are zeros in unlist(unclass(temp), use.names = FALSE); if TRUE, add 1. Add "" to attr(temp, "types"). Unlike how dfm.tokens_xptr() handles remove_padding, we remove the pad column in the dfm.