simpleError in .logThis, object 'logFile' not found at end of addIterativeLSI

[markphillippebworth]

Describe the bug
A clear and concise description of what the bug is.

addIterativeLSI runs successfully until the last step, and it spits out the error:

<simpleError in .logThis(sampledCellNames, "cellNames supplied", logFile = logFile): object 'logFile' not found

I've tried deleting the ArchRLogs folder and re-running, but I hit the same error.
The error occurs in ArchR 1.0.1 and 1.0.2.
ArchR-addIterativeLSI-131318064369-Date-2021-03-22_Time-17-03-45.log

To Reproduce
To help us optimally address your issue, please try to reproduce this issue using the tutorial hematopoiesis dataset and provide us the command(s) to reproduce your bug. Our first question to you will be "can you reproduce this with the tutorial dataset" so please do this.

This is the code I ran. I tried with using the standard TileMatrix and a subset FeatureMatrix, same error.
I've tried without sampleCellsFinal, but I hit memory limits. I ran this line about 4-8 weeks ago, and it ran successfully on the same files.

Proj <- addIterativeLSI(
ArchRProj = Proj,
useMatrix = "FeatureMatrix",
name = "IterativeLSI",
iterations = 6,
sampleCellsFinal = 300000,
force = TRUE
)

Session Info
If you do not have a log file because the function that caused the error does not produce one, please paste the output of "sessionInfo()" here.

Additional context
Add any other context about the problem here.

[markphillippebworth]

I tried setting addArchRLogging(useLogs = FALSE), but it spit out this error when I ran addIterativeLSI().

Error in .validInput(input = logFile, name = "logFile", valid = c("character")) :
Input value for 'logFile' is not a character, (logFile = NULL) please supply valid input!

I then tried filLogs <- createLogFile("addIterativeLSI"), but that fails if useLogs = FALSE.

So I set useLogs = TRUE, generated a valid logFile to pass to addIterativeLSI, and then turned logging off.

With that, addIterativeLSI was able to start, but it still hit the same error.

I also tried reducing iterations to 2, same issue.

[markphillippebworth]

I also tried editing our the code to remove all logging steps, but then it's giving me warnings, and doesn't produce an LSI.

Checking Inputs...
Filtering 30 dims correlated > 0.75 to log10(depth + 1)
Warning messages:
1: In outLSI$scaleDims <- scaleDims : Coercing LHS to a list
2: In outLSI$scaleDims <- scaleDims : Coercing LHS to a list

[rcorces]

I have not been able to parse this one out. I cant figure out where the error is coming from. I know that you said that the same code did not produce errors previously. Was this using the same input? Can you reproduce this problem with smaller datasets (especially the tutorial dataset)?

[markphillippebworth]

My guess is that all 30 LSI dims are coming out as correlated to log10(depth +1), which means that nothing is passed to the logFile or something like that.

Which is why when I run iterativeLSI with any log steps, it completes without an error, but not LSI matrix is generated.

I'll run the tutorial data now. I've been using basically the same set of arrowFiles.

[markphillippebworth]

It failed on the tutorial data when I set preojectCellsPre = TRUE, and sampleCellsFinal = 5000.

proj <- addIterativeLSI(
ArchRProj = proj,
name = "IterativeLSI",
iterations = 2,
projectCellsPre = TRUE,
sampleCellsFinal = 5000,
force = TRUE
)

But it was successful without those parameters.

Both projectCellsPre and sampleCellsFinal produced the error independently.

[rcorces]

I am able to reproduce this error with the tutorial data as you've indicated. though I havent figured out where the error is coming from yet. Will try to sort this out soon.

[markphillippebworth]

If it helps, this error was not occurring in January (maybe February?). So it is most likely linked to a recent change in 1.0.1.

[rcorces]

There were a lot of changes to the iterative LSI code in version 1.0.0 (October 28) but not much in 1.0.1. (February 21).
https://github.com/GreenleafLab/ArchR/commits/master/R/IterativeLSI.R

I'll keep looking to see what I can find.

[jgranja24]

Hi @markphillippebworth try installing the newest release. I believe I fixed your issue now for where the 'logFile' are missing --

devtools::install_github("GreenleafLab/ArchR", 
	ref="release_1.0.2", repos = BiocManager::repositories())

[markphillippebworth]

New error:

addIterativeLSI(
ArchRProj = proj,
name = "IterativeLSI",
iterations = 2,
projectCellsPre = TRUE,
sampleCellsFinal = 1000,
force = TRUE
)

<simpleError in .getPartialMatrix(ArrowFiles = ArrowFiles, featureDF = featureDF, useMatrix = useMatrix, cellNa
mes = cellNames, doSampleCells = TRUE, sampledCellNames = sampledCellNames, tmpPath = tmpPath, useIndex = useIn
dex, threads = threads, verbose = FALSE, logFile = logFile): Error not all cellNames found in partialMatrix>

---

Error in .logError(e, fn = ".LSIPartialMatrix", info = "", errorList = errorList, :
Exiting See Error Above

ArchR-addIterativeLSI-11b37d1a372a-Date-2021-04-06_Time-21-27-50.log

[markphillippebworth]

Same error for my data or the tutorial data.

[jgranja24]

Sorry I found the bug --

  if(doSampleCells){

    matFiles <- lapply(mat, function(x) x[[2]]) %>% Reduce("c", .)
    mat <- lapply(mat, function(x) x[[1]]) %>% Reduce("cbind", .)
    if(!all(cellNames %in% colnames(mat))){
      .logThis(sampledCellNames, "cellNames supplied", logFile = logFile)
      .logThis(colnames(mat), "cellNames from matrix", logFile = logFile)
      stop("Error not all cellNames found in partialMatrix")
    }
    mat <- mat[,sampledCellNames, drop = FALSE]
    mat <- .checkSparseMatrix(mat, length(sampledCellNames))

    .logDiffTime("Successfully Created Partial Matrix", tstart, verbose = verbose)

    return(list(mat = mat, matFiles = matFiles))

  }else{

    mat <- Reduce("cbind", mat)
    if(!all(cellNames %in% colnames(mat))){
      .logThis(cellNames, "cellNames supplied", logFile = logFile)
      .logThis(colnames(mat), "cellNames from matrix", logFile = logFile)
      stop("Error not all cellNames found in partialMatrix")
    }
    mat <- mat[,cellNames, drop = FALSE]
    mat <- .checkSparseMatrix(mat, length(cellNames))
    
    .logDiffTime("Successfully Created Partial Matrix", tstart, verbose = verbose)

    return(mat)

  }

is now

if(doSampleCells){

    matFiles <- lapply(mat, function(x) x[[2]]) %>% Reduce("c", .)
    mat <- lapply(mat, function(x) x[[1]]) %>% Reduce("cbind", .)
    if(!all(sampledCellNames %in% colnames(mat))){
      .logThis(sampledCellNames, "cellNames supplied", logFile = logFile)
      .logThis(colnames(mat), "cellNames from matrix", logFile = logFile)
      stop("Error not all cellNames found in partialMatrix")
    }
    mat <- mat[,sampledCellNames, drop = FALSE]
    mat <- .checkSparseMatrix(mat, length(sampledCellNames))

    .logDiffTime("Successfully Created Partial Matrix", tstart, verbose = verbose)

    return(list(mat = mat, matFiles = matFiles))

  }else{

    mat <- Reduce("cbind", mat)
    if(!all(cellNames %in% colnames(mat))){
      .logThis(cellNames, "cellNames supplied", logFile = logFile)
      .logThis(colnames(mat), "cellNames from matrix", logFile = logFile)
      stop("Error not all cellNames found in partialMatrix")
    }
    mat <- mat[,cellNames, drop = FALSE]
    mat <- .checkSparseMatrix(mat, length(cellNames))
    
    .logDiffTime("Successfully Created Partial Matrix", tstart, verbose = verbose)

    return(mat)

  }

[jgranja24]

should work now

devtools::install_github("GreenleafLab/ArchR", 
	ref="release_1.0.2", repos = BiocManager::repositories())