RStan yields incorrect line numbers in compiler and runtime errors

[WardBrian]

Description:

It has come up several times on the forums that the line numbers reported by RStan are incorrect. The most recent is here:
https://discourse.mc-stan.org/t/stan-error-messages/34820

The reportee (@cdriveraus) provided a script which shows this by inserting a bad line at a known position in the source and showing what the compiler reports:

library(curl)

# URL of the raw file on GitHub
github_raw_url <- "https://raw.githubusercontent.com/cdriveraus/ctsem/master/inst/stan/ctsm.stan"

# Read the file content directly from the URL
stanmodeltext <- readLines(curl(github_raw_url))
line_number <- c()
# Split the code into lines
for(linei in 800:1000){
  # code_lines <- strsplit(stanmodeltext, "\n")[[1]]
  
  # Generate a random line of code
  random_line <- "function(x);"
  
  # Insert the random line at a random position
  insert_position <- linei#sample(length(code_lines), 1)
  modified_code <- append(stanmodeltext, random_line, after = insert_position - 1)
  
  # Join the modified lines back into a single string
  modified_stan_code <- paste(modified_code, collapse = "\n")
  Sys.sleep(.02)
  error_string=try(rstan::stan_model(model_code = modified_stan_code))
  #extract the string " , line xxx' from the error message
  line_number <- c(line_number,as.numeric(gsub(".*line (\\d+).*", "\\1", error_string)))
    
    plot(1:length(line_number),line_number)
    abline(min(line_number),1)
}

I have modified the above code to use both cmdstanr and raw V8 calls to stancjs, and both times have gotten a plot which was a straight line:

Possible causes

Rstan does a lot of preprocessing on files before passing them to stanc3js:

rstan/rstan/rstan/R/stanc.R

Lines 62 to 90 in 50c3c6f

	includes <- grep("^[[:blank:]]*#include ", model_code)
	while(length(includes) > 0) {
	for (i in rev(includes)) {
	header <- sub("^[[:blank:]]*#include[[:blank:]]+", "", model_code[i])
	header <- gsub('\\"', '', header)
	header <- gsub("\\'", '', header)
	header <- sub("<", "", header, fixed = TRUE)
	header <- sub(">", "", header, fixed = TRUE)
	header <- sub("//.*$", "", header)
	header <- sub("/\\*.*$", "", header)
	header <- sub("#.*$", "", header)
	header <- sub("[[:blank:]]*$", "", header)
	files <- file.path(isystem, header)
	existent <- file.exists(files)
	if (any(existent))
	model_code <- append(model_code, values = readLines(files[which(existent)[1]]),
	after = i)
	else model_code <- append(model_code, values = readLines(header), after = 1)
	model_code[i] <- ""
	}
	includes <- grep("^[[:blank:]]*#include ", model_code)
	}
	model_code <- gsub('#include /(.*$)', '#include "\\1"', model_code)
	has_pound <- any(grepl("#", model_code, fixed = TRUE))
	if (has_pound && isFALSE(auto_format)) {
	unprocessed <- tempfile(fileext = ".stan")
	processed <- tempfile(fileext = ".stan")

Oddly enough, the ctsm file which is used in the above example contains no #includes, so I would not expect it to be running.

I believe the same preprocessing is what leads to #1067