Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions codegenerator/cli/npm/envio/src/Batch.res
Original file line number Diff line number Diff line change
Expand Up @@ -528,3 +528,23 @@ let findFirstEventBlockNumber = (batch: t, ~chainId) => {
}
result.contents
}

let findLastEventItem = (batch: t, ~chainId) => {
let idx = ref(batch.items->Array.length - 1)
let result = ref(None)
while idx.contents >= 0 && result.contents === None {
let item = batch.items->Array.getUnsafe(idx.contents)
switch item {
| Internal.Event(_) as eventItem => {
let eventItem = eventItem->Internal.castUnsafeEventItem
if eventItem.chain->ChainMap.Chain.toChainId === chainId {
result := Some(eventItem)
} else {
idx := idx.contents - 1
}
}
| Internal.Block(_) => idx := idx.contents - 1
}
}
result.contents
}
3 changes: 2 additions & 1 deletion codegenerator/cli/npm/envio/src/Envio.res
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,8 @@ let experimental_createEffect = (
Internal.effectOutput,
>
),
callsCount: 0,
activeCallsCount: 0,
prevCallStartTimerRef: %raw(`null`),
// This is the way to make the createEffect API
Comment on lines +71 to 73
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

Avoid %raw(null); align with an option-typed ref or a valid default

prevCallStartTimerRef is set via %raw(null) while typed as Hrtime.timeRef. Unsafe and bypasses the type system; may crash if used unguarded.

  • If you adopt option<Hrtime.timeRef> in Internal.res, initialize with None here:
-    activeCallsCount: 0,
-    prevCallStartTimerRef: %raw(`null`),
+    activeCallsCount: 0,
+    prevCallStartTimerRef: None,
  • If keeping a non-optional type, initialize with a valid Hrtime ref (e.g., Hrtime.makeTimer()) and add a boolean flag to indicate “active timing” instead of null.

Based on learnings.


🏁 Script executed:

#!/bin/bash
rg -nP --type=res -C3 'prevCallStartTimerRef'

Length of output: 27



---

🏁 Script executed:

```shell
#!/bin/bash
# Find Envio.res file
fd -name "Envio.res"

Length of output: 230


🏁 Script executed:

#!/bin/bash
# Search for prevCallStartTimerRef without file type constraint
rg -n -C3 "prevCallStartTimerRef" codegenerator/

Length of output: 3874


🏁 Script executed:

#!/bin/bash
# Look for Internal.res file
fd -name "Internal.res" codegenerator/

Length of output: 230



---

🏁 Script executed:

```shell
#!/bin/bash
# Properly find Internal.res file
fd Internal.res codegenerator/

Length of output: 44


🏁 Script executed:

#!/bin/bash
# Find Hrtime type definition
rg -n "type timeRef" codegenerator/

Length of output: 146


🏁 Script executed:

#!/bin/bash
# Check LoadLayer context to see if first read happens on initial creation
rg -n -B10 "prevCallStartTimerRef" codegenerator/cli/templates/static/codegen/src/LoadLayer.res | head -60

Length of output: 1495


🏁 Script executed:

#!/bin/bash
# Search for where effect record is first created to understand initialization order
rg -n "hadActiveCalls" codegenerator/cli/templates/static/codegen/src/LoadLayer.res -B5

Length of output: 464


Avoid %raw(null); align with an option-typed ref or a valid default

The concern is valid. prevCallStartTimerRef is initialized with %raw(\null`)at line 72 in Envio.res while typed as the non-optionalHrtime.timeRef` in Internal.res, which bypasses type safety.

Although current usage in LoadLayer.res guards against reading null on the first call (if hadActiveCalls at line 141), this fragile defensive programming should be replaced with proper typing:

  • Adopt option<Hrtime.timeRef> and initialize with None, or
  • Initialize with a valid Hrtime.timeRef default (e.g., Hrtime.makeTimer())
🤖 Prompt for AI Agents
In codegenerator/cli/npm/envio/src/Envio.res around lines 71–73, replace the
unsafe %raw(`null`) for prevCallStartTimerRef with a properly-typed value:
change prevCallStartTimerRef to option<Hrtime.timeRef> and initialize it to None
(or, if you prefer the alternate approach, initialize with a valid
Hrtime.makeTimer()); then update all places that read prevCallStartTimerRef to
pattern-match on None/Some (or provide a safe default) instead of assuming a
non-null value so you preserve type safety and remove the raw null bypass.

// work without the need for users to call S.schema themselves,
// but simply pass the desired object/tuple/etc.
Expand Down
4 changes: 3 additions & 1 deletion codegenerator/cli/npm/envio/src/Internal.res
Original file line number Diff line number Diff line change
Expand Up @@ -299,7 +299,9 @@ type effect = {
cache: option<effectCacheMeta>,
output: S.t<effectOutput>,
input: S.t<effectInput>,
mutable callsCount: int,
// The number of functions that are currently running.
mutable activeCallsCount: int,
mutable prevCallStartTimerRef: Hrtime.timeRef,
}
Comment on lines +302 to 305
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

prevCallStartTimerRef should be optional; current design invites null-sentinel misuse

Field is typed as Hrtime.timeRef but Envio.res initializes it with raw null. Without an option type, downstream code can crash if it dereferences without a guard.

Recommended change:

-  mutable prevCallStartTimerRef: Hrtime.timeRef,
+  mutable prevCallStartTimerRef: option<Hrtime.timeRef>,

Then initialize with None in Envio.res and handle Some(_) where used. This removes %raw(null) and restores type safety. If keeping non-optional, provide a non-null sentinel from Hrtime (e.g., Hrtime.makeTimer()) and ensure all sites don’t assume null.

Check usages and legacy name:


🏁 Script executed:

#!/bin/bash
# Ensure all uses guard null/option appropriately
rg -nP -C3 'prevCallStartTimerRef' 

# Ensure no stale references to callsCount remain
rg -nP -C2 '\bcallsCount\b'

Length of output: 3874


Change prevCallStartTimerRef to option<Hrtime.timeRef> for type safety.

The field is initialized with %raw(\null`)at Envio.res:72 but typed as non-optionalHrtime.timeRefat Internal.res:304. It's dereferenced directly in LoadLayer.res (lines 144, 174) without option guards. While usage appears guarded byhadActiveCalls`, the non-optional type with null initialization is unsafe and fragile.

Apply the recommended change:

-  mutable prevCallStartTimerRef: Hrtime.timeRef,
+  mutable prevCallStartTimerRef: option<Hrtime.timeRef>,

Update initialization in Envio.res:72 to prevCallStartTimerRef: None, and wrap assignments and dereferences in LoadLayer.res with option patterns (e.g., Some(timerRef)).

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
// The number of functions that are currently running.
mutable activeCallsCount: int,
mutable prevCallStartTimerRef: Hrtime.timeRef,
}
// The number of functions that are currently running.
mutable activeCallsCount: int,
mutable prevCallStartTimerRef: option<Hrtime.timeRef>,
}
🤖 Prompt for AI Agents
In codegenerator/cli/npm/envio/src/Internal.res around lines 302-305, change the
type of prevCallStartTimerRef from Hrtime.timeRef to option<Hrtime.timeRef>;
also update the initializer in codegenerator/cli/npm/envio/src/Envio.res at line
72 to prevCallStartTimerRef: None; then update LoadLayer.res at the call sites
around lines 144 and 174 to wrap assignments with Some(timerRef) when storing a
timer and to pattern-match (e.g., switch/if-let/when Some(timerRef)) before
dereferencing, handling the None case as appropriate.

let cacheTablePrefix = "envio_effect_"
let cacheOutputSchema = S.json(~validate=false)->(Utils.magic: S.t<Js.Json.t> => S.t<effectOutput>)
Expand Down
44 changes: 35 additions & 9 deletions codegenerator/cli/npm/envio/src/Prometheus.res
Original file line number Diff line number Diff line change
Expand Up @@ -580,20 +580,46 @@ module ProgressBatchCount = {
}
}

module ProgressLatency = {
let gauge = SafeGauge.makeOrThrow(
~name="envio_progress_latency",
~help="The latency in milliseconds between the latest processed event creation and the time it was written to storage.",
~labelSchema=chainIdLabelsSchema,
)

let set = (~latencyMs, ~chainId) => {
gauge->SafeGauge.handleInt(~labels=chainId, ~value=latencyMs)
}
}

let effectLabelsSchema = S.object(s => {
s.field("effect", S.string)
})

module EffectCallsCount = {
let gauge = SafeGauge.makeOrThrow(
module EffectCalls = {
let timeCounter = SafeCounter.makeOrThrow(
~name="envio_effect_calls_time",
~help="Processing time taken to call the Effect function. (milliseconds)",
~labelSchema=effectLabelsSchema,
)

let sumTimeCounter = SafeCounter.makeOrThrow(
~name="envio_effect_calls_sum_time",
~help="Cumulative time spent calling the Effect function during the indexing process. (milliseconds)",
~labelSchema=effectLabelsSchema,
)

let totalCallsCount = SafeCounter.makeOrThrow(
~name="envio_effect_calls_count",
~help="The number of calls to the effect. Including both handler execution and cache hits.",
~help="Cumulative number of resolved Effect function calls during the indexing process.",
~labelSchema=effectLabelsSchema,
)

let set = (~callsCount, ~effectName) => {
gauge->SafeGauge.handleInt(~labels=effectName, ~value=callsCount)
}
let activeCallsCount = SafeGauge.makeOrThrow(
~name="envio_effect_active_calls_count",
~help="The number of Effect function calls that are currently running.",
~labelSchema=effectLabelsSchema,
)
}

module EffectCacheCount = {
Expand Down Expand Up @@ -629,8 +655,8 @@ module StorageLoad = {
~labelSchema=operationLabelsSchema,
)

let totalTimeCounter = SafeCounter.makeOrThrow(
~name="envio_storage_load_total_time",
let sumTimeCounter = SafeCounter.makeOrThrow(
~name="envio_storage_load_sum_time",
~help="Cumulative time spent loading data from storage during the indexing process. (milliseconds)",
~labelSchema=operationLabelsSchema,
)
Expand Down Expand Up @@ -686,7 +712,7 @@ module StorageLoad = {
)
operations->Utils.Dict.deleteInPlace(operation)
}
totalTimeCounter->SafeCounter.handleInt(
sumTimeCounter->SafeCounter.handleInt(
~labels={operation},
~value=timerRef->Hrtime.timeSince->Hrtime.toMillis->Hrtime.intFromMillis,
)
Expand Down
9 changes: 1 addition & 8 deletions codegenerator/cli/templates/static/codegen/src/Env.res
Original file line number Diff line number Diff line change
Expand Up @@ -70,14 +70,7 @@ module Benchmark = {
| _ => false
}

let shouldSavePrometheus = self =>
switch self {
| Prometheus => true
// Always save benchmarks in Prometheus in TUI mode
// This is needed for Dev Console profiler
// FIXME: This doesn't take into account the arg from the CLI
| _ => !tuiOffEnvVar
}
let shouldSavePrometheus = _ => true

let shouldSaveData = self => self->shouldSavePrometheus || self->shouldSaveJsonFile
Comment on lines +73 to 75
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

Forcing Prometheus “on” ignores ENVIO_SAVE_BENCHMARK_DATA and breaks config expectations

This unconditionally enables Prometheus saving, making SaveDataStrategy.shouldSaveData always true regardless of user input. Likely unintended and a breaking behavior change for users who disable metrics or only want json-file.

Suggested fix (restore gating by variant and Bool(true)):

-    let shouldSavePrometheus = _ => true
+    let shouldSavePrometheus = self =>
+      switch self {
+      | Prometheus
+      | Bool(true) => true
+      | _ => false
+      }

Optionally, if “always-on Prometheus” is truly desired, please:

  • Rename the strategy or add a separate env (e.g., ENVIO_PROMETHEUS_ENABLED) to make intent explicit.
  • Document the change in release notes.

Run to confirm no other code assumes shouldSaveData can be false:


🏁 Script executed:

#!/bin/bash
rg -nP --type=res -C2 '\bBenchmark\.shouldSaveData\b|SaveDataStrategy\.shouldSaveData'

Length of output: 27


🏁 Script executed:

#!/bin/bash
# Search for shouldSaveData usage in ReScript files
rg -nP -C2 'shouldSaveData' --glob='*.res'

Length of output: 3349


🏁 Script executed:

#!/bin/bash
# Find SaveDataStrategy type definition and its variants
rg -nP -A10 'type.*SaveDataStrategy\b|type t\s*=|^[[:space:]]*(Bool|Prometheus|Json)' codegenerator/cli/templates/static/codegen/src/Env.res | head -50

Length of output: 343


🏁 Script executed:

#!/bin/bash
# Find ENVIO_SAVE_BENCHMARK_DATA references to confirm environment variable mapping
rg -nP 'ENVIO_SAVE_BENCHMARK_DATA|saveDataStrategy' codegenerator/cli/templates/static/codegen/src/Env.res | head -30

Length of output: 202


Forcing Prometheus "on" unconditionally breaks SaveDataStrategy configuration

The change at line 73 forces shouldSavePrometheus = _ => true, which makes line 75's shouldSaveData logic always evaluate to true via short-circuit OR evaluation (true || X = true). This breaks the configuration system—users setting Bool(false) (the default) via ENVIO_SAVE_BENCHMARK_DATA=false have their preference ignored, and benchmarking cannot be disabled. Code at EventProcessing.res:457 and GlobalState.res:372, 896 contains conditional checks implying shouldSaveData should be able to return false.

Restore gating by SaveDataStrategy variant:

-    let shouldSavePrometheus = _ => true
+    let shouldSavePrometheus = self =>
+      switch self {
+      | Prometheus
+      | Bool(true) => true
+      | _ => false
+      }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
let shouldSavePrometheus = _ => true
let shouldSaveData = self => self->shouldSavePrometheus || self->shouldSaveJsonFile
let shouldSavePrometheus = self =>
switch self {
| Prometheus
| Bool(true) => true
| _ => false
}
let shouldSaveData = self => self->shouldSavePrometheus || self->shouldSaveJsonFile
🤖 Prompt for AI Agents
In codegenerator/cli/templates/static/codegen/src/Env.res around lines 73 to 75,
the helper shouldSavePrometheus is hardcoded to always true which forces
shouldSaveData to always be true and ignores
SaveDataStrategy/ENVIO_SAVE_BENCHMARK_DATA; revert this by making
shouldSavePrometheus derive from the SaveDataStrategy variant (or the ENV flag)
instead of returning true unconditionally — implement pattern matching on
self.saveDataStrategy so Prometheus is only enabled for the variant(s) that
require it (and ensure shouldSaveData uses that result).

}
Expand Down
219 changes: 92 additions & 127 deletions codegenerator/cli/templates/static/codegen/src/Index.res
Original file line number Diff line number Diff line change
Expand Up @@ -54,57 +54,7 @@ let stateSchema = S.union([
})),
])

// let setApiTokenEnv = {
// let initialted = ref(None)
// let envPath = NodeJs.Path.resolve([".env"])
// async apiToken => {
// // Execute once even with multiple calls
// if initialted.contents !== Some(apiToken) {
// initialted := Some(apiToken)

// let tokenLine = `ENVIO_API_TOKEN="${apiToken}"`

// try {
// // Check if file exists
// let exists = try {
// await NodeJs.Fs.Promises.access(envPath)
// true
// } catch {
// | _ => false
// }

// if !exists {
// // Create new file if it doesn't exist
// await NodeJs.Fs.Promises.writeFile(
// ~filepath=envPath,
// ~content=tokenLine ++ "\n",
// ~options={encoding: "utf8"},
// )
// } else {
// // Read existing file
// let content = await NodeJs.Fs.Promises.readFile(~filepath=envPath, ~encoding=Utf8)

// // Check if token is already set
// if !Js.String.includes(content, "ENVIO_API_TOKEN=") {
// // Append token line if not present
// await NodeJs.Fs.Promises.appendFile(
// ~filepath=envPath,
// ~content="\n" ++ tokenLine ++ "\n",
// ~options={encoding: "utf8"},
// )
// }
// }
// } catch {
// | Js.Exn.Error(err) => {
// Js.Console.error("Error setting up ENVIO_API_TOKEN to the .env file:")
// Js.Console.error(err)
// }
// }
// }
// }
// }

let startServer = (~getState, ~config: Config.t, ~shouldUseTui as _) => {
let startServer = (~getState, ~config: Config.t, ~consoleBearerToken: option<string>) => {
open Express

let app = makeCjs()
Expand All @@ -117,7 +67,10 @@ let startServer = (~getState, ~config: Config.t, ~shouldUseTui as _) => {
}

res->setHeader("Access-Control-Allow-Methods", "GET, POST, PUT, DELETE, OPTIONS")
res->setHeader("Access-Control-Allow-Headers", "Origin, X-Requested-With, Content-Type, Accept")
res->setHeader(
"Access-Control-Allow-Headers",
"Origin, X-Requested-With, Content-Type, Accept, Authorization",
)

if req.method === Options {
res->sendStatus(200)
Expand All @@ -134,28 +87,36 @@ let startServer = (~getState, ~config: Config.t, ~shouldUseTui as _) => {
res->sendStatus(200)
})

app->get("/console/state", (_req, res) => {
res->json(getState()->S.reverseConvertToJsonOrThrow(stateSchema))
})
let checkIsAuthorizedConsole = req => {
switch consoleBearerToken {
| None => false
| Some(token) =>
switch req.headers->Js.Dict.get("authorization") {
| Some(authorization) if authorization === `Bearer ${token}` => true
| _ => false
}
}
}

app->post("/console/syncCache", (_req, res) => {
(config.persistence->Persistence.getInitializedStorageOrThrow).dumpEffectCache()
->Promise.thenResolve(_ => res->json(Boolean(true)))
->Promise.done
})
app->get("/console/state", (req, res) => {
let state = if req->checkIsAuthorizedConsole {
getState()
} else {
Disabled({})
}

// Keep /console/state exposed, so it can return `disabled` status
// if shouldUseTui {
// app->post("/console/api-token", (req, res) => {
// switch req.query->Utils.Dict.dangerouslyGetNonOption("value") {
// | Some(apiToken) if Some(apiToken) !== Env.envioApiToken =>
// setApiTokenEnv(apiToken)->Promise.done
// | _ => ()
// }
res->json(state->S.reverseConvertToJsonOrThrow(stateSchema))
})

// res->sendStatus(200)
// })
// }
app->post("/console/syncCache", (req, res) => {
if req->checkIsAuthorizedConsole {
(config.persistence->Persistence.getInitializedStorageOrThrow).dumpEffectCache()
->Promise.thenResolve(_ => res->json(Boolean(true)))
->Promise.done
} else {
res->json(Boolean(false))
}
})

PromClient.collectDefaultMetrics()

Expand Down Expand Up @@ -291,63 +252,67 @@ let main = async () => {

startServer(
~config,
~shouldUseTui,
~getState=if shouldUseTui {
() =>
switch gsManagerRef.contents {
| None => Initializing({})
| Some(gsManager) => {
let state = gsManager->GlobalStateManager.getState
let appState = state->makeAppState
Active({
envioVersion,
chains: appState.chains->Js.Array2.map(c => {
let cf = state.chainManager.chainFetchers->ChainMap.get(c.chain)
{
chainId: c.chain->ChainMap.Chain.toChainId->Js.Int.toFloat,
poweredByHyperSync: c.poweredByHyperSync,
latestFetchedBlockNumber: c.latestFetchedBlockNumber,
currentBlockHeight: c.currentBlockHeight,
numBatchesFetched: c.numBatchesFetched,
endBlock: c.endBlock,
firstEventBlockNumber: switch c.progress {
| SearchingForEvents => None
| Syncing({firstEventBlockNumber}) | Synced({firstEventBlockNumber}) =>
Some(firstEventBlockNumber)
},
latestProcessedBlock: switch c.progress {
| SearchingForEvents => None
| Syncing({latestProcessedBlock}) | Synced({latestProcessedBlock}) =>
Some(latestProcessedBlock)
},
timestampCaughtUpToHeadOrEndblock: switch c.progress {
| SearchingForEvents
| Syncing(_) =>
None
| Synced({timestampCaughtUpToHeadOrEndblock}) =>
Some(timestampCaughtUpToHeadOrEndblock)
},
numEventsProcessed: switch c.progress {
| SearchingForEvents => 0
| Syncing({numEventsProcessed})
| Synced({numEventsProcessed}) => numEventsProcessed
},
numAddresses: cf.fetchState->FetchState.numAddresses,
}
}),
indexerStartTime: appState.indexerStartTime,
isPreRegisteringDynamicContracts: false,
rollbackOnReorg: config.historyConfig.rollbackFlag === RollbackOnReorg,
isUnorderedMultichainMode: switch config.multichain {
| Unordered => true
| Ordered => false
},
})
}
}
} else {
() => Disabled({})
~consoleBearerToken={
// The most simple check to verify whether we are running in development mode
// and prevent exposing the console to public, when creating a real deployment.
if Env.Db.password === "testing" {
Some("testing")
} else {
None
}
},
~getState=() =>
switch gsManagerRef.contents {
| None => Initializing({})
| Some(gsManager) => {
let state = gsManager->GlobalStateManager.getState
let appState = state->makeAppState
Active({
envioVersion,
chains: appState.chains->Js.Array2.map(c => {
let cf = state.chainManager.chainFetchers->ChainMap.get(c.chain)
{
chainId: c.chain->ChainMap.Chain.toChainId->Js.Int.toFloat,
poweredByHyperSync: c.poweredByHyperSync,
latestFetchedBlockNumber: c.latestFetchedBlockNumber,
currentBlockHeight: c.currentBlockHeight,
numBatchesFetched: c.numBatchesFetched,
endBlock: c.endBlock,
firstEventBlockNumber: switch c.progress {
| SearchingForEvents => None
| Syncing({firstEventBlockNumber}) | Synced({firstEventBlockNumber}) =>
Some(firstEventBlockNumber)
},
latestProcessedBlock: switch c.progress {
| SearchingForEvents => None
| Syncing({latestProcessedBlock}) | Synced({latestProcessedBlock}) =>
Some(latestProcessedBlock)
},
timestampCaughtUpToHeadOrEndblock: switch c.progress {
| SearchingForEvents
| Syncing(_) =>
None
| Synced({timestampCaughtUpToHeadOrEndblock}) =>
Some(timestampCaughtUpToHeadOrEndblock)
},
numEventsProcessed: switch c.progress {
| SearchingForEvents => 0
| Syncing({numEventsProcessed})
| Synced({numEventsProcessed}) => numEventsProcessed
},
numAddresses: cf.fetchState->FetchState.numAddresses,
}
}),
indexerStartTime: appState.indexerStartTime,
isPreRegisteringDynamicContracts: false,
rollbackOnReorg: config.historyConfig.rollbackFlag === RollbackOnReorg,
isUnorderedMultichainMode: switch config.multichain {
| Unordered => true
| Ordered => false
},
})
}
},
)

await config.persistence->Persistence.init(~chainConfigs=config.chainMap->ChainMap.values)
Expand Down
Loading