Fix memory usage for polymorphic types. #7049

kwxm · 2025-04-17T01:21:25Z

Fixes https://github.com/IntersectMBO/plutus-private/issues/1498

While costing the array builtins it appeared that the time taken by listToArray differed for different types of list elements. What was actually happening was that the list was wrapped in a ListCostedByLength wrapper but this was ignored when the denotation was executed, so the default size measure was used instead, and this measured the total memory footprint of the list. So two lists of the same length could have different "sizes" and when working out the dependency of time on size you would get different slopes for lists with elements of different types.

This fixes that problem (which should only happen when SomeConstant is involved, which at present only happens for polymorphic types) by changing the default memory usage for lists and arrays to be the length, and the memory usage for pairs to be equal to 1. This is arguably the correct thing to do because polymorphic builtins should be parametrically polymorphic, and the time taken by such functions should not depend on the contents of polymorhpic arguments (although if you really want to you can do weird things with the builtin machinery, so it may be possible to produce counterexamples to this) . The polymorphic builtins which have previously been deployed on the chain all have constant time and memory usage, so this change will not affect any existing code. The only function which this will affect is listToArray, which has not yet been deployed.

This also has the benefit of making a number of Agda conformance tests pass which did not pass previously, because the Agda code has the default cost measures hard-coded and doesn't know about newtype wrappers for alternative size measures. Since we now only have one size measure for the polymorphic types this problem has now gone away for polymorphic functions, although it still persists for a few non-polymorphic functions.

This may still be a bit fragile: see https://github.com/IntersectMBO/plutus-private/issues/1046 for a possible longer-term solution.

kwxm · 2025-04-17T01:23:32Z

plutus-core/plutus-core/src/PlutusCore/Evaluation/Machine/ExMemoryUsage.hs

+ for polymorphic types].
+-}
+
+{- Note [ExMemoryUsage for polymorphic types].  For polymorphic types such as


@effectfully I'm not confident that this comment is completely accurate, so feel free to make corrections.

Maybe we should say something about this in Builtins.hs as well, although there's already hundreds of lines of comments in there.

Great comment.

Yeah, maybe Builtins needs to talk about costing too, but we can do that separately.

plutus-core/plutus-core/src/PlutusCore/Evaluation/Machine/ExMemoryUsage.hs

Unisay · 2025-04-17T09:03:59Z

plutus-core/plutus-core/src/PlutusCore/Evaluation/Machine/ExMemoryUsage.hs

+    memoryUsage _ = singletonRose 1
+    {-# INLINE memoryUsage #-}


The indentation is inconsistent across these instances: some use 4 spaces, some 3.
I suggest raking the whole file with fourmolu

Suggested change

memoryUsage _ = singletonRose 1

{-# INLINE memoryUsage #-}

memoryUsage _ = singletonRose 1

{-# INLINE memoryUsage #-}

I suggest raking the whole file with fourmolu

I'll do that before merging. It changes almost every line though, so I won't do it right now.

Unisay · 2025-04-17T09:09:05Z

plutus-metatheory/src/Cost/Size.lagda.md

-      1 + defaultConstantMeasure (tmCon t x)
-      + defaultConstantMeasure (tmCon u y)
+defaultConstantMeasure (tmCon (list t) l) = Utils.length l
+defaultConstantMeasure (tmCon (pair t u) (x , y)) = 1


I am wondering why pair has cost 1 a not, say, 2? What logic is applied here?

What logic is applied here?

No logic. Nothing we ever do with pairs should depend on the size (famous last words), so it shouldn't matter what it is.

It may, if add a builtin serializing a list of pairs or whatever.

But for the time being let's just make it maxBound or whatever instead of 1? So that if our assumption that we'll never need a builtin where that matters gets violated, we'll notice that.

It may, if add a builtin serializing a list of pairs or whatever.

I don't think that we can do that polymorphically unless we assume that everything in the universe of types is serialisable. We could write a function that would serialise (integer,bytestring) pairs or something but that would be monomorphic and I think that in that case it'd be ok to use a wrapper.

Unisay · 2025-04-17T09:10:02Z

plutus-core/plutus-core/src/PlutusCore/Evaluation/Machine/ExMemoryUsage.hs

+    {-# INLINE memoryUsage #-}
+
+instance ExMemoryUsage [a] where
+  memoryUsage l = singletonRose . fromIntegral $ length l


Suggested change

memoryUsage l = singletonRose . fromIntegral $ length l

memoryUsage = singletonRose . fromIntegral . length

For some reason I thought that we weren't doing that in this file, but maybe I'm just imagining that.

Unisay · 2025-04-17T09:10:32Z

plutus-core/plutus-core/src/PlutusCore/Evaluation/Machine/ExMemoryUsage.hs

+  {-# INLINE memoryUsage #-}
+
+instance ExMemoryUsage (Vector a) where
+    memoryUsage l = singletonRose . fromIntegral $ Vector.length l


Suggested change

memoryUsage l = singletonRose . fromIntegral $ Vector.length l

memoryUsage = singletonRose . fromIntegral . Vector.length

Unisay

Looks good! (modulo formatting/typos).

effectfully

Great stuff.

plutus-core/plutus-core/src/PlutusCore/Evaluation/Machine/ExMemoryUsage.hs

effectfully · 2025-04-17T20:54:06Z

plutus-core/plutus-core/src/PlutusCore/Evaluation/Machine/ExMemoryUsage.hs

+ for polymorphic types].
+-}
+
+{- Note [ExMemoryUsage for polymorphic types].  For polymorphic types such as


Great comment.

Yeah, maybe Builtins needs to talk about costing too, but we can do that separately.

plutus-core/plutus-core/src/PlutusCore/Evaluation/Machine/ExMemoryUsage.hs

effectfully · 2025-04-17T21:02:24Z

plutus-core/plutus-core/src/PlutusCore/Evaluation/Machine/ExMemoryUsage.hs

+{- A naive traversal for size.  This accounts for the number of nodes in a Data
+   object and also the sizes of the contents of the nodes.  This is not ideal,
+   but it seems to be the best we can do.  At present this only comes into play
+   for 'equalsData', which is implemented using the derived implementation of


Not just equalsData, but also serialiseData?

effectfully · 2025-04-17T21:07:10Z

plutus-metatheory/src/Cost/Size.lagda.md

-      1 + defaultConstantMeasure (tmCon t x)
-      + defaultConstantMeasure (tmCon u y)
+defaultConstantMeasure (tmCon (list t) l) = Utils.length l
+defaultConstantMeasure (tmCon (pair t u) (x , y)) = 1


It may, if add a builtin serializing a list of pairs or whatever.

But for the time being let's just make it maxBound or whatever instead of 1? So that if our assumption that we'll never need a builtin where that matters gets violated, we'll notice that.

zliu41 · 2025-04-21T20:07:31Z

What was actually happening was that the list was wrapped in a ListCostedByLength wrapper but this was ignored when the denotation was executed

Why was it ignored- is it a bug and is it fixed in this PR? I'm asking because there are other newtype wrappers like NumBytesCostedAsNumWords and we don't want those to be ignored.

kwxm · 2025-04-22T12:53:16Z

Why was it ignored- is it a bug and is it fixed in this PR? I'm asking because there are other newtype wrappers like NumBytesCostedAsNumWords and we don't want those to be ignored.

@effectfully will be able to fill in the details, but my understanding is that this only happens for wrappers around polymorphic types: the code that converts those back into Haskell types is unaware of the wrappers and so it always uses the default size measure. For monomorphic types like integers, bytestrings, etc, the wrappers are recognised and the correct size measure is used. This hasn't shown up before because all of the functions that we have that operate on polymorphic types were constant time, unlike listToArray.

kwxm added 3 commits April 13, 2025 23:24

Make length the default size measure for lists and arrays

9d5cc86

Add some comments

65406d9

Fix Agda memory usage for pairs

6a72bb6

kwxm added Builtins Costing Anything relating to costs, fees, gas, etc. No Changelog Required Add this to skip the Changelog Check labels Apr 17, 2025

kwxm requested review from Unisay, zliu41 and effectfully April 17, 2025 01:21

kwxm temporarily deployed to github-pages April 17, 2025 01:21 — with GitHub Actions Inactive

kwxm commented Apr 17, 2025

View reviewed changes

Unisay reviewed Apr 17, 2025

View reviewed changes

plutus-core/plutus-core/src/PlutusCore/Evaluation/Machine/ExMemoryUsage.hs Outdated Show resolved Hide resolved

Unisay reviewed Apr 17, 2025

View reviewed changes

plutus-core/plutus-core/src/PlutusCore/Evaluation/Machine/ExMemoryUsage.hs Outdated Show resolved Hide resolved

Unisay reviewed Apr 17, 2025

View reviewed changes

Unisay approved these changes Apr 17, 2025

View reviewed changes

Fix some typos

dd15de3

kwxm temporarily deployed to github-pages April 17, 2025 11:36 — with GitHub Actions Inactive

effectfully approved these changes Apr 17, 2025

View reviewed changes

More typos

c5110a8

kwxm temporarily deployed to github-pages April 17, 2025 22:05 — with GitHub Actions Inactive

kwxm added 2 commits April 23, 2025 19:12

Address review comments

30fe50c

Merge branch 'master' into kwxm/costing/fix-list-memusage

eea2450

kwxm temporarily deployed to github-pages April 23, 2025 18:18 — with GitHub Actions Inactive

kwxm enabled auto-merge (squash) April 23, 2025 18:21

kwxm merged commit 04c0cf0 into master Apr 23, 2025
7 checks passed

kwxm deleted the kwxm/costing/fix-list-memusage branch April 23, 2025 19:56

	memoryUsage l = singletonRose . fromIntegral $ length l
	memoryUsage = singletonRose . fromIntegral . length

Fix memory usage for polymorphic types. #7049

Fix memory usage for polymorphic types. #7049

Uh oh!

Conversation

kwxm commented Apr 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kwxm Apr 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kwxm Apr 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kwxm Apr 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Unisay left a comment

Choose a reason for hiding this comment

Uh oh!

effectfully left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zliu41 commented Apr 21, 2025

Uh oh!

kwxm commented Apr 22, 2025

Uh oh!

Uh oh!

Uh oh!

kwxm commented Apr 17, 2025 •

edited

Loading

kwxm Apr 17, 2025 •

edited

Loading

kwxm Apr 17, 2025 •

edited

Loading

kwxm Apr 17, 2025 •

edited

Loading