Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a LayoutResolver to compute layout from VM types #11529

Merged
merged 1 commit into from
May 10, 2023
Merged

Conversation

dariorussi
Copy link
Contributor

@dariorussi dariorussi commented Apr 30, 2023

Description

  • Move expensive_check_sui_conservation to node init
  • Expose a layout resolver based on a VM session
  • Decouple MoveResolver traits from Storage traits

In some of the places where we want to use the layout resolver, we
don't have access to all the information to implement StorageView,
but we only need to use the BackingPackageStore.

To support these cases, we need to provide a state view to
LinkageView that can serve packages, and then shim the Module and
Resource resolution functions.

This requires that LinkageView no longer accept a reference
parameter, but a value (so we can inject state into it) -- similar to
the trick we pulled to allow Session to own a LinkageView. Then
implementors (like TemporaryStore) that only need to be passed by
reference can be passed by explicit reference,

i.e. LinkageView<'state, TemporaryStore> becomes
LinkageView<&'state TemporaryStore>.

However, we cannot impose a &'state S: StorageView bound, because
some APIs in StorageView require a mutable reference, so we needed
to decouple the resolution traits (now SuiResolver) and the storage
traits, to impose twin constraints:

fn ...<'state, S>
where
  S: StorageView
  &'state S: SuiResolver
  • Logging improvements
  • Bring invariant_violation! macro into sui-types.
  • Create a variant of it -- make_invariant_violation! -- that
    creates the error, but does not wrap it in a result of return it.
  • Add format! string support to these macros
  • Use this macro for invariant violations within conservation checking
    with extra context on what is failing (what stage, what type is
    involved)
  • Enable telemetry_subscription in the move_package_upgrade_tests to
    see the logs from authorities during tests.
  • (Temporary) Print the IDs of published/upgraded packages in the
    failing test, for context.

Test Plan

Existing tests


If your changes are not user-facing and not a breaking change, you can skip the following section. Otherwise, please indicate what changed, and then add to the Release Notes section as highlighted during the release process.

Type of Change (Check all that apply)

  • user-visible impact
  • breaking change for a client SDKs
  • breaking change for FNs (FN binary must upgrade)
  • breaking change for validators or node operators (must upgrade binaries)
  • breaking change for on-chain data layout
  • necessitate either a data wipe or data migration

Release notes

@vercel
Copy link

vercel bot commented Apr 30, 2023

The latest updates on your projects. Learn more about Vercel for Git ↗︎

4 Ignored Deployments
Name Status Preview Comments Updated (UTC)
explorer ⬜️ Ignored (Inspect) May 10, 2023 10:14pm
explorer-storybook ⬜️ Ignored (Inspect) May 10, 2023 10:14pm
sui-wallet-kit ⬜️ Ignored (Inspect) May 10, 2023 10:14pm
wallet-adapter ⬜️ Ignored (Inspect) May 10, 2023 10:14pm

@dariorussi dariorussi marked this pull request as ready for review April 30, 2023 22:41
@dariorussi dariorussi requested review from lxfind and amnn April 30, 2023 22:41
// an epoch and the SUI conservation check will fail. This also initialize
// the expected_network_sui_amount table.
store
.expensive_check_sui_conservation()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand this change. As in the comment, this is only safe to call at genesis (i.e. when the db table is empty). If it's not genesis, it could be in the middle of an epoch and it does not guarantee conservation at all.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a way to check for genesis at this point, or that the DB was otherwise opened? @dariorussi, the thing we missed when looking at the code yesterday is that within AuthorityStore::open_inner this call is guarded by a check that the database was previously empty, implying that the node is starting from scratch, and we will need some way to recreate that check at the new location:

// Only initialize an empty database.
if store
.database_is_empty()
.expect("Database read should not fail at init.")
{

The reason we need to make this change is that the new implementation of the check will require access to the VM which is created as part of the AuthorityPerEpochStore, (which itself is created after the AuthorityStore::open that used to perform this check).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let me look at this again then, I guess it has to be protected...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@amnn I noticed :) and looking into how to workaround that.
@lxfind I think I can move the creation of the AuthorityPerpetualTables out of AuthorityStore and do the check there (before calling into AuthorityStore) and protect the check that way for both prod and testing.
I'll have code up soon, but the question @amnn asked is fair, if there was a simpler way

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lxfind, Can we rely on the fact that the checks themselves use their own DB and initialise it if it's empty?

if let Some(expected_imbalance) = self
.perpetual_tables
.expected_storage_fund_imbalance
.get(&())
.expect("DB read cannot fail")
{
fp_ensure!(
imbalance == expected_imbalance,
SuiError::from(
format!(
"Inconsistent state detected at epoch {}: total storage rebate: {}, storage fund balance: {}, expected imbalance: {}",
system_state.epoch, total_storage_rebate, storage_fund_balance, expected_imbalance
).as_str()
)
);
} else {
self.perpetual_tables
.expected_storage_fund_imbalance
.insert(&(), &imbalance)
.expect("DB write cannot fail");
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lxfind, Can we rely on the fact that the checks themselves use their own DB and initialise it if it's empty?

Not sure exactly what you meant.
But the earlier proposal of creating tables outside sounds reasonable thing to try

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I meant "table" there instead of "DB". I was referring to the fact that within the expensive check, there is a conditional (linked above) that checks whether there are already entries for expected_storage_fund_imbalance (and expected_network_sui_amount) in the perpetual tables. If there are, then it does a conservation check, and if there are not, it initialises those tables.

If it's not genesis, it could be in the middle of an epoch and it does not guarantee conservation at all.

This bit still doesn't make sense to me: How do we lose the conservation guarantee, assuming that even if the node is started mid-epoch, we initialise its conservation-related tables before it starts processing any transactions?

I'm assuming here that as long as we don't have any transactions in flight, we would expect the live object set at any time during an epoch to conserve SUI as well.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The missing pieces are the gas rewards that's accumulated in Rust/db, not on-chain.
So if we do conservation check in the middle of an epoch, it's going to be very tricky to figure out how much gas reward that has accumulated so far.

@amnn
Copy link
Member

amnn commented Apr 30, 2023

(The change to introduce the new Session API is otherwise good though)

@lxfind
Copy link
Contributor

lxfind commented May 1, 2023

I assume we cannot just create a new VM on-the-fly for the conservation check?

@lxfind
Copy link
Contributor

lxfind commented May 1, 2023

By the way, the primary conservation check is in fact called during reconfiguration. The one at genesis is just to initialize it. We will need to deal with both if we need to make changes.

@@ -162,6 +175,16 @@ impl<'a> TestAuthorityBuilder<'a> {
&ExpensiveSafetyCheckConfig::default(),
);

if empty_database {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am assuming we want this in testing as well?
is this too much of a change?

@@ -248,6 +255,16 @@ impl SuiNode {
&config.expensive_safety_check_config,
);

if empty_database {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reasonable?

@dariorussi
Copy link
Contributor Author

By the way, the primary conservation check is in fact called during reconfiguration. The one at genesis is just to initialize it. We will need to deal with both if we need to make changes.

correct and the one in reconfiguration is in authority so with "easy" access to the VM.
The call while creating the AuthorityStore is the "complicated" one

@dariorussi
Copy link
Contributor Author

made a change to create perpetual tables outside the AuthorityStore, let's see if it seems reasonable.
Thanks!

@amnn
Copy link
Member

amnn commented May 1, 2023

I assume we cannot just create a new VM on-the-fly for the conservation check?

We can do this, but then it may involve doing a lot of extra work to load the relevant packages into the VM during the conservation check that we would otherwise avoid by using the same VM.

@dariorussi
Copy link
Contributor Author

I assume we cannot just create a new VM on-the-fly for the conservation check?

We can do this, but then it may involve doing a lot of extra work to load the relevant packages into the VM during the conservation check that we would otherwise avoid by using the same VM.

right, as @amnn said we thought it was a bit too much and it was nice to use the same VM as what is created for the authority state.
It also played a role that the solution we are implementing is a "temporary" solution until we review the layout story entirely

@lxfind
Copy link
Contributor

lxfind commented May 1, 2023

We can do this, but then it may involve doing a lot of extra work to load the relevant packages into the VM during the conservation check that we would otherwise avoid by using the same VM.

It uses an internal module cache so it might not be too bad

@amnn
Copy link
Member

amnn commented May 1, 2023

It does, but it will entirely replicate that inside the VM, multiple times over.

@lxfind lxfind requested a review from mystenmark May 1, 2023 00:48
Copy link
Contributor

@lxfind lxfind left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks reasonable change to me. Now that we have empty_table outside there, I wonder if it could be used to simplify other things (not sure, but worth a quick check)

&config.db_path().join("store"),
Some(perpetual_options.options),
));
let empty_database = perpetual_tables
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe leave a comment here saying that empty database indicates genesis

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or maybe just rename the variable to is_genesis?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@dariorussi
Copy link
Contributor Author

Looks reasonable change to me. Now that we have empty_table outside there, I wonder if it could be used to simplify other things (not sure, but worth a quick check)

let's chat tomorrow about it as I keep going with the implementation of the overall change.
We will also take a look at what else could be simplified

@dariorussi dariorussi force-pushed the upgrade_type3 branch 4 times, most recently from 975ba60 to 0046346 Compare May 2, 2023 15:38
Copy link
Member

@amnn amnn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good so far, just reviewing along the way!

Comment on lines 55 to 65
let ty = match load_type(&mut self.session, &type_tag) {
Err(_) => {
return Err(SuiError::FailObjectLayout {
st: format!("{}", struct_tag),
})
}
Ok(ty) => ty,
};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: let ... else, although, is it worth including the error as well?

Suggested change
let ty = match load_type(&mut self.session, &type_tag) {
Err(_) => {
return Err(SuiError::FailObjectLayout {
st: format!("{}", struct_tag),
})
}
Ok(ty) => ty,
};
let Ok(ty) = load_type(&mut self.session, &type_tag) else {
return Err(SuiError::FailObjectLayout { st: format!("{}", struct_tag) });
};

Comment on lines 68 to 78
match layout {
Err(_) => Err(SuiError::FailObjectLayout {
st: format!("{}", struct_tag),
}),
Ok(type_layout) => match type_layout {
MoveTypeLayout::Struct(layout) => Ok(layout),
_ => Err(SuiError::FailObjectLayout {
st: format!("{}", struct_tag),
}),
},
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit, same deal:

Suggested change
match layout {
Err(_) => Err(SuiError::FailObjectLayout {
st: format!("{}", struct_tag),
}),
Ok(type_layout) => match type_layout {
MoveTypeLayout::Struct(layout) => Ok(layout),
_ => Err(SuiError::FailObjectLayout {
st: format!("{}", struct_tag),
}),
},
}
let Ok(MoveTypeLayout::Struct(layout)) = layout else {
return Err(SuiError::FailObjectLayout {
st: format!("{}", struct_tag),
})
}

&config.db_path().join("store"),
Some(perpetual_options.options),
));
let empty_database = perpetual_tables
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or maybe just rename the variable to is_genesis?

pub trait LayoutResolver {
fn get_layout(
&mut self,
object: &MoveObject,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Should this accept the full object, or just its type?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure maybe both API in time?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

leaving it as is for now

@@ -1494,7 +1512,11 @@ impl<S: ChildObjectResolver> Storage for TemporaryStore<S> {

impl<S: BackingPackageStore> BackingPackageStore for TemporaryStore<S> {
fn get_package_object(&self, package_id: &ObjectID) -> SuiResult<Option<Object>> {
self.store.get_package_object(package_id)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know if this change is going to have broader implications -- I think it might inadvertantly introduce incomplete support for calling into packages that you just published. It would be better to limit this support to some kind of wrapper type that we only use for our purposes.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please double check where this is used to make sure the semantic change is ok

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah I agree on both and meant to flag it in the PR but figured I'd mention f2f.
This was vital to make things work and I wondered if that was a good thing or not.
We can do the wrapper and I'll review the usage, meanwhile adding @sblackshear who may have introduced this originally

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@awelc thoughts about this change?

@@ -1533,6 +1521,9 @@ impl AuthorityStore {
let mut task_objects = vec![];
mem::swap(&mut pending_objects, &mut task_objects);
let package_cache_clone = package_cache.clone();
// TODO: hook up the TypeLayoutResolver. It may require some traits implementation
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Finish this TODO?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah this needs to be fixed and working on it, it'll go away soon

None => {
let perpetual_tables =
Arc::new(AuthorityPerpetualTables::open(&path.join("store"), None));
let empty_database = perpetual_tables
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need this in test authority builder.
Test authority builder is used to create a single authority, which won't be able to run epoch-conservation check anyway.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok so removing the change for this file entirely

@@ -1494,7 +1512,11 @@ impl<S: ChildObjectResolver> Storage for TemporaryStore<S> {

impl<S: BackingPackageStore> BackingPackageStore for TemporaryStore<S> {
fn get_package_object(&self, package_id: &ObjectID) -> SuiResult<Option<Object>> {
self.store.get_package_object(package_id)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please double check where this is used to make sure the semantic change is ok

@dariorussi dariorussi requested a review from sblackshear May 2, 2023 17:14
@dariorussi dariorussi force-pushed the upgrade_type3 branch 2 times, most recently from bf0470e to 8af9b71 Compare May 3, 2023 12:49
@dariorussi dariorussi force-pushed the upgrade_type3 branch 5 times, most recently from 18a2572 to a0f77ec Compare May 9, 2023 23:07
@dariorussi dariorussi requested a review from tnowacki May 9, 2023 23:40
@dariorussi
Copy link
Contributor Author

ok this is ready so let's try to get a review and make sure everything is good

let ty = match load_type(&mut self.session, &type_tag) {
Ok(ty) => ty,
Err(err) => {
println!("ERROR: {:?}", err);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we keeping the println?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, thanks for pointing it out!

@@ -1697,7 +1699,16 @@ impl AuthorityState {
})?;

let layout = match request.object_format_options {
Some(format) => object.get_layout(format, epoch_store.module_cache().as_ref())?,
Some(format) => match object.data.try_as_move() {
Copy link
Contributor

@awelc awelc May 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit. Use map on the Option instead of match (in both cases)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll think about it, thanks

/// This function is intended to be called *after* we have charged for gas + applied the storage
/// rebate to the gas object, but *before* we have updated object versions. If
/// `do_expensive_checks` is false, this function will only check conservation of object storage
/// rea `epoch_fees` and `epoch_rebates` are only set for advance epoch transactions. The
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"rea"?

Copy link
Contributor Author

@dariorussi dariorussi May 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wish we did not lint this comment and left it alone.... lol....

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is my fault -- I just couldn't read it how it was before. I couldn't figure out what "rea" meant before, my guess is "rebates", but I didn't want to make the edit based on a guess, so I left it as it was.

@dariorussi dariorussi force-pushed the upgrade_type3 branch 6 times, most recently from c807e84 to 4dce6dc Compare May 10, 2023 13:25
Copy link
Member

@amnn amnn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for working on this @dariorussi !

Comment on lines 61 to 65
let ty = match load_type(&mut self.session, &type_tag) {
Ok(ty) => ty,
Err(_err) => {
return Err(SuiError::FailObjectLayout {
st: format!("{}", struct_tag),
});
}
};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: another opportunity for let ... else

Suggested change
let ty = match load_type(&mut self.session, &type_tag) {
Ok(ty) => ty,
Err(_err) => {
return Err(SuiError::FailObjectLayout {
st: format!("{}", struct_tag),
});
}
};
let Ok(ty) = load_type(&mut self.session, &type_tag) else {
return Err(SuiError::FailObjectLayout {
st: format!("{}", struct_tag),
});
};

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment on lines 1701 to 1767
let layout = match request.object_format_options {
Some(format) => object.get_layout(format, epoch_store.module_cache().as_ref())?,
Some(format) => match object.data.try_as_move() {
None => None,
Some(move_obj) => {
let epoch_store = self.load_epoch_store_one_call_per_task();
let move_vm = epoch_store.move_vm();
let mut layout_resolver =
TypeLayoutResolver::new(move_vm, self.database.as_ref());
Some(layout_resolver.get_layout(move_obj, format)?)
}
},
None => None,
};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or alternatively, match on both optional values in one go, in an if let:

let layout = if let (Some(format), Some(move_obj) = (
    request.object.object_format_options, 
    object.data.try_as_move(),
) {
    let epoch_store = self.load_epoch_store_one_call_per_task();
    let move_vm = epoch_store.move_vm();
    let mut layout_resolver = TypeLayoutResolver::new(move_vm, self.database.as_ref());
    Some(layout_resolver.get_layout(move_obj, format)?)
} else {
    None
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not know... you guys are really obsessed with removing matches.... :)
let me try in the code base and see if I like it

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like match, promise! I tend to use it when the match arms are symmetrical (i.e. not a happy path + error case), and there are more than two. But in other cases, I find them less clear than if let or let ... else.

Comment on lines -2178 to -2241
// threading the epoch_store through this API does not
// seem possible, so we just read it from the state (self) and fetch
// the module cache out of it.
// Notice that no matter what module cache we get things
// should work
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should preserve this comment, I think.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not think so, to be honest.
I added that comment but the way the code is and will be moving that is really a non issue.
We will have a layout cache and types are immutable so the issue is really not there at all

Comment on lines -2274 to -2337
// threading the epoch_store through this API does not
// seem possible, so we just read it from the state (self) and fetch
// the module cache out of it.
// Notice that no matter what module cache we get things
// should work
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto on preserving the comment here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see above

Comment on lines -2178 to -2241
// threading the epoch_store through this API does not
// seem possible, so we just read it from the state (self) and fetch
// the module cache out of it.
// Notice that no matter what module cache we get things
// should work
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should preserve this comment, I think.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see above

@@ -38,6 +39,17 @@ impl<S: ObjectStore> GetModule for PackageObjectCache<S> {
}
}

impl<S: BackingPackageStore + ModuleResolver<Error = SuiError>> ModuleResolver
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think, technically speaking, you only need to constrain S to implement BackingPackageStore. If it also implemented ModuleResolver<Error = SuiError>, we could implement this as self.store.get_module(id). The reason we don't do that is in the case of TemporaryStore, its BackingPackageStore implementation inspects written_objects looking for packages, while its ModuleResolver implementation does not.

@@ -52,6 +52,33 @@ macro_rules! exit_main {
};
}

// {$builder:expr, ($addr:expr)::$module_name:ident::$func:ident($($args:expr),* $(,)?)} => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is my bad, but I left this comment here (because I can never remember macro rules' pattern syntax), could you get rid of it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@@ -366,6 +393,8 @@ pub enum SuiError {
TransactionAlreadyExecuted { digest: TransactionDigest },
#[error("Object ID did not have the expected type")]
BadObjectType { error: String },
#[error("Fail to retrive Object layout for {st}")]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit, sp.

Suggested change
#[error("Fail to retrive Object layout for {st}")]
#[error("Fail to retrieve Object layout for {st}")]

/// This function is intended to be called *after* we have charged for gas + applied the storage
/// rebate to the gas object, but *before* we have updated object versions. If
/// `do_expensive_checks` is false, this function will only check conservation of object storage
/// rea `epoch_fees` and `epoch_rebates` are only set for advance epoch transactions. The
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is my fault -- I just couldn't read it how it was before. I couldn't figure out what "rea" meant before, my guess is "rebates", but I didn't want to make the edit based on a guess, so I left it as it was.

Comment on lines 135 to 136
let epoch_store = state.load_epoch_store_one_call_per_task();
let move_vm = epoch_store.move_vm();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's at least pull this call out of the nested loop, given that it's threateningly named "one_call_per_task".

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually moved everything that I could out of the loop (thanks for pointing out) given that I do not see reasons to keep it in the loop

@dariorussi dariorussi force-pushed the upgrade_type3 branch 3 times, most recently from af4afb5 to 2c6d710 Compare May 10, 2023 18:27
@dariorussi dariorussi changed the title Move expensive_check_sui_conservation to node init Create a LayoutResolver to compute layout from VM types May 10, 2023
@dariorussi
Copy link
Contributor Author

made suggested changes unless otherwise commented

- Move expensive_check_sui_conservation to node init
- Expose a layout resolver based on a VM session
- Decouple MoveResolver traits from Storage traits

In some of the places where we want to use the layout resolver, we
don't have access to all the information to implement `StorageView`,
but we only need to use the `BackingPackageStore`.

To support these cases, we need to provide a state view to
`LinkageView` that can serve packages, and then shim the Module and
Resource resolution functions.

This requires that `LinkageView` no longer accept a reference
parameter, but a value (so we can inject state into it) -- similar to
the trick we pulled to allow `Session` to own a `LinkageView`.  Then
implementors (like `TemporaryStore`) that only need to be passed by
reference can be passed by explicit reference,

i.e. `LinkageView<'state, TemporaryStore>` becomes
`LinkageView<&'state TemporaryStore>`.

However, we cannot impose a `&'state S: StorageView` bound, because
some APIs in `StorageView` require a mutable reference, so we needed
to decouple the resolution traits (now `SuiResolver`) and the storage
traits, to impose twin constraints:

```
fn ...<'state, S>
where
  S: StorageView
  &'state S: SuiResolver
```

- Logging improvements
- Bring `invariant_violation!` macro into `sui-types`.
- Create a variant of it -- `make_invariant_violation!` -- that
  creates the error, but does not wrap it in a result of return it.
- Add `format!` string support to these macros
- Use this macro for invariant violations within conservation checking
  with extra context on what is failing (what stage, what type is
  involved)
- Enable telemetry_subscription in the `move_package_upgrade_tests` to
  see the logs from authorities during tests.
- (Temporary) Print the IDs of published/upgraded packages in the
  failing test, for context.
@dariorussi dariorussi merged commit 7c6acca into main May 10, 2023
@dariorussi dariorussi deleted the upgrade_type3 branch May 10, 2023 22:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants