-
Notifications
You must be signed in to change notification settings - Fork 20.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
accounts/abi: Abi binding support for nested arrays, fixes #15648, including nested array unpack fix #15676
accounts/abi: Abi binding support for nested arrays, fixes #15648, including nested array unpack fix #15676
Conversation
Thank you for your contribution! Your commits seem to not adhere to the repository coding standards
Please check the contribution guidelines for more details. This message was auto-generated by https://gitcop.com |
aa52575
to
ac6ff7e
Compare
Thank you for your contribution! Your commits seem to not adhere to the repository coding standards
Please check the contribution guidelines for more details. This message was auto-generated by https://gitcop.com |
ac6ff7e
to
68e3cef
Compare
@GitCop Added an ":" and split that one commit message, quality suggestions bot 👍 |
68e3cef
to
0d5e1d1
Compare
For no particular good reason, the OSX Go 1.9 travis CI jobs fails:
This happens in a package that wasn't touched in this PR, nor uses the accounts/abi/bind package. I suspect this is because of the test being prone to a race condition (Execution literally depends on |
Thanks for the PR! I assume you already have examples that you based the implementation on, which you perhaps can place in the |
@holiman Sure, I'll make some tests. But that'll probably be next week. 🎅 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As @holiman said, please add more tests to make sure that nothing got broken, and also address the minor nitpicks in my review. Nice job otherwise!
accounts/abi/bind/bind.go
Outdated
case "32": | ||
return len(parts[0]), "int" | ||
case "64": | ||
return len(parts[0]), "long" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Couple remarks here:
- Instead of a
switch
, it would make more sense to have the type mapping in some map and write itreturn len(parts[0]), sizeMap[parts[2]]
. On top of being more concise, it would be less confusing to find out thatBingInt
is the fallback. - Can we update the regex to only accept between 1 and 2 digits?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
-
The
switch
isn't pretty, a map lookup may work better yes. It's not that this code is performance-critical anyway. On the other hand though, the mapping to go keywords is constant, and shouldn't change. And Go doesn't offer a constantmap
(afaik). So we either have a pseudo-constant map, or a (slightly less verbose) switch case. -
No. From the solidity docs: "
uint
andint
are aliases foruint256
andint256
, respectively.". And these are mapped to aBigInt
. So it's 0 or more digits to parse, and I don't see a reason to limit it with a maximum. (Soldity has a theoretical limit of 2^256, but I don't see why this should be enforced in this regex)
I'll see if I can improve the clarity when I write the tests. 👍
For now, happy holidays 🎄
eb6618d
to
7207fc5
Compare
@gballet When preparing the tests, I found another more serious problem related to nested-arrays. So I was writing the tests for the nested arrays, but discovered that the unpacking of nested arrays happened incorrectly. Packing of nested arrays works correctly, but unpacking only works for non-nested arrays/slices. The problem is that the unpacker assumes that each element of an array is 32 bytes (because of padding), but elements of a (nested) array-type can be different sizes. Note that the current code still unpacks nested arrays, but the data is read wrongly; it only reads the first entry correctly, and starts incrementing with too small increments from there, misaligning & duplicating the read data. I fixed it, but I'm not fond of my solution. Unpacking an individual element doesn't tell us the size of the consumed bytes, so I had to find another solution... For now, it calculates the size of the element by packing it, and it works correctly now, but the solution could be way nicer if we let You can find my debugging code here: https://github.com/protolambda/go-eth-nested-array-debug Edit: I read the ABI spec more extensively, and found a problem with my previously solution of packing; slices can be "seen" as 32 bytes elements, since the contents are located and accessed differently than arrays. However, arrays are still packed in-place, and can be more than 32 bytes. I removed the packing, and found a far more elegant solution to fix the array problem 👍 . |
accounts/abi/unpack.go
Outdated
if err != nil { | ||
return nil, err | ||
} | ||
i += len(interPacked) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
By packing it, and using the length as the increment, it guarantees correct element alignment (given that packing works correctly). However, getting a "consumed" size from the toGoType
call would be much more efficient. I decided to go with this inefficient solution first, since changing toGoType
affects other code.
Thank you for your contribution! Your commits seem to not adhere to the repository coding standards
Please check the contribution guidelines for more details. This message was auto-generated by https://gitcop.com |
Fixed unpacking of multi-dimensional arrays, without the packing step I took previously, and more efficient. 👍 Added a test for the bindings generated for the deeply nested (more than one level of nesting) arrays, as well as a test for both the unpacking and packing of the arrays. |
e08174f
to
5d42cf6
Compare
Thank you for your contribution! Your commits seem to not adhere to the repository coding standards
Please check the contribution guidelines for more details. This message was auto-generated by https://gitcop.com |
@GitCop Great. :I |
5d42cf6
to
b274e9b
Compare
Rebased to resolve merge conflicts, and rebased again to resolve GitCop. PR should be good to go now. @gballet Can you take a look? Edit: noticed that one of my tests still used an import from my test repo, which made it fail on travis CI. This, along with some improved naming, is fixed now. Edit 2: Forgot to go-fmt 2 lines... Fixed it. Edit 3: triggered a rebuild, the CI failed to download the linter module |
5951552
to
c5db3c5
Compare
Well, one test, in non-related code, fails in one of the CI jobs because of a race condition... :(
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have a few comments inline, and a more general one: since there is a lot of code repeats in your bindUnnestedTypes* / arrayBind* , see if instead you can use a general regexp like (address|uint|int|bool...)(\d)*
and then use a name conversion map depending on the output language.
// | ||
// Returned array sizes are in the same order as solidity signatures; inner array size first. | ||
// Array sizes may also be "", indicating a dynamic array. | ||
func wrapArray(stringKind string, innerLen int, innerMapping string) (string, []string) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be simpler, more readable and arguably more efficient to use the following regular expression \[(\d*)\]
along with FindAllStringSubmatch
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yup, probably a good idea. I'll simplify the implementation 👍
accounts/abi/bind/bind.go
Outdated
//append a "[]" for each array size. | ||
// (Full arraySizes is used in signature for consistency with the go-lang version) | ||
for i := 0; i < len(arraySizes); i++ { | ||
//normally, like with Go, you could declare an array size. Not in Java. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please update this comment to make it more factual like "java array type declarations do not include the length".
We are not interested in starting yet another religious war on the theme of whose language is better. Also, your comment is inaccurate: Java has arrays while Go has slices, and the typing in Java is more dynamic than what Go offers, so not enforcing the array size in the type declaration makes sense (and is therefore 'normal'). You don't like Java? That makes two of us. Still, this is not the place to vent out.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You're right, I'll change it. And I wasn't meaning to start a "religious war", it was written jokingly (i.e., you can in Java, but not in the same compact way).
accounts/abi/bind/bind.go
Outdated
out := inner | ||
//append a "[]" for each array size. | ||
// (Full arraySizes is used in signature for consistency with the go-lang version) | ||
for i := 0; i < len(arraySizes); i++ { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This loop can be rewritten: inner + strings.Repeat("[]", len(arraySizes)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good spot, I started of with the Go implementation, which has changing additions, instead of repeating "[]".
accounts/abi/unpack.go
Outdated
@@ -93,6 +93,17 @@ func readFixedBytes(t Type, word []byte) (interface{}, error) { | |||
|
|||
} | |||
|
|||
func getDeepSizeForType(t *Type) int { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nitpicking here: I would prefer getTotalArraySize rather than "deep" size.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's like a deep copy; it repeats the same operation up to the deepest array level. Total array size would work too, but it misses the aspect of nesting imho.
accounts/abi/unpack.go
Outdated
elemSize := 32 | ||
if t.T == ArrayTy { | ||
elemSize = getDeepSizeForType(t.Elem) | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would just write those 4 lines: elemSize := getDeepSizeForType(t.Elem)
since getDeppSizeForType
will return 32
if this is not an array.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, it doesn't. It is the element type that is being passed. So it returns 32 when the element is not array. Say you call getDeepSizeForType(t.Elem)
with t
being a slice, then we need the elemSize to be 32 bytes, since it's only the pointer to the contents of a slice-slot being counted, but t.Elem
may be ArrayTy
, resulting in something different than the 32 bytes.
{ | ||
def: `[{"type": "uint32[2][3][4]"}]`, | ||
enc: "000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000003000000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000050000000000000000000000000000000000000000000000000000000000000006000000000000000000000000000000000000000000000000000000000000000700000000000000000000000000000000000000000000000000000000000000080000000000000000000000000000000000000000000000000000000000000009000000000000000000000000000000000000000000000000000000000000000a000000000000000000000000000000000000000000000000000000000000000b000000000000000000000000000000000000000000000000000000000000000c000000000000000000000000000000000000000000000000000000000000000d000000000000000000000000000000000000000000000000000000000000000e000000000000000000000000000000000000000000000000000000000000000f000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000110000000000000000000000000000000000000000000000000000000000000012000000000000000000000000000000000000000000000000000000000000001300000000000000000000000000000000000000000000000000000000000000140000000000000000000000000000000000000000000000000000000000000015000000000000000000000000000000000000000000000000000000000000001600000000000000000000000000000000000000000000000000000000000000170000000000000000000000000000000000000000000000000000000000000018", | ||
want: [4][3][2]uint32{{{1, 2}, {3, 4}, {5, 6}}, {{7, 8}, {9, 10}, {11, 12}}, {{13, 14}, {15, 16}, {17, 18}}, {{19, 20}, {21, 22}, {23, 24}}}, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you should also check that errors get caught if e.g. the string is shorter or longer than the total size, as well as if the type specification is wrong.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I only fixed the iteration step being incorrect, not anything about the unpacking of types themselves. When the input length doesn't match up by accident, toGoType
will handle it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Alright
Well, there is, but they're not mine. I dislike the switch cases too (and yes, a mapping for each conversion language would be nicer) but the switch-structure works as the previous author intended, without problems, and I'm fine with that. No need to refactor it even more. |
1dbb74c
to
7eb7ef5
Compare
I'm aware, I only meant that you gave it a thought. Just because it's been done by someone else doesn't mean they're better at it than you are ^^ This being said, I heard that there is a refactoring on the way so I'll just fix it when it's rolled in. Thank you for your work so far, let me know when you're done with the regexp part and I'll review again. |
Well, I found another bug related to nested arrays. When unpacking arguments, it only considered the outer-size of arrays. This resulted in binary contents of nested arrays "overflowing" (effectively) into the contents of the arguments following after the array. Also see the explanation in the commit message. @gballet Would you be so kind to re-review again, to include this small bugfix? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
setting it to 'request changes' to make sure the PR doesn't get merged before I've had a chance to look at the new commit
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me
Now that 1.8.0 is out the door, we're lifting the "pr-embargo" and will start merging a bit more. I'm hoping to merge #15770 soon, and then some other abi-related PRs. |
@@ -80,6 +80,21 @@ func (arguments Arguments) Unpack(v interface{}, data []byte) error { | |||
return arguments.unpackAtomic(v, data) | |||
} | |||
|
|||
// Computes the full size of an array; | |||
// i.e. counting nested arrays, which count towards size for unpacking. | |||
func getArraySize(arr *Type) int { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've done a rebase on master, and find that if don't even use this method, but leave it as is, I get no test failures. I'll push my rebased branch, with the method still present (but unused), and you can take a look and check if
- The method is not needed, or,
- An additional testcase is needed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Check out my my branch: https://github.com/ethereum/go-ethereum/compare/master...holiman:pr_15676?expand=1 . It's rebased on master. PTAL that I didn't miss something, and why it's still working without using getArraySize
(I guess the corresponding place in the master-code is here: https://github.com/ethereum/go-ethereum/compare/master...holiman:pr_15676?expand=1#diff-04b076f4fee5b5afe22e6efa614e4aceR200 )
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We probably need tests for the argument unpacking then, it may be very uncommon, but there was definitely a bug.
How to reproduce it (will test later with your rebase, this is what needed to be fixed in the original PR starting point):
- Some argument value needs to be a nested array. The abi format packs the array pointers in-place, hence no standard 32 byte argument length, but a variable length based on the array.
- There needs to be another argument value, after the nested array, that will be offset because of the miscalculated length of the previous argument (the nested array). If there isn't any, then there is (effectively) nothing wrong. But when there is, the argument is read from the wrong position, causing array data from the array argument to be read into content of following argument(s).
So it shows up very rarely, doesn't cause a panic, and only affects multi-valued arguments.
Also consider that there can be multiple levels of nested arrays, all packed together. Hence the for-loop that multiplies them. For completeness: multiplying in the size calculation works because of the array-size symmetry, no need to recursively sum all sizes. Other collections such as slices are just pointers, same size as any other argument, so nothing wrong there.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
https://github.com/ethereum/go-ethereum/compare/master...holiman:pr_15676?expand=1#diff-04b076f4fee5b5afe22e6efa614e4aceR200
Is indeed the relevant part, but it doesn't cover arrays with multiple levels of nesting, only the surface. I'll rebase it with the multi-level size calculation, and add a test for it 👍
Also: - reduce usage of regexes a bit. - fix minor Java syntax problems Fixes ethereum#15648
The code previously assumed the arrays/slices were always 1 level deep. While the packing supports nested arrays (!!!). The current code for unpacking doesn't return the "consumed" length, so this fix had to work around that by calculating it (i.e. packing and getting resulting length) after the unpacking of the array element. It's far from ideal, but unpacking behaviour is fixed now.
Removed the temporary workaround of packing to calculate size, which was incorrect for slice-like types anyway. Full size of nested arrays is used now.
Test unpacking of an array nested more than one level.
Same as the deep nested array unpack test, but the other way around.
Test the usage of bindings that were generated for methods with multi-dimensional (and not just a single extra dimension, like foo[2][3]) array arguments and returns. edit: trigger rebuild, CI failed to fetch linter module.
wrapArray uses a regex now, and arrayBindingJava is improved.
The full step size for unpacking an array is now retrieved with "getFullElemSize".
Previously, the code only considered the outer-size of the array, ignoring the size of the contents. This was fine for most types, but nested arrays are packed directly into it, and count towards the total size. This resulted in arguments following a nested array to replicate some of the binary contents of the array. The fix: for arrays, calculate their complete contents size: count the arg.Type.Elem.Size when Elem is an Array, and repeat when their child is an array too, etc. The count is the number of 32 byte elements, similar to how it previously counted, but nested.
05e164a
to
1566af2
Compare
Arguments with a deeply nested array should not cause the next arguments to be read from the wrong position.
1566af2
to
c788f13
Compare
I missed that you had done this -- and I won't be available to review for a week. Maybe @gballet can review/merge? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. @protolambda Before I merge, I have a final request: could you please find a more descriptive and accurate name than virtualArgs
? This is an offset, and I think the name should reflect that, because this code is complicated (thank you for your effort and patience!) and it will help people reading it to better understand what it is about.
@gballet The name may be a bit confusing yes, but I didn't introduce it. The And if you read closer, it sort of makes sense, it counts things hidden from the surface argument count. Hence the "virtual". I do think they're not quite "arguments" however, more like partial arguments that just so happen to be exactly as long. Also, I added some extra comments just above the first usage to explain what is happening, but kept the old naming (constantly renaming things is something I try to avoid). @holiman Do you think it's a good idea to change it? |
Yup, that was my 'bad'. I also think it sort of makes sense, but I'm not opposed to changing it to something better. Thanks for the hard work, @protolambda , and sorry that it took so long to get it in! |
…5648, including nested array unpack fix (ethereum#15676) * accounts/abi/bind: support for multi-dim arrays Also: - reduce usage of regexes a bit. - fix minor Java syntax problems Fixes ethereum#15648 * accounts/abi/bind: Add some more documentation * accounts/abi/bind: Improve code readability * accounts/abi: bugfix for unpacking nested arrays The code previously assumed the arrays/slices were always 1 level deep. While the packing supports nested arrays (!!!). The current code for unpacking doesn't return the "consumed" length, so this fix had to work around that by calculating it (i.e. packing and getting resulting length) after the unpacking of the array element. It's far from ideal, but unpacking behaviour is fixed now. * accounts/abi: Fix unpacking of nested arrays Removed the temporary workaround of packing to calculate size, which was incorrect for slice-like types anyway. Full size of nested arrays is used now. * accounts/abi: deeply nested array unpack test Test unpacking of an array nested more than one level. * accounts/abi: Add deeply nested array pack test Same as the deep nested array unpack test, but the other way around. * accounts/abi/bind: deeply nested arrays bind test Test the usage of bindings that were generated for methods with multi-dimensional (and not just a single extra dimension, like foo[2][3]) array arguments and returns. edit: trigger rebuild, CI failed to fetch linter module. * accounts/abi/bind: improve array binding wrapArray uses a regex now, and arrayBindingJava is improved. * accounts/abi: Improve naming of element size func The full step size for unpacking an array is now retrieved with "getFullElemSize". * accounts/abi: support nested nested array args Previously, the code only considered the outer-size of the array, ignoring the size of the contents. This was fine for most types, but nested arrays are packed directly into it, and count towards the total size. This resulted in arguments following a nested array to replicate some of the binary contents of the array. The fix: for arrays, calculate their complete contents size: count the arg.Type.Elem.Size when Elem is an Array, and repeat when their child is an array too, etc. The count is the number of 32 byte elements, similar to how it previously counted, but nested. * accounts/abi: Test deep nested arr multi-arguments Arguments with a deeply nested array should not cause the next arguments to be read from the wrong position.
…5648, including nested array unpack fix (ethereum#15676) * accounts/abi/bind: support for multi-dim arrays Also: - reduce usage of regexes a bit. - fix minor Java syntax problems Fixes ethereum#15648 * accounts/abi/bind: Add some more documentation * accounts/abi/bind: Improve code readability * accounts/abi: bugfix for unpacking nested arrays The code previously assumed the arrays/slices were always 1 level deep. While the packing supports nested arrays (!!!). The current code for unpacking doesn't return the "consumed" length, so this fix had to work around that by calculating it (i.e. packing and getting resulting length) after the unpacking of the array element. It's far from ideal, but unpacking behaviour is fixed now. * accounts/abi: Fix unpacking of nested arrays Removed the temporary workaround of packing to calculate size, which was incorrect for slice-like types anyway. Full size of nested arrays is used now. * accounts/abi: deeply nested array unpack test Test unpacking of an array nested more than one level. * accounts/abi: Add deeply nested array pack test Same as the deep nested array unpack test, but the other way around. * accounts/abi/bind: deeply nested arrays bind test Test the usage of bindings that were generated for methods with multi-dimensional (and not just a single extra dimension, like foo[2][3]) array arguments and returns. edit: trigger rebuild, CI failed to fetch linter module. * accounts/abi/bind: improve array binding wrapArray uses a regex now, and arrayBindingJava is improved. * accounts/abi: Improve naming of element size func The full step size for unpacking an array is now retrieved with "getFullElemSize". * accounts/abi: support nested nested array args Previously, the code only considered the outer-size of the array, ignoring the size of the contents. This was fine for most types, but nested arrays are packed directly into it, and count towards the total size. This resulted in arguments following a nested array to replicate some of the binary contents of the array. The fix: for arrays, calculate their complete contents size: count the arg.Type.Elem.Size when Elem is an Array, and repeat when their child is an array too, etc. The count is the number of 32 byte elements, similar to how it previously counted, but nested. * accounts/abi: Test deep nested arr multi-arguments Arguments with a deeply nested array should not cause the next arguments to be read from the wrong position.
The binding generator only matched one level of the multi-dimensional array types that were parsed by the other binding code.
Fix:
Before merging: This is my first PR here, suggestions & reviews are very much appreciated 👍