Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[NRBF] Address issues discovered by Threat Model #106629

Merged
merged 13 commits into from
Sep 16, 2024

Conversation

adamsitnik
Copy link
Member

@adamsitnik adamsitnik commented Aug 19, 2024

fixes #106644

@adamsitnik adamsitnik added binaryformatter-migration Issues related to the removal of BinaryFormatter and migrations away from it area-System.Formats.Nrbf labels Aug 19, 2024
@adamsitnik adamsitnik self-assigned this Aug 19, 2024
Copy link

Note regarding the new-api-needs-documentation label:

This serves as a reminder for when your PR is modifying a ref *.cs file and adding/modifying public APIs, please make sure the API implementation in the src *.cs file is documented with triple slash comments, so the PR reviewers can sign off that change.

1 similar comment
Copy link

Note regarding the new-api-needs-documentation label:

This serves as a reminder for when your PR is modifying a ref *.cs file and adding/modifying public APIs, please make sure the API implementation in the src *.cs file is documented with triple slash comments, so the PR reviewers can sign off that change.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on what we have discussed and what @bartonjs wrote here: #103713 (comment)

I believe the type names and assembly names should not be provided in the exception messages.

private static long GetJaggedArrayTotalElementsCount(BinaryArrayRecord jaggedArrayRecord)
{
long result = 0;
Queue<BinaryArrayRecord>? jaggedArrayRecords = null;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The decoder does not perform any recursion when decoding the jagged array. So does this method, but at a cost of potential Queue allocation.

@adamsitnik adamsitnik changed the title [NRB] Address issues discovered by Threat Model [NRBF] Address issues discovered by Threat Model Aug 19, 2024
- rename TotalElementsCount to FlattenedLength
- Ensure lack of recursive call
- add a comment explaining null nuances
- don't include member name in the exception message
- remove unused resource
- make the argument int rather than an object to make it clear it's never a string
@@ -126,26 +126,23 @@
<data name="Serialization_UnexpectedNullRecordCount" xml:space="preserve">
<value>Unexpected Null Record count.</value>
</data>
<data name="Serialization_MaxArrayLength" xml:space="preserve">
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this resource was not being used for a while

Comment on lines +57 to +58
// It is possible to have binary array records have an element type of array without being marked as jagged.
|| TypeName.GetElementType().IsArray;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JeremyKuhne this handles the scenario you have mentioned offline (there is also a test for that). Thank you again for pointing this out!

@adamsitnik adamsitnik marked this pull request as ready for review August 20, 2024 18:00
Comment on lines 230 to 240
case SerializationRecordType.ObjectNullMultiple:
// All nulls need to be included, as it's another form of possible attack.
checked
{
result += ((NullsRecord)item).NullCount;
}
break;
default:
result++;
break;
}
Copy link
Member

@bartonjs bartonjs Sep 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't the arrays already know their lengths? I'd expect FlattenedLength to be a very simple

long length = here.Length;

foreach (element in here.Children)
{
    if (element.IsArray)
    {
        length += element.FlattenedLength;
    }
}

return length;

(or the queue variant)

And that approach would already have counted the null elements. Ensuring that the number of records matches the length is a different problem (one I feel GetArray tries to solve?).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bartonjs You are right. This suggestion has greatly simplified the code and solved the other problem (arrays not being included themselves)

Comment on lines 211 to 227
case SerializationRecordType.ArraySingleString:
ArrayRecord nestedArrayRecord = (ArrayRecord)record;
if (nestedArrayRecord.IsJagged)
{
(jaggedArrayRecords ??= new()).Enqueue((BinaryArrayRecord)nestedArrayRecord);
}
else
{
Debug.Assert(nestedArrayRecord is not BinaryArrayRecord, "Ensure lack of recursive call");
checked
{
// In theory somebody could create a payload that would represent
// a very nested array with total elements count > long.MaxValue.
result += nestedArrayRecord.FlattenedLength;
}
}
break;
Copy link
Member

@bartonjs bartonjs Sep 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't counting the arrays themselves. Assuming that's intentional, you should have a comment at the top of the method (or before this switch) that explains what does, and what doesn't count.

object[][] objs = new object[1_000_000][];
objs.Fill(Array.Empty<object>());

Doesn't feel like it should have a smaller count than

object[][] objs = new object[1_000_000][];
objs.Fill(null);

And right now it feels like the first one says 0, and the second says a million.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW, I can understand not counting the arrays as answering how many int values might come out of an int[][][]; but then it gets weird when the nulls get counted, because they're not int values.

So I guess there's a bit of "what is this number supposed to mean?". The comment can explain that, and then the code can be judged against the comment.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, I've fixed that.

}
else
{
Debug.Assert(nestedArrayRecord is not BinaryArrayRecord, "Ensure lack of recursive call");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can read the assert, but I don't understand it. Why can't a BinaryArrayRecord be nested within another BinaryArrayRecord? Why would that necessarily mean that a recursive call happened?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've eliminated the potential recursive call and added a comment.

# Conflicts:
#	src/libraries/System.Formats.Nrbf/src/System/Formats/Nrbf/RectangularArrayRecord.cs
@adamsitnik
Copy link
Member Author

@MihuBot fuzz NrbfDecoder

Copy link
Contributor

@buyaa-n buyaa-n left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM and looks @bartonjs's feedbacks addressed as needed

# Conflicts:
#	src/libraries/System.Formats.Nrbf/src/System/Formats/Nrbf/RectangularArrayRecord.cs
@adamsitnik
Copy link
Member Author

/ba-g DeadLetter is unrelated

@adamsitnik adamsitnik merged commit 0154a2f into dotnet:main Sep 16, 2024
80 of 85 checks passed
adamsitnik added a commit to adamsitnik/runtime that referenced this pull request Sep 16, 2024
* introduce ArrayRecord.FlattenedLength

* do not include invalid Type or Assembly names in the exception messages, as it's most likely corrupted/tampered/malicious data and could be used as a vector of attack.

* It is possible to have binary array records have an element type of array without being marked as jagged
jtschuster pushed a commit to jtschuster/runtime that referenced this pull request Sep 17, 2024
* introduce ArrayRecord.FlattenedLength

* do not include invalid Type or Assembly names in the exception messages, as it's most likely corrupted/tampered/malicious data and could be used as a vector of attack.

* It is possible to have binary array records have an element type of array without being marked as jagged
carlossanlop pushed a commit that referenced this pull request Sep 17, 2024
* [NRBF] Don't use Unsafe.As when decoding DateTime(s) (#105749)

* Add NrbfDecoder Fuzzer (#107385)

* [NRBF] Fix bugs discovered by the fuzzer (#107368)

* bug #1: don't allow for values out of the SerializationRecordType enum range

* bug #2: throw SerializationException rather than KeyNotFoundException when the referenced record is missing or it points to a record of different type

* bug #3: throw SerializationException rather than FormatException when it's being thrown by BinaryReader (or sth else that we use)

* bug #4: document the fact that IOException can be thrown

* bug #5: throw SerializationException rather than OverflowException when parsing the decimal fails

* bug #6: 0 and 17 are illegal values for PrimitiveType enum

* bug #7: throw SerializationException when a surrogate character is read (so far an ArgumentException was thrown)
# Conflicts:
#	src/libraries/System.Formats.Nrbf/src/System/Formats/Nrbf/NrbfDecoder.cs

* [NRBF] throw SerializationException when a surrogate character is read (#107532)

 (so far an ArgumentException was thrown)

* [NRBF] Fuzzing non-seekable stream input (#107605)

* [NRBF] More bug fixes (#107682)

- Don't use `Debug.Fail` not followed by an exception (it may cause problems for apps deployed in Debug)
- avoid Int32 overflow
- throw for unexpected enum values just in case parsing has not rejected them
- validate the number of chars read by BinaryReader.ReadChars
- pass serialization record id to ex message
- return false rather than throw EndOfStreamException when provided Stream has not enough data
- don't restore the position in finally 
- limit max SZ and MD array length to Array.MaxLength, stop using LinkedList<T> as List<T> will be able to hold all elements now
- remove internal enum values that were always illegal, but needed to be handled everywhere
- Fix DebuggerDisplay

* [NRBF] Comments and bug fixes from internal code review (#107735)

* copy comments and asserts from Levis internal code review

* apply Levis suggestion: don't store Array.MaxLength as a const, as it may change in the future

* add missing and fix some of the existing comments

* first bug fix: SerializationRecord.TypeNameMatches should throw ArgumentNullException for null Type argument

* second bug fix: SerializationRecord.TypeNameMatches should know the difference between SZArray and single-dimension, non-zero offset arrays (example: int[] and int[*])

* third bug fix: don't cast bytes to booleans

* fourth bug fix: don't cast bytes to DateTimes

* add one test case that I've forgot in previous PR
# Conflicts:
#	src/libraries/System.Formats.Nrbf/src/System/Formats/Nrbf/SerializationRecord.cs

* [NRBF] Address issues discovered by Threat Model  (#106629)

* introduce ArrayRecord.FlattenedLength

* do not include invalid Type or Assembly names in the exception messages, as it's most likely corrupted/tampered/malicious data and could be used as a vector of attack.

* It is possible to have binary array records have an element type of array without being marked as jagged

---------

Co-authored-by: Buyaa Namnan <bunamnan@microsoft.com>
sirntar pushed a commit to sirntar/runtime that referenced this pull request Sep 30, 2024
* introduce ArrayRecord.FlattenedLength

* do not include invalid Type or Assembly names in the exception messages, as it's most likely corrupted/tampered/malicious data and could be used as a vector of attack.

* It is possible to have binary array records have an element type of array without being marked as jagged
@github-actions github-actions bot locked and limited conversation to collaborators Oct 17, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-System.Formats.Nrbf binaryformatter-migration Issues related to the removal of BinaryFormatter and migrations away from it new-api-needs-documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[API Proposal]: ArrayRecord.TotalElementsCount
4 participants