Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Preview] Change Feed: Adds SDK changes required for Full-Fidelity Change Feed #3197

Merged
merged 40 commits into from
Aug 1, 2022
Merged
Show file tree
Hide file tree
Changes from 30 commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
7838f35
new wire format implementation and tests
philipthomas-MSFT Mar 1, 2022
42b169c
default gateway mode on FFCF
philipthomas-MSFT May 16, 2022
b59dea9
new wire format implementation and tests
philipthomas-MSFT Mar 1, 2022
284517c
default gateway mode on FFCF
philipthomas-MSFT May 16, 2022
51970ef
removing some comments
philipthomas-MSFT May 16, 2022
2c9be66
various changes based on PR feedback
philipthomas-MSFT May 17, 2022
09d3549
changed metadata to changefeedmetadata based on PR feedback
philipthomas-MSFT May 17, 2022
993dc26
renaming public contract for metadata, current and previous lsn
philipthomas-MSFT May 17, 2022
7011062
itemchanges converter/resolver/serializer changes. changes to use Uni…
philipthomas-MSFT May 18, 2022
5b3691c
left out a summary on metadata
philipthomas-MSFT May 18, 2022
e0177c9
removed comments
philipthomas-MSFT May 18, 2022
1046526
removed unnecessary using directive
philipthomas-MSFT May 18, 2022
8e64a82
adding comment explaining why we are defaulting gateway mode for full…
philipthomas-MSFT May 18, 2022
1422b00
small correction on comment
philipthomas-MSFT May 18, 2022
60ced7e
removed unnecessar using directive
philipthomas-MSFT May 18, 2022
caf7b87
add summary
philipthomas-MSFT May 18, 2022
ab6a83f
did not need UserSerializer, i think. will pu some thought on it
philipthomas-MSFT May 19, 2022
f6fa2d4
main change was reverting AllOperations back to FullFidelity. Intenti…
philipthomas-MSFT May 24, 2022
94f8bc8
removed datetmeoffsetconverter(again), some formatting issues, renami…
philipthomas-MSFT May 24, 2022
4425a78
proposing we keep user serializer enabled that works for container op…
philipthomas-MSFT Jun 1, 2022
4f32fa8
removed whitelist condition in CosmosSerializer
philipthomas-MSFT Jun 1, 2022
22cd9ff
fix some tests
philipthomas-MSFT Jun 1, 2022
8860488
reverting back to JsonProperty and whitelisting ChangeFeedItemChanges<T>
philipthomas-MSFT Jun 3, 2022
1c12bbe
include TTL. However, I feel that the testing needs revisting because…
philipthomas-MSFT Jun 6, 2022
c0b69ab
removed a dup test that was breaking due to wire format metadata chan…
philipthomas-MSFT Jun 6, 2022
c362617
running UpdateContracts.ps1
philipthomas-MSFT Jun 9, 2022
178a168
re-ran UpdateContracts.ps1 while using the public cosmos db emulator …
philipthomas-MSFT Jun 13, 2022
8068ae9
some tests changes mostly
philipthomas-MSFT Jun 13, 2022
1e46776
add example and remarks for ChangeFeedItemChanges<>
philipthomas-MSFT Jun 13, 2022
ba1b630
running UpdateContract.ps1 again after the last change
philipthomas-MSFT Jun 20, 2022
feebdb6
Update Microsoft.Azure.Cosmos/tests/Microsoft.Azure.Cosmos.EmulatorTe…
philipthomas-MSFT Jun 22, 2022
37e1f04
removing Current from CurrentLogSequenceNumber; changing comment on P…
philipthomas-MSFT Jun 22, 2022
884068a
Merge branch 'users/philipthomas/fullfidelitycfp' of https://github.c…
philipthomas-MSFT Jun 22, 2022
146196b
re-running UpdateContracts.ps1 for the LogSequenceNumber change from …
philipthomas-MSFT Jun 29, 2022
1d2da2b
Merge branch 'master' into users/philipthomas/fullfidelitycfp
philipthomas-MSFT Jul 26, 2022
364f913
fixed a test because of the emulator change to include deleted operat…
philipthomas-MSFT Jul 28, 2022
4f33d82
Merge branch 'master' into users/philipthomas/fullfidelitycfp
philipthomas-MSFT Jul 28, 2022
11d48f1
Changed ChangeFeedItemChanges to singluar
philipthomas-MSFT Jul 28, 2022
c9cbd95
Merge branch 'users/philipthomas/fullfidelitycfp' of https://github.c…
philipthomas-MSFT Jul 28, 2022
2102b7b
renaming to Lsn
philipthomas-MSFT Jul 29, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,17 @@ namespace Microsoft.Azure.Cosmos.ChangeFeed

internal sealed class ChangeFeedModeFullFidelity : ChangeFeedMode
{
public static readonly string FullFidelityHeader = "Full-Fidelity Feed"; // HttpConstants.A_IMHeaderValues.FullFidelityFeed

public static ChangeFeedMode Instance { get; } = new ChangeFeedModeFullFidelity();

internal override void Accept(RequestMessage requestMessage)
{
requestMessage.Headers.Add(HttpConstants.HttpHeaders.A_IM, ChangeFeedModeFullFidelity.FullFidelityHeader);
requestMessage.UseGatewayMode = true;
philipthomas-MSFT marked this conversation as resolved.
Show resolved Hide resolved
philipthomas-MSFT marked this conversation as resolved.
Show resolved Hide resolved

// Above, defaulting to Gateway is necessary for Full-Fidelity Change Feed for the Split-handling logic resides within Compute Gateway.
// TODO: If and when, this changes, it will be necessary to remove this.

requestMessage.Headers.Add(HttpConstants.HttpHeaders.A_IM, HttpConstants.A_IMHeaderValues.FullFidelityFeed);
requestMessage.Headers.Add(HttpConstants.HttpHeaders.ChangeFeedWireFormatVersion, Constants.ChangeFeedWireFormatVersions.SeparateMetadataWithCrts);
}
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
//------------------------------------------------------------
// Copyright (c) Microsoft Corporation. All rights reserved.
//------------------------------------------------------------

namespace Microsoft.Azure.Cosmos
{
using Newtonsoft.Json;

/// <summary>
/// The typed response that contains the current, previous, and metadata change feed resource when <see cref="ChangeFeedMode"/> is initialized to <see cref="ChangeFeedMode.FullFidelity"/>.
philipthomas-MSFT marked this conversation as resolved.
Show resolved Hide resolved
/// </summary>
philipthomas-MSFT marked this conversation as resolved.
Show resolved Hide resolved
/// <example>
/// <code>
/// <![CDATA[
/// public class ToDoActivity
/// {
/// public string type { get; set; }
/// public string id { get; set; }
/// public string status { get; set; }
/// }
///
/// ChangeFeedMode changeFeedMode = ChangeFeedMode.FullFidelity;
/// PartitionKey partitionKey = new PartitionKey(@"learning");
/// ChangeFeedStartFrom changeFeedStartFrom = ChangeFeedStartFrom.Now(FeedRange.FromPartitionKey(partitionKey));
///
/// using (FeedIterator<ChangeFeedItemChanges<ToDoActivity>> feedIterator = container.GetChangeFeedIterator<ChangeFeedItemChanges<ToDoActivity>>(
/// changeFeedStartFrom: changeFeedStartFrom,
/// changeFeedMode: changeFeedMode))
/// {
/// while (feedIterator.HasMoreResults)
/// {
/// FeedResponse<ChangeFeedItemChanges<ToDoActivity>> feedResponse = await feedIterator.ReadNextAsync();
///
/// if (feedResponse.StatusCode != HttpStatusCode.NotModified)
/// {
/// IEnumerable<ChangeFeedItemChanges<ToDoActivity>> feedResource = feedResponse.Resource;
///
/// foreach(ChangeFeedItemChanges<ToDoActivity> itemChanges in feedResource)
/// {
/// ToDoActivity currentToDoActivity = itemChanges.Current;
/// ToDoActivity previousToDoActivity = itemChanges.Previous;
/// ChangeFeedMetadata toDoActivityMetadata = itemChanges.Metadata;
/// }
/// }
/// }
/// }
/// ]]>
/// </code>
/// </example>
/// <remarks><see cref="ChangeFeedItemChanges{T}"/> is an optional helper class that uses Newtonsoft serialization libraries. Users are welcome to create their own custom helper class.</remarks>
#if PREVIEW
public
#else
internal
#endif
class ChangeFeedItemChanges<T>
philipthomas-MSFT marked this conversation as resolved.
Show resolved Hide resolved
{
/// <summary>
/// The full fidelity change feed current item.
/// </summary>
[JsonProperty(PropertyName = "current")]
public T Current { get; set; }
kirankumarkolli marked this conversation as resolved.
Show resolved Hide resolved

/// <summary>
/// The full fidelity change feed metadata.
/// </summary>
[JsonProperty(PropertyName = "metadata", NullValueHandling = NullValueHandling.Ignore)]
public ChangeFeedMetadata Metadata { get; set; }

/// <summary>
philipthomas-MSFT marked this conversation as resolved.
Show resolved Hide resolved
/// The previous image on replace is not going to be exposed by default, this was done to address COGs concerns from PITR.
philipthomas-MSFT marked this conversation as resolved.
Show resolved Hide resolved
/// For deletes previous image is always going to be provided. To opt-in for previous image for replaces, currently we would
/// need to set naming config for a customer(enablePreviousImageForReplaceInFFCF, that’s to for critical scenarios when
/// there is high business need, no billing is afffected). Later we will add explicit opt-in in Portal, and customer would
/// get a higher bill – these changes need to be developed.
/// </summary>
[JsonProperty(PropertyName = "previous", NullValueHandling = NullValueHandling.Ignore)]
public T Previous { get; set; }
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
//------------------------------------------------------------
// Copyright (c) Microsoft Corporation. All rights reserved.
//------------------------------------------------------------

namespace Microsoft.Azure.Cosmos
{
using System;
using Microsoft.Azure.Documents;
using Newtonsoft.Json;
using Newtonsoft.Json.Converters;

/// <summary>
/// The metadata of a change feed resource with <see cref="ChangeFeedMode"/> is initialized to <see cref="ChangeFeedMode.FullFidelity"/>.
/// </summary>
#if PREVIEW
public
#else
internal
#endif
class ChangeFeedMetadata
philipthomas-MSFT marked this conversation as resolved.
Show resolved Hide resolved
{
/// <summary>
/// New instance of meta data for <see cref="ChangeFeedItemChanges{T}"/> created.
/// </summary>
/// <param name="conflictResolutionTimestamp"></param>
/// <param name="currentLogSequenceNumber"></param>
/// <param name="operationType"></param>
/// <param name="previousLogSequenceNumber"></param>
public ChangeFeedMetadata(
philipthomas-MSFT marked this conversation as resolved.
Show resolved Hide resolved
DateTime conflictResolutionTimestamp,
long currentLogSequenceNumber,
ChangeFeedOperationType operationType,
long previousLogSequenceNumber)
{
this.ConflictResolutionTimestamp = conflictResolutionTimestamp;
this.CurrentLogSequenceNumber = currentLogSequenceNumber;
this.OperationType = operationType;
this.PreviousLogSequenceNumber = previousLogSequenceNumber;
}

/// <summary>
/// The conflict resolution timestamp.
/// </summary>
[JsonProperty(PropertyName = "crts", NullValueHandling = NullValueHandling.Ignore)]
[JsonConverter(typeof(UnixDateTimeConverter))]
public DateTime ConflictResolutionTimestamp { get; }
kirankumarkolli marked this conversation as resolved.
Show resolved Hide resolved

/// <summary>
/// The current logical sequence number.
/// </summary>
[JsonProperty(PropertyName = "lsn", NullValueHandling = NullValueHandling.Ignore)]
public long CurrentLogSequenceNumber { get; }
philipthomas-MSFT marked this conversation as resolved.
Show resolved Hide resolved

/// <summary>
/// The change feed operation type.
/// </summary>
[JsonProperty(PropertyName = "operationType")]
[JsonConverter(typeof(StringEnumConverter))]
public ChangeFeedOperationType OperationType { get; }

/// <summary>
philipthomas-MSFT marked this conversation as resolved.
Show resolved Hide resolved
/// The previous logical sequence number.
/// </summary>
[JsonProperty(PropertyName = "previousImageLSN", NullValueHandling = NullValueHandling.Ignore)]
public long PreviousLogSequenceNumber { get; }

/// <summary>
/// Used to distinquish explicit deletes (e.g. via DeleteItem) from deletes caused by TTL expiration (a collection may define time-to-live policy for documents).
/// </summary>
[JsonProperty(PropertyName = "timeToLiveExpired", NullValueHandling= NullValueHandling.Ignore)]
public bool TimeToLiveExpired { get; }
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
//------------------------------------------------------------
// Copyright (c) Microsoft Corporation. All rights reserved.
//------------------------------------------------------------

namespace Microsoft.Azure.Cosmos
{
using System.Runtime.Serialization;

/// <summary>
/// The operation type of a change feed resource with <see cref="ChangeFeedMode"/> is initialized to <see cref="ChangeFeedMode.FullFidelity"/>. Upsert operations will yield <see cref="Create"/> or <see cref="Replace"/>.
/// </summary>
#if PREVIEW
public
#else
internal
#endif
enum ChangeFeedOperationType
{
/// <summary>
/// The create operation type.
/// </summary>
[EnumMember(Value = "create")]
Create,

/// <summary>
philipthomas-MSFT marked this conversation as resolved.
Show resolved Hide resolved
/// The replace operation type.
/// </summary>
[EnumMember(Value = "replace")]
Replace,

/// <summary>
/// The delete operation type.
/// </summary>
[EnumMember(Value = "delete")]
Delete,
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,7 @@ private CosmosSerializer GetSerializer<T>()
string directAssemblyName = typeof(Documents.PartitionKeyRange).Assembly.GetName().Name;
string inputAssemblyName = inputType.Assembly.GetName().Name;
bool inputIsClientOrDirect = string.Equals(inputAssemblyName, clientAssemblyName) || string.Equals(inputAssemblyName, directAssemblyName);
bool typeIsWhiteListed = inputType == typeof(Document);
bool typeIsWhiteListed = inputType == typeof(Document) || (inputType.IsGenericType && inputType.GetGenericTypeDefinition() == typeof(ChangeFeedItemChanges<>));

if (!typeIsWhiteListed && inputIsClientOrDirect)
{
Expand Down
Loading