Adding more information to LogFileStats + minor updates to tests

[eliasyishak]

This update will include more information for the stats returns from the log file. The following keys have been added

minsFromStartDateTime – how many minutes have elapsed since the oldest log item
minsFromEndDateTime – how many minutes have elapsed since the earliest log item
recordCount – total count of records in the log file
eventCount – a map object that has each event sent as the key and a count of how many times that event was sent

{
    "startDateTime": "2023-02-22 15:23:24.410921",
    "minsFromStartDateTime": 20339,  // NEW
    "endDateTime": "2023-03-08 15:46:36.318211",
    "minsFromEndDateTime": 156,  // NEW
    "sessionCount": 7,
    "flutterChannelCount": 2,
    "toolCount": 1,
    "recordCount": 23,  // NEW
    "eventCount": {  // NEW
        "hot_reload_time": 16,
        "analytics_collection_enabled": 7
    }
}

This will help with survey conditions that involve the presence of an event, count exceeding N for an event being called, etc.

Included in this PR also has local_time in the body of the request to include the timezone offset so that the local time can be used for querying if desired

"local_time": {
  "value": "2023-01-31 14:32:14.592898 -0500"
}

[christopherfujino]

What about just storing startDateTime and endDateTime as ints, either from millisecondsSinceEpoch or microsecondsSinceEpoch?

[eliasyishak]

Oh shoot, I totally forgot that I'm using LogFileStats as a data class essentially. It only appears as a string when you print, or invoke toString() on the entire class. And that was really just for debugging when developing.

So really, when we perform operations with these value, we don't even convert it from ints into DateTime. Thoughts on still leaving it this way?

  @override
  String toString() => jsonEncode(<String, Object?>{
        'startDateTime': startDateTime.toString(),
        'minsFromStartDateTime': minsFromStartDateTime,
        'endDateTime': endDateTime.toString(),
        'minsFromEndDateTime': minsFromEndDateTime,
        'sessionCount': sessionCount,
        'flutterChannelCount': flutterChannelCount,
        'toolCount': toolCount,
        'recordCount': recordCount,
        'eventCount': eventCount,
      });

[christopherfujino]

Ahh ok, in that case keeping it a DateTime makes sense. However, at what point do we serialize this to disk, and in what format are we serializing these timestamps?

[eliasyishak]

So we don't actually serialize anything after the log file stats have been created. This data class will instead be used for checking conditions for a survey and returning a Boolean to show the survey or not.

But these logs are persisted before this step on the user's machine in the same form that they are sent to GA. And in the local log file, it is being persisted as a date time string so that we know the time the user sent the event in their locale. GA also assigns its own time stamp in milliseconds since epoch time in UTC so that all events can be compared chronologically.

[christopherfujino]

If it's never serialized, how is it part of userProps https://github.com/dart-lang/tools/blob/main/pkgs/unified_analytics/lib/src/log_handler.dart#L226? Isn't that map being read from disk?

[eliasyishak]

So the workflow starts with an event sent, and the events usually look like the below being sent in the body of a POST request

{
  "client_id": "46cc0ba6-f604-4fd9-aa2f-8a20beb24cd4",
  "events": [{ "name": "testing", "params": { "time_ns": 345 } }],
  "user_properties": {
    "session_id": { "value": 1673466750423 },
    "flutter_channel": { "value": "ey-test-channel" },
    "host": { "value": "macos" },
    "flutter_version": { "value": "Flutter 3.6.0-7.0.pre.47" },
    "dart_version": { "value": "Dart 2.19.0" },
    "tool": { "value": "flutter-tools" },
    "local_time": { "value": "2023-01-11 14:53:31.471816" }
  }
}

In the above, we do serialize the DateTime to a string in the form yyyy-mm-dd hh:mm:ss.ffff which is the default toString output for a datetime. This is so in GA and BigQuery, we can look at the table and understand the data easily and deserializing within BigQuery is trivial.

Once the event is sent, the entire payload of the POST request is saved into the log file. It is at this point that we read it in and deserialize the string back into a DateTime.

After we have created the LogFileStats data class, we don't serialize that object again. Once we have survey conditions, we will do checks within dart code.

final LogFileStats query = _logHandler.logFileStats();

if (query.toolCount > 2) {
  print('The user has used more than 2 tools');
}

It is only when we print out the query object from above that the data in the class gets serialized again to a string, and this is really for debugging purposes that I included that toString method for LogFileStats

[christopherfujino]

Oh ok, never mind, I misread what you said, you said it is persisted because we specifically want it in the user's locale? So GA is expecting a localized string?

[eliasyishak]

Yep, it is. And that's because GA already assigns a timestamp for the event which is in UTC and in milliseconds since epoch time

Also it's not really "expecting" it, we defined the schema of the table to be that way

The below also shows how we store the user props data with the local time

[christopherfujino]

LGTM