Skip to content

Fix cannot sync data between mobile and web #3750

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 10 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions docs/adr/0066-synchronize-caching.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# 66. Synchronize caching

Date: 2025-06-18

## Status

Accepted

## Context

Synchronizing server and database data is quite troublesome,
when we use many protocols to receive mails (FCM, WebSocket, Memory operator).
In addition, we are using `Hive` database to store data,
it does not support multiple isolate to access the database.
This leads to inconsistent database data, sometimes corrupted and lost data.

## Decision

To overcome this problem we need a database that can support multiple isolate.
So we need to use [Hive_CE](https://pub.dev/packages/hive_ce) to replace `Hive` database.
Hive CE is a spiritual continuation of Hive v2 with the following new features, such as:

- Isolate support through IsolatedHive
- Automatic type adapter generation using the GenerateAdapters annotation
- No more manually adding annotations to every type and field
- Generate adapters for classes outside the current package
- A HiveRegistrar extension that lets you register all your generated adapters in one call
- Extends the maximum type ID from 223 to 65439
...

## Consequences

- Data is always synchronized with the latest
20 changes: 20 additions & 0 deletions lib/features/base/upgradeable/upgrade_hive_database_steps_v17.dart
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@

import 'package:tmail_ui_user/features/base/upgradeable/upgrade_database_steps.dart';
import 'package:tmail_ui_user/features/caching/caching_manager.dart';
import 'package:tmail_ui_user/features/caching/config/hive_cache_config.dart';

class UpgradeHiveDatabaseStepsV17 extends UpgradeDatabaseSteps {

final CachingManager _cachingManager;

UpgradeHiveDatabaseStepsV17(this._cachingManager);

@override
Future<void> onUpgrade(int oldVersion, int newVersion) async {
if (oldVersion > 0 && oldVersion < newVersion && newVersion == 17) {
await HiveCacheConfig.instance.setUp(isolated: false);
await _cachingManager.migrateHiveToIsolatedHive();
await _cachingManager.closeHive(isolated: false);
}
}
}
180 changes: 98 additions & 82 deletions lib/features/caching/caching_manager.dart
Original file line number Diff line number Diff line change
Expand Up @@ -5,122 +5,113 @@ import 'package:fcm/model/type_name.dart';
import 'package:jmap_dart_client/jmap/account_id.dart';
import 'package:jmap_dart_client/jmap/core/user_name.dart';
import 'package:model/extensions/account_id_extensions.dart';
import 'package:tmail_ui_user/features/caching/clients/account_cache_client.dart';
import 'package:tmail_ui_user/features/caching/clients/email_cache_client.dart';
import 'package:tmail_ui_user/features/caching/clients/fcm_cache_client.dart';
import 'package:tmail_ui_user/features/caching/clients/firebase_registration_cache_client.dart';
import 'package:tmail_ui_user/features/caching/clients/hive_cache_version_client.dart';
import 'package:tmail_ui_user/features/caching/clients/mailbox_cache_client.dart';
import 'package:tmail_ui_user/features/caching/clients/new_email_hive_cache_client.dart';
import 'package:tmail_ui_user/features/caching/clients/opened_email_hive_cache_client.dart';
import 'package:tmail_ui_user/features/caching/clients/recent_login_url_cache_client.dart';
import 'package:tmail_ui_user/features/caching/clients/recent_login_username_cache_client.dart';
import 'package:tmail_ui_user/features/caching/clients/recent_search_cache_client.dart';
import 'package:tmail_ui_user/features/caching/clients/session_hive_cache_client.dart';
import 'package:tmail_ui_user/features/caching/clients/state_cache_client.dart';
import 'package:tmail_ui_user/features/caching/config/hive_cache_config.dart';
import 'package:tmail_ui_user/features/caching/manager/session_cache_manger.dart';
import 'package:tmail_ui_user/features/caching/utils/cache_utils.dart';
import 'package:tmail_ui_user/features/caching/utils/caching_constants.dart';
import 'package:tmail_ui_user/features/cleanup/data/local/recent_login_url_cache_manager.dart';
import 'package:tmail_ui_user/features/cleanup/data/local/recent_login_username_cache_manager.dart';
import 'package:tmail_ui_user/features/cleanup/data/local/recent_search_cache_manager.dart';
import 'package:tmail_ui_user/features/login/data/local/account_cache_manager.dart';
import 'package:tmail_ui_user/features/login/data/local/authentication_info_cache_manager.dart';
import 'package:tmail_ui_user/features/login/data/local/encryption_key_cache_manager.dart';
import 'package:tmail_ui_user/features/login/data/local/oidc_configuration_cache_manager.dart';
import 'package:tmail_ui_user/features/login/data/local/token_oidc_cache_manager.dart';
import 'package:tmail_ui_user/features/mailbox/data/local/mailbox_cache_manager.dart';
import 'package:tmail_ui_user/features/mailbox/data/local/state_cache_manager.dart';
import 'package:tmail_ui_user/features/mailbox/data/model/state_type.dart';
import 'package:tmail_ui_user/features/mailbox_dashboard/data/local/local_spam_report_manager.dart';
import 'package:tmail_ui_user/features/offline_mode/manager/new_email_cache_manager.dart';
import 'package:tmail_ui_user/features/offline_mode/manager/opened_email_cache_manager.dart';
import 'package:tmail_ui_user/features/offline_mode/manager/sending_email_cache_manager.dart';
import 'package:tmail_ui_user/features/push_notification/data/keychain/keychain_sharing_manager.dart';
import 'package:tmail_ui_user/features/push_notification/data/local/fcm_cache_manager.dart';
import 'package:tmail_ui_user/features/push_notification/presentation/extensions/type_name_extension.dart';
import 'package:tmail_ui_user/features/thread/data/local/email_cache_manager.dart';

class CachingManager {
final MailboxCacheClient _mailboxCacheClient;
final StateCacheClient _stateCacheClient;
final EmailCacheClient _emailCacheClient;
final RecentSearchCacheClient _recentSearchCacheClient;
final RecentLoginUrlCacheClient _recentLoginUrlCacheClient;
final RecentLoginUsernameCacheClient _recentLoginUsernameCacheClient;
final AccountCacheClient _accountCacheClient;
final FcmCacheClient _fcmCacheClient;
final FirebaseRegistrationCacheClient _firebaseRegistrationCacheClient;
final MailboxCacheManager _mailboxCacheManager;
final StateCacheManager _stateCacheManager;
final EmailCacheManager _emailCacheManager;
final RecentSearchCacheManager _recentSearchCacheManager;
final RecentLoginUrlCacheManager _recentLoginUrlCacheManager;
final RecentLoginUsernameCacheManager _recentLoginUsernameCacheManager;
final AccountCacheManager _accountCacheManager;
final FCMCacheManager _fcmCacheManager;
final HiveCacheVersionClient _hiveCacheVersionClient;
final NewEmailHiveCacheClient _newEmailHiveCacheClient;
final OpenedEmailHiveCacheClient _openedEmailHiveCacheClient;
final NewEmailCacheManager _newEmailCacheManager;
final OpenedEmailCacheManager _openedEmailCacheManager;
final FileUtils _fileUtils;
final SendingEmailCacheManager _sendingEmailCacheManager;
final SessionHiveCacheClient _sessionHiveCacheClient;
final SessionCacheManager _sessionCacheManager;
final LocalSpamReportManager _localSpamReportManager;
final KeychainSharingManager _keychainSharingManager;
final TokenOidcCacheManager _tokenOidcCacheManager;
final OidcConfigurationCacheManager _oidcConfigurationCacheManager;
final EncryptionKeyCacheManager _encryptionKeyCacheManager;
final AuthenticationInfoCacheManager _authenticationInfoCacheManager;

CachingManager(
this._mailboxCacheClient,
this._stateCacheClient,
this._emailCacheClient,
this._recentSearchCacheClient,
this._recentLoginUrlCacheClient,
this._recentLoginUsernameCacheClient,
this._accountCacheClient,
this._fcmCacheClient,
this._firebaseRegistrationCacheClient,
this._mailboxCacheManager,
this._stateCacheManager,
this._emailCacheManager,
this._recentSearchCacheManager,
this._recentLoginUrlCacheManager,
this._recentLoginUsernameCacheManager,
this._accountCacheManager,
this._fcmCacheManager,
this._hiveCacheVersionClient,
this._newEmailHiveCacheClient,
this._openedEmailHiveCacheClient,
this._newEmailCacheManager,
this._openedEmailCacheManager,
this._fileUtils,
this._sendingEmailCacheManager,
this._sessionHiveCacheClient,
this._sessionCacheManager,
this._localSpamReportManager,
this._keychainSharingManager,
this._tokenOidcCacheManager,
this._oidcConfigurationCacheManager,
this._encryptionKeyCacheManager,
this._authenticationInfoCacheManager,
);

Future<void> clearAll() async {
await Future.wait([
_stateCacheClient.clearAllData(),
_mailboxCacheClient.clearAllData(),
_emailCacheClient.clearAllData(),
_fcmCacheClient.clearAllData(),
_firebaseRegistrationCacheClient.clearAllData(),
_accountCacheClient.clearAllData(),
_stateCacheManager.clear(),
_mailboxCacheManager.clear(),
_emailCacheManager.clear(),
_fcmCacheManager.clear(),
_accountCacheManager.clear(),
_localSpamReportManager.clear(),
if (PlatformInfo.isMobile)
...[
_sessionHiveCacheClient.clearAllData(),
_newEmailHiveCacheClient.clearAllData(),
_openedEmailHiveCacheClient.clearAllData(),
_sessionCacheManager.clear(),
_newEmailCacheManager.clear(),
_openedEmailCacheManager.clear(),
_sendingEmailCacheManager.clearAllSendingEmails(),
],
if (PlatformInfo.isIOS)
_keychainSharingManager.delete()
], eagerError: true);
}

Future<void> clearData() async {
await Future.wait([
_stateCacheClient.clearAllData(),
_mailboxCacheClient.clearAllData(),
_emailCacheClient.clearAllData(),
_fcmCacheClient.clearAllData(),
_firebaseRegistrationCacheClient.clearAllData(),
_localSpamReportManager.clear(),
if (PlatformInfo.isMobile)
...[
_newEmailHiveCacheClient.clearAllData(),
_openedEmailHiveCacheClient.clearAllData(),
_sendingEmailCacheManager.clearAllSendingEmails(),
]
], eagerError: true);
}

Future<void> clearEmailAndStateCache({AccountId? accountId, UserName? userName}) {
log('CachingManager::clearEmailAndStateCache:userName = $userName');
if (accountId != null && userName != null) {
final emailKey = TupleKey(accountId.asString, userName.value).encodeKey;
final stateKey = StateType.email.getTupleKeyStored(accountId, userName);

return Future.wait([
_emailCacheClient.clearAllDataContainKey(emailKey),
_stateCacheClient.deleteItem(stateKey),
_emailCacheManager.deleteByKey(emailKey),
_stateCacheManager.deleteByKey(stateKey),
if (PlatformInfo.isMobile)
clearFCMEmailStateCache(accountId: accountId, userName: userName),
]);
} else {
final stateKey = StateType.email.getTupleKeyStoredWithoutAccount();
return Future.wait([
_emailCacheClient.clearAllData(),
_stateCacheClient.deleteItem(stateKey),
_emailCacheManager.clear(),
_stateCacheManager.deleteByKey(stateKey),
if (PlatformInfo.isMobile) clearFCMEmailStateCache(),
]);
}
Expand All @@ -131,14 +122,14 @@ class CachingManager {
if (accountId != null && userName != null) {
final emailKey = TupleKey(accountId.asString, userName.value).encodeKey;
return Future.wait([
_newEmailHiveCacheClient.clearAllDataContainKey(emailKey),
_openedEmailHiveCacheClient.clearAllDataContainKey(emailKey),
_newEmailCacheManager.deleteByKey(emailKey),
_openedEmailCacheManager.deleteByKey(emailKey),
clearAllFileInStorage(),
]);
} else {
return Future.wait([
_newEmailHiveCacheClient.clearAllData(),
_openedEmailHiveCacheClient.clearAllData(),
_newEmailCacheManager.clear(),
_openedEmailCacheManager.clear(),
clearAllFileInStorage(),
]);
}
Expand All @@ -155,8 +146,10 @@ class CachingManager {

Future<void> clearMailboxCache() {
return Future.wait([
_stateCacheClient.deleteItem(StateType.mailbox.getTupleKeyStoredWithoutAccount()),
_mailboxCacheClient.clearAllData(),
_stateCacheManager.deleteByKey(
StateType.mailbox.getTupleKeyStoredWithoutAccount(),
),
_mailboxCacheManager.clear(),
], eagerError: true);
}

Expand All @@ -169,9 +162,8 @@ class CachingManager {
return _hiveCacheVersionClient.getLatestVersion();
}

Future<void> closeHive() async {
return await HiveCacheConfig.instance.closeHive();
}
Future<void> closeHive({bool isolated = true}) =>
HiveCacheConfig.instance.closeHive(isolated: isolated);

Future<void> clearAllFileInStorage() async {
await _fileUtils.removeFolder(CachingConstants.openedEmailContentFolderName);
Expand All @@ -180,26 +172,50 @@ class CachingManager {

Future<void> clearFCMEmailStateCache({AccountId? accountId, UserName? userName}) async {
if (accountId != null && userName != null) {
await _fcmCacheClient.deleteItem(
await _fcmCacheManager.deleteByKey(
TypeName.emailDelivery.getTupleKeyStored(accountId, userName));
await _fcmCacheClient.deleteItem(
await _fcmCacheManager.deleteByKey(
TypeName.emailType.getTupleKeyStored(accountId, userName));
} else {
await _fcmCacheClient.deleteItem(
await _fcmCacheManager.deleteByKey(
TypeName.emailDelivery.getTupleKeyStoredWithoutAccount());
await _fcmCacheClient.deleteItem(
await _fcmCacheManager.deleteByKey(
TypeName.emailType.getTupleKeyStoredWithoutAccount());
}
}

Future<void> clearLoginRecentData() async {
await Future.wait([
_recentLoginUrlCacheClient.clearAllData(),
_recentLoginUsernameCacheClient.clearAllData(),
_recentLoginUrlCacheManager.clear(),
_recentLoginUsernameCacheManager.clear(),
]);
}

Future<void> clearRecentSearchData() async {
await _recentSearchCacheClient.clearAllData();
await _recentSearchCacheManager.clear();
}

Future<void> migrateHiveToIsolatedHive() async {
await Future.wait([
_tokenOidcCacheManager.migrateHiveToIsolatedHive(),
_accountCacheManager.migrateHiveToIsolatedHive(),
_stateCacheManager.migrateHiveToIsolatedHive(),
_mailboxCacheManager.migrateHiveToIsolatedHive(),
_emailCacheManager.migrateHiveToIsolatedHive(),
_fcmCacheManager.migrateHiveToIsolatedHive(),
_oidcConfigurationCacheManager.migrateHiveToIsolatedHive(),
_encryptionKeyCacheManager.migrateHiveToIsolatedHive(),
_authenticationInfoCacheManager.migrateHiveToIsolatedHive(),
_recentLoginUrlCacheManager.migrateHiveToIsolatedHive(),
_recentLoginUsernameCacheManager.migrateHiveToIsolatedHive(),
_recentSearchCacheManager.migrateHiveToIsolatedHive(),
if (PlatformInfo.isMobile)
...[
_sessionCacheManager.migrateHiveToIsolatedHive(),
_sendingEmailCacheManager.migrateHiveToIsolatedHive(),
_newEmailCacheManager.migrateHiveToIsolatedHive(),
_openedEmailCacheManager.migrateHiveToIsolatedHive(),
]
]);
}
}
2 changes: 1 addition & 1 deletion lib/features/caching/config/cache_version.dart
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@

class CacheVersion {
static const int hiveDBVersion = 16;
static const int hiveDBVersion = 17;
}
19 changes: 19 additions & 0 deletions lib/features/caching/config/fcm_isolate_name_server.dart
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
import 'dart:ui' as flutter;

import 'package:hive_ce/hive.dart';

class FcmIsolateNameServer extends IsolateNameServer {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it uses for what?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is a mechanism for sharing ports between isolates. IsolateNameServer allows registering and looking up a SendPort by name, allowing the secondary isolate to connect to the primary isolate or vice versa without passing the port through the constructor.

const FcmIsolateNameServer();

@override
dynamic lookupPortByName(String name) =>
flutter.IsolateNameServer.lookupPortByName(name);

@override
bool registerPortWithName(dynamic port, String name) =>
flutter.IsolateNameServer.registerPortWithName(port, name);

@override
bool removePortNameMapping(String name) =>
flutter.IsolateNameServer.removePortNameMapping(name);
}
Loading
Loading