Release notes from HPI

v0.5.20241019: switch min version to 3.9

2024-10-19T18:17:32Z

What's Changed

general: python3.9 reached EOL, switch min version by @karlicoss in #400

Full Changelog: v0.5.20240924...v0.5.20241019

hotfix for my.core.time

2024-09-23T21:05:57Z

What's Changed

core.time: hotfix for default force_abbreviations attribute by @karlicoss in #399

Full Changelog: v0.5.20240923...v0.5.20240924

rolling release

2024-09-23T20:09:51Z

What's Changed

core

my.core: fix list constructor in always_support_sequence and add some tests by @karlicoss in #395
my.core.structure: add support for .tar.gz archives by @karlicoss in #390

modules

my.google.takeout.parser: speedup event merging on newer google_takeout_parser versions by @karlicoss in #389
my.github.gdpr/my.zulip.organization: use kompress support for tar.gz if it's available by @karlicoss in #392
my.fbmessenger.android: exclude unsent messages to avoid duplication by @karlicoss in #393
migrate bluemaestro and pdfs configs to use Protocol and @Property by @karlicoss in #384
migrate some modules to 'lazy config' pattern and clean up tests by @karlicoss in #386
small fixes for my.twitter.android, dedupicate tweets in my.twitter.archive by @karlicoss in #394
my.youtube.takeout: deduplicate watched videos and sort out a few minor errors by @karlicoss in #397

misc

update ruff config, enable/suppress some checks by @karlicoss in #387
ci: update mypy config and make ruff config more consistent with other projects by @karlicoss in #388
tox: some prep for potentially using uv on CI instead of pip by @karlicoss in #396
tox: try using uv for CI, should result in speedup by @karlicoss in #398

Full Changelog: v0.5.20240824...v0.5.20240923

rolling release

2024-08-24T11:28:57Z

What's Changed

Mostly big cleanup & deprecations for core.common module #380 #382

New core dependency: typing-extensions

Lots of cleanup/deprecations in core. This mostly shouldn't break any runtime code, but mypy might complain -- so best to migrate the imports appropriately

core.common: deprecate outdated LazyLogger alias
core.common: move Json, datetime_aware, datetime_naive, is_namedtuple, asdict to my.core.types
my.core: deprecate Path/dataclass imports from my.core during type checking

runtime still works for backwards compatibility
core.common: move assert_subpackage to my.core.internal
core.common: move mime-related stuff to my.core.mime

no backward compat, unlikely it was used by anyone else
core.common: move stats-related stuff to my.core.stats and add more thorough tests/docs

deprecate core.common.stat and core.common.Stats with backwards compatibility
core: cleanup my.core.common.unique_everseen
- move to my.core.utils.itertools
- more robust check for hashable types -- now checks in runtime (since the one based on types purely isn't necessarily sound)
- add more testing
my.core.common: move warn_if_empty to my.core.utils.itertools, cleanup and add more tests
core.common: move away import related stuff to my.core.utils.imports

moving without backward compatibility, since it's extremely unlikely they are used for any external modules

in fact, unclear if these methods still have much value at all, but keeping for now just in case
core.common: move listify to core.utils.itertools, use better typing annotations for it

also some minor refactoring of my.rss
core: cleanup itertool style helpers
- deprecate group_by_key, should use itertool.bucket instead
- move make_dict and ensure_unique to my.core.utils.itertools
core: cleanup deprecations, exclude from type checking and show runtime warnings

among affected things:
- core.common.assert_never
- core.common.cproperty
- core.common.isoparse
- core.common.mcachew
- core.common.the
- core.common.tzdatetime
- core.compat.sqlite_backup
core: cleanup deprecations, exclude from type checking and show runtime warnings

among affected things:
- core.common.assert_never
- core.common.cproperty
- core.common.isoparse
- core.common.mcachew
- core.common.the
- core.common.tzdatetime
- core.compat.sqlite_backup

Full Changelog: v0.4.20240810...v0.5.20240824

rolling release

2024-08-10T16:47:47Z

What's Changed

fix for new instagram gdpr format by @karlicoss in #371
fix legacy google takeout by @karlicoss in #372
fbmessenger.android: fix minor issue with processing thread participants by @karlicoss in #373
fix my.twitter.android by @karlicoss in #374
tests: use updated conftest from pymplate, this allows to run individual test modules properly by @karlicoss in #375
improve testing consistency, simplify tox by @karlicoss in #376
replace deprecated utcnow and utcfromtimestamp by @karlicoss in #378

Full Changelog: v0.4.20240506...v0.4.20240810

rolling release

2024-06-05T21:49:04Z

What's Changed

monzo.monzoexport: initial module by @karlicoss in #346
time.tz.via_location: more consistent behaviour wrt caching by @karlicoss in #351
my.codeforces/my.topcoder: move to the top level & cleanup by @karlicoss in #352
my.body.sleep: massive speedup for average temperature calculation by @karlicoss in #353
gpslogger: catch broken xml file error by @seanbreckenridge in #356
initial module for official twitter app for android by @karlicoss in #359
my.youtube.takeout: fix exception handling by @karlicoss in #361
fbmessenger.android: expose contacts by @karlicoss in #362
whatsapp.android: expose all entities extracted from the db by @karlicoss in #363
google takeout: support multiple locales, resolves #330 by @seanbreckenridge in #364
my.google.maps: intitial module for extracting saved places data from Android app by @karlicoss in #365
my.twitter.android: better detection of own user id by @karlicoss in #368
my.tinder.android: better error handing and fix case with empty db by @karlicoss in #369
smscalls: parse mms from smscalls export by @seanbreckenridge in #370

Full Changelog: v0.4.20231102...v0.4.20240506

v0.4.20231102, fix release script

2023-11-01T01:58:48Z

not ideal that the date in the version is the future date.. but oh well

What's Changed

ci: fix wrong release command by @karlicoss in #344

Full Changelog: v0.4.20231101...v0.4.20231102

v0.4.20231101

2023-11-01T01:55:54Z

Big update, even though things should be backwards compatible, it's recommended to install the package again if you use editable git checkout (pip3 install --user -e /path/to/hpi_repo)

Also worth installing optional dependencies (pip3 install --user -e /path/to/hpi_repo[optional])

What's Changed

Core:

core: implement more methods for ZipPath and better support for get_files by @karlicoss in #285
query: add --warn-exceptions, dateparser, docs by @seanbreckenridge in #290
my.core.pandas: rely on typing annotations from types-pandas by @karlicoss in #292
core: experimental import of my._init_hook to configure logging/warnings/env variables by @karlicoss in #298
core/logging: overhaul and many improvements -- mainly to deprecate abandoned logzero by @karlicoss in #299
minor module updates + logging/warnings improvements by @karlicoss in #300
general: move reddit tests into my/tests + tweak my.core.cfg to be more reliable by @karlicoss in #295
core/cli: allow user to bypass PEP 668 by @seanbreckenridge in #302
core/cli: update vendored completion files by @seanbreckenridge in #304
core/logging: fix issue with logger setup called multiple times when called with different levels by @karlicoss in #310
my.core.logging: compatibility with HPI_LOGS by @seanbreckenridge in #307
core/stats: exclude contextmanagers from guess_stats by @karlicoss in #334
move hpi specific compat stuff to core.hpi_compate + minor updates for hypothesis and kobo by @karlicoss in #335
core/config: implement a warning if config is imported from the dir other than MY_CONFIG by @karlicoss in #336
general: improve logging during file processing in various modules by @karlicoss in #339
general: enhancle logging for various modules by @karlicoss in #340
general: deprecate some old methods by hiding behind TYPE_CHECKING by @karlicoss in #342
core: add helper for more_iterable to check that all types involved are hashable by @karlicoss in #343
core/kompress: move vendorized kompress to _deprecated, use kompress library directly by @karlicoss in #319
more consistent handling of zip archives in get_files, less kompress boilerplate in modules by @karlicoss in #321
experimental: add a hacky helper to import "original/shadowed" modules from within overlays by @karlicoss in #322
core/stats: less duplication in output, report stats of first item, report input stats by @karlicoss in #324
core.logging: ignore CollapseLogsHandler if we're not attached to a terminal by @karlicoss in #329
dcotor: make compileall check a bit more defensive by @karlicoss in #332

modules:

my.whatsapp.android: initial module by @karlicoss in #284
smscalls: make some fields optional by @seanbreckenridge in #287
cli: add option to output locations to gpx files by @seanbreckenridge in #286
smscalls: make checking for keys stricter by @seanbreckenridge in #289
browser: fix duplicate logs when fetching loglevel by @seanbreckenridge in #309
new module: Harmonic app for Hackernews by @karlicoss in #313
location: make accuracy default config floats by @seanbreckenridge in #314
reddit.rexport: big cleanup + use new types from rexport.dal by @karlicoss in #320
instagram: better normalising/error handling by @karlicoss in #325
updates for whatsapp.android and tinder.android by @karlicoss in #326
twitter.archive: deduplicate results via json.dumps by @karlicoss in #327
reddit.rexport: experiment with using optional cpu pool (can potentially be used by other HPI modules) by @karlicoss in #328
small zulip and dogsheep updates by @karlicoss in #331
my.emfit: cleanup and pass cpu pool by @karlicoss in #333
my.stackexchange.stexport: small cleanup & stat improvements by @karlicoss in #337
fbmessenger.android: support processing msys database by @karlicoss in #341

Full Changelog: v0.3.20230327...v0.4.20231101

release

2023-03-27T02:28:29Z

What's Changed

vk_messages_backup updates by @karlicoss in #274
location fallback by @seanbreckenridge in #263
add example for denylist, update ci by @seanbreckenridge in #275
core.logging/core.kompress updates by @karlicoss in #277
google_takeout: add semantic location history by @seanbreckenridge in #278
cli completion doc updates, hide legacy import warning by @seanbreckenridge in #279
some updates for bumble/fbmessenger/twint by @karlicoss in #280
better unification of my.instagram and various enhancements by @karlicoss in #282
docs: fix typos by @kianmeng in #281
telgram_backup updates by @karlicoss in #283

Full Changelog: v0.3.20230221...v0.3.20230327

release

2023-02-21T02:45:20Z

What's Changed

my.telegram: initial module by @karlicoss in #270
my.fbmessenger.android updates by @karlicoss in #271
my.telegram.telegram_backup enhancements by @karlicoss in #273

Full Changelog: v0.3.20230209...v0.3.20230221