Release notes from pyterrier

0.13.0

2025-01-13T22:32:22Z

What's Changed

A new feature release, demonstrating the new Artifact API, and making a fix in when fields are created in Terrier indexes by IterDictIndexer.

Significant improvements:

New feature: Artifact API by @seanmacavaney in #436 - we'll be improving this feature and its documentation in future releases.
Improvement: terrier.IterDictIndexer doesn't create a field-index when not requested by @cmacdonald in #525

Minor changes:

bibliography file in documentation by @seanmacavaney in #519
ruff github action by @seanmacavaney in #523
Remaining mypy errors by @seanmacavaney in #524
RemoteDataset: get one file from a zip by @cmacdonald in #529
use decorate_batch properly in Terrier by @cmacdonald in #528
saves process name such that jps can shows a name for Pyterrier processes by @cmacdonald in #527

Full Changelog: 0.12.1...0.13.0

0.12.1

2024-12-19T12:56:20Z

0.12.1 - released 19/12/2024

Wrapping up various improvements developed in the last few weeks.

New Feature:

Common prefix pipeline computation in pt.Experiment by @cmacdonald, with @seanmacavaney and @Parry-Parry in #514

Improvements:

PRF compilation by @cmacdonald in #504
More compilation improvements by @cmacdonald in #506
pt.debug.pdb by @seanmacavaney in #510
Guessing batch size for indexing pipelines by @seanmacavaney in #510

Minor:

Dataset Decompression Improvements by @cmacdonald in #512
support verbose=True on by_query iter by @cmacdonald in #513
pass kwargs through to the underlying open functions by @cmacdonald in #518

Documentation:

more extension docs by @seanmacavaney in #511
Transformer docs by @cmacdonald in #507

Full Changelog: 0.12.0...0.12.1

0.12.0

2024-12-05T18:10:08Z

What's Changed in PyTerrier 0.12.0

0.12.0 has an API change for Transformer, making it easier to both implement and call Transformers using iter-dicts rather than DataFrames (DataFrame support is also maintained). This release also completely refreshes the .compile() implementation, making it easier to implement pipelines that can be optimised.

Major:

API change: Transformer.transform_iter() returns an iter-dict generator by @cmacdonald in #481
Transformer.compile() improvements by @seanmacavaney in #480
deprecate caching operator ~ by @cmacdonald in #483
more extensions with integrated documentation by @seanmacavaney in #503

Minor:

addresses #423 IRDS warnings on pt.list_datasets() by @cmacdonald in #485
bump maven plugin versions by @cmacdonald in #495
integrated extension documentation by @seanmacavaney in #497
move to pyproject.toml by @cmacdonald in #490
Auto-generated Citations from DBLP by @seanmacavaney in #498
dont consume arbitrary unused kwargs in TerrierIndexer, update overridden properties by @cmacdonald in #500
More types by @cmacdonald in #484
Add Apple Silicon GHAs by @cmacdonald in #478

Full Changelog: 0.11.0...0.12.0

0.11.0

2024-08-23T16:21:13Z

What's Changed

Significant update that has refactored much of the PyTerrier source code and renamed many classes as we progress towards a PyTerrier 1.0 release.

The most significant changes are:

pt.init() is no longer required 😃. If necessary pt.java methods can be used to change Java initialisation
pt.BatchRetrieve is now pt.terrier.Retriever, and similar changes for other Terrier indexers and retrievers
pt.AnseriniBatchRetrieve is now in its own separate project, PyTerrier-Anserini, with various improvements

All changes are backwards compatible in this release - deprecation warnings will guide you how to update your code.

More details below:

Improvements

Move all Java/JNIUS code into pt.java, move all Terrier code into pt.terrier; remove pt.init() by @seanmacavaney in #447
dynamic module loading by @seanmacavaney in #461
Incorporate Retrieval Scores into RM3 by @mam10eks in #453
pt.apply for making an indexer by @cmacdonald in #467
query_toks support for terrier.Retriever by @cmacdonald in #466
add save_mode='warn' and save_mode='error' to pt.Experiment (warn as default) by @cmacdonald in #408

### Refactoring

Deprecate DFIndexer by @cmacdonald in #457
pt.terrier.rewrite revisions - remove Axiomatic, remove terrier-prf by @seanmacavaney in #472
shims for deprecated modules by @seanmacavaney in #476
text_loader abstraction for pt.text.get_text by @seanmacavaney in #469
move Anserini to a separate project by @seanmacavaney in #473

Documentation

Add RankVicuna and RankZephyr Plugins by @kaustubhdhole in #441
Update tuning.rst by @albertoueda in #446
Add PyTerrier_ChatNoir to the plugin section by @mam10eks in #452
Remove nptyping dependency to assure numpy 2 compatability by @cmacdonald in #445

Minor

change all tests to use new terrier retriever names, but check old names too by @cmacdonald in #458
Parallel fixes by @seanmacavaney in #462
fix logger error by @seanmacavaney in #464
Add comments to requirements.txt by @cmacdonald in #465
failing anserini tests due to version 0.36.0, disabling for now by @seanmacavaney in #468
remove the writing of a default terrier.properties file by @cmacdonald in #470
fix test_maven by @seanmacavaney in #471
Python 3.12 in GHA by @cmacdonald in #459
Bump most JDK version tested in GHA to 21 by @cmacdonald in #475
Update pt.terrier.Retriever str and repr #474

New Contributors

@kaustubhdhole made their first contribution in #441
@mam10eks made their first contribution in #452

Full Changelog: 0.10.1...0.11.0

0.10.1

2024-05-02T18:14:55Z

Minor release with minor improvements and bug fixes.

What's Changed

Bugfix: Delete baseline pvalue from correction method input by @JorgeGabin in #440
Fix: fix msmarco location by @cmacdonald in #435
Feature: added corpus_iter for Terrier index by @cmacdonald in #426
remove sklearn as required dependency by @cmacdonald in #410
Add troubleshoot for installation and certification error by @Krissy510 in #411
fix parsing of trecxml topics by @lukaszett in #414
paired t-tost by @seanmacavaney in #420
read_results optimization by @seanmacavaney in #421
pickling QE pipelines to parallelised QE gridsearch by @cmacdonald in #430
Require Python 3.8 minimum by @cmacdonald in #431
Bump logback from 1.2.0 to 1.2.13 in /terrier-python-helper by @dependabot
improved error message pt.apply.query - from #433 by @cmacdonald in #434
Improved testing of FeaturesBatchRetrieve by @cmacdonald in #437

New Contributors

@Krissy510 made their first contribution in #411
@JorgeGabin made their first contribution in #440

Full Changelog: 0.10.0...0.10.1

0.10.0

2023-11-02T17:20:50Z

What's Changed

New Features

Transformer.__call__ now supports both dataframe and iterdicts by @cmacdonald in #381
Terrier: Custom stopwords by @cmacdonald in #372
Terrier: Access the stemmer of Terrier from PyTerrier by @cmacdonald in #382
Terrier: Improved API for loading Terrier indices into memory by @cmacdonald in #386

Improvements

added tokenizer as arg for pt.text.sliding by @mihirs16 in #387
addresses #367 - include qid in pt.apply Exception by @cmacdonald in #370
addresses #377: pt.apply.query() raises exception if the query column does not exist by @cmacdonald in #380
let pt.tqdm exist without pt.init() by @cmacdonald in #399
deprecate pt.Utils by @cmacdonald in #384
removes two warnings by @cmacdonald in #385
work on test failure by @cmacdonald in #401
Test pyterrier with newer Python versions by @cmacdonald in #400
bump supported Anserini version by @cmacdonald in #406, addresses #404
Terrier: allow to put term and LexiconEntry into a tuple by @cmacdonald in #369

Bugs:

stringify properties and controls, addresses #357 by @cmacdonald in #358
fix bug in metadata size warning by @seanmacavaney in #362

Documentation

Update pipeline_examples.md by @gurcankavakci in #359
Fixed typo by @hermlon in #364
Update ltr.rst by @Hermi-Mire in #371
Update transformer.rst by @albertoueda in #383
clarify docstring for indexing with regards to metadata by @lukaszett in #394
Query Rewriting & Expansion by @cakiki in #402, #403

New Contributors

@gurcankavakci made their first contribution in #359
@hermlon made their first contribution in #364
@Hermi-Mire made their first contribution in #371
@lukaszett made their first contribution in #394
@cakiki made their first contribution in #402
@mihirs16 made their first contribution in #387

Full Changelog: 0.9.2...0.10.0

0.9.2

2022-12-19T12:31:05Z

Minor release with minor improvements and bug fixes.

What's Changed

add sbert example notebook by @cmacdonald in #344
Update scikit-learn requirement from the deprecated sklearn, which was causing build errors at some times.
adding batching operations to apply.generic() and apply.by_query() by @cmacdonald in #351 - thanks to Xun Zhou, University of Michigan via #350
improve error messages for invalid indexing configurations by @cmacdonald in #349 -- thanks to @maxhenze in #348
Various empty dataframe fixes by @cmacdonald in #353 -- thanks to report by Prithvijit Dasgupta, University of Michigan in #352
improved error message for add_ranks by @cmacdonald in #354

Full Changelog: 0.9.1...0.9.2

0.9.1

2022-11-11T15:20:23Z

Bugfix release addressing a problem with pretokenised indices on Windows

What's Changed

Nofifo pretok indexing fixes by @cmacdonald in #343

Full Changelog: 0.9.0...0.9.1

0.9.0

2022-11-10T19:23:23Z

Significant update - refactoring of public API (e.g. pt.transformer.TransformerBase -> pt.Transformer) and support in the Terrier backend for making indices from pre-tokenised documents. Python 3.10 is now supported.

What's Changed

fix error in IRDSDataset when a query field is named "query" by @seanmacavaney in #303
Fix type annotation by @heinrichreimer in #313
addresses #315 IRDS corpus_iter are not subscriptable by @cmacdonald in #316
Missing comma in bm25_qe example by @JohnGiorgi in #319
Argument meta should be supplied as dictionary by @JohnGiorgi in #320
use Jnius 1.4 by @cmacdonald in #249
Python 3.10 support by @cmacdonald in #322
Lz4 support for pt.io.autoopen() by @cmacdonald in #323
addresses #326 faster version of add_ranks for single queries by @cmacdonald in #327
addresses #321 pt.apply.doc_score batching by @cmacdonald in #325
IterDictIndexer can index pre-tokenised documents by @cmacdonald in #328
Bump logback-core from 1.2.0 to 1.2.9 in /terrier-python-helper by @dependabot in #336
documenting BM25F controls and tuning by @cmacdonald in #296, addresses #294
0.9refactor by @cmacdonald in #314, #339, addresses #271
pt.Experiment() alters the input measures list to drop "mrt" #301
Expose Termpipelines in Terrier index backend by @cmacdonald in #338
pt.rewrite.tokenise() impl by @cmacdonald in #340 addresses #252 #253
upgraded GitHub actions by @cmacdonald in #341, #342
fix LTR groupby for xgboost & lightgbm by @cmacdonald in #284

New Contributors

@heinrichreimer made their first contribution in #313
@JohnGiorgi made their first contribution in #319

Full Changelog: 0.8.1...0.9.0

0.8.1

2022-04-10T08:56:23Z

Minor release with minor improvements and bug fixes.

What's Changed

fixed bug with is_transformer by @seanmacavaney in #274
addresses #275 issue k in kmaxavg, improved testing by @cmacdonald in #276
defer loading ir_datasets by @seanmacavaney in #280
Set meta and meta_lengths in constructor by @MWschutte in #282
Anserini fixes by @cmacdonald in #279, reported by @Azouu
prevent use of nptyping v2 by @cmacdonald in #291, reported by @tabonnet
SourceTransformer pass through extra columns, addresses #287 by @cmacdonald in #288, reported by @Xiao0728
more transformers with repr by @cmacdonald in #289

New Contributors

@MWschutte made their first contribution in #282

Full Changelog: 0.8.0...0.8.1