-
Notifications
You must be signed in to change notification settings - Fork 763
Release Notes Heritrix 3.4.0 20220727
Alex Osborne edited this page Jul 30, 2022
·
15 revisions
Summary of changes since Release Notes - Heritrix 3.4.0-20210923 - see the full changelog for more details.
- JDK 18 compatibility
- DNS over HTTPS support
- SOCKS5 proxy support
- robotsTxtOnly robots policy which obeys robots.txt rules but ignores HTML robots meta tags
- CandidatesProcessor gained a
seedsRedirectNewSeedsAllowTLDs
option to disallow bare top level domains being added as additional seeds when an initial seed redirects to them #461 - Configurable URL matching and extraction for sitemaps #441
- Dependencies updated:
- Spring Framework 5.3.20
- JSch 0.1.52
- gson 2.8.9
- Dependencies removed:
- JNA (symlink creation is now done using the standard Java API)
- Fixed HTML srcset attribute only matching in lowercase #477
- Fixed sitemap links (M) being considered transclusions when limiting hop depth #469
- Fixed "java.lang.NoClassDefFoundError: Could not initialize class org.archive.util.CLibrary" on Apple Silicon #467
- Fixed Heritrix crashing on unexpected characters in the Content-Length header #449
- Fixed StringIndexOutOfBoundsException on exact major Java versions like the first JDK 18 release #439
- Fixed dnsjava NIO selector thread consuming 100% CPU after terminating job #425
- Fixed
<link href=...>
tags being treated as embed (E) links for rel values where they shouldn't be #263 - Fixed "RIS already open for ToeThread..." exception when crawling https pages via a proxy #191
- Fixed setting maxLogFileSize in BDBModule #464
- Fixed group id is too big error when building on some systems #448
- Fixed CrawlURI.hashCode() NullPointException sometimes breaking Browse Beans #488
- Fixed ownership conflict over /tmp/Crashpad when running the Chrome Extractor under different users
Structured Guides:
User Guide
- Introduction
- New Features in 3.0 and 3.1
- Your First Crawl
- Checkpointing
- Main Console Page
- Profiles
- Heritrix Output
- Common Heritrix Use Cases
- Jobs
- Configuring Jobs and Profiles
- Processing Chains
- Credentials
- Creating Jobs and Profiles
- Outside the User Interface
- A Quick Guide to Creating a Profile
- Job Page
- Frontier
- Spring Framework
- Multiple Machine Crawling
- Heritrix3 on Mac OS X
- Heritrix3 on Windows
- Responsible Crawling
- Adding URIs mid-crawl
- Politeness parameters
- BeanShell Script For Downloading Video
- crawl manifest
- JVM Options
- Frontier queue budgets
- BeanShell User Notes
- Facebook and Twitter Scroll-down
- Deduping (Duplication Reduction)
- Force speculative embed URIs into single queue.
- Heritrix3 Useful Scripts
- How-To Feed URLs in bulk to a crawler
- MatchesListRegexDecideRule vs NotMatchesListRegexDecideRule
- WARC (Web ARChive)
- When taking a snapshot Heritrix renames crawl.log
- YouTube
- H3 Dev Notes for Crawl Operators
- Development Notes
- Spring Crawl Configuration
- Build Box
- Potential Cleanup-Refactorings
- Future Directions Brainstorming
- Documentation Wishlist
- Web Spam Detection for Heritrix
- Style Guide
- HOWTO Ship a Heritrix Release
- Heritrix in Eclipse