Tags: hyphanet/plugin-WebOfTrust
Tags
Web of Trust Version 0.4.5 build0020 ------------------------------------------------------------------------ SUMMARY (detailed changelog is below) This is an intermediate maintenance build which caters mostly to developers, testers and paranoid users. Especially Freenet core developers shall benefit from its two primary goals: 1) It prepares WoT for the removal of the library freenet-ext.jar and its contained db4o library from Freenet. It does so by bundling db4o inside WoT's JAR and making WoT independent from the naming of the JARs which Freenet depends on. 2) The cloud service Travis CI, which was previously only used for running WoT unit tests, now automatically uploads the resulting WoT binary to Freenet after it has compiled and tested it. Users and testers can also benefit from aspect 2 by using it to have an independent service validate that the published WoT binary matches the source code, see the below full changelog at issue 0007018. The bundling of db4o should in theory work on existing Freenet installations where db4o is both contained in Freenet and in WoT - but it would be nice to have some confirmation from testers that this does indeed work on various operating systems and Java versions. So feedback on whether it works on your installation would be appreciated. TESTING Make sure to backup the WebOfTrust directory (located in the directory Freenet is installed to) before you use this! Easy high level testing: - Check the Community / Statistics page for unusual stuff. E.g.: - Data loss, e.g. decreasing count of own identities, non-own identities or trust values as compared to your database before installation of the testing version. - High count of "Failed files" at the Identity file queue / Identity file processor section. - No significant progress in average downloaded / queued / processed identity XML files even after a day of uptime. Tedious low level testing: - Check "freenet-latest.log" for ERRORs of plugins.WebOfTrust.* You can filter for this by configuring logging as: - Set "Minimum priority to log messages at" to "NONE". - Set "Detailed priority thresholds" to "plugins.WebOfTrust:ERROR". Now any logging except WoT errors is disabled. Besides you should probably increase "Log rotation interval" to "48HOUR" (yes, no S in HOUR), the default is very low, 1 hour IIRC. Before you report anything from the log file make VERY sure to review it for private data! It might even contain the secret insert URIs of your identities! DOWNLOAD Until it has been shipped with a Freenet release this build is available using "WebOfTrust Testing Versions" at "Configuration / Plugins" of a standard Freenet node. You need to unload the regular WoT first! CHANGELOG - prefixed with the bugtracker issue number - 0007018: [Code quality] Travis CI: Upload WebOfTrust.jar to Freenet using pyFreenet (xor) Travis CI is a cloud service which allows us to automatically compile WoT from the source code and run the unit tests. WoT's Travis script has been amended to spawn a Freenet node and upload the compiled WoT binary to Freenet. This can be a great stress reduction for release managers as the trust they need to put into the security of their machine is greatly reduced if they can choose to build the binaries elsewhere. The overall security of that might be a bit better because Travis is a large, widely used service and thereby they have more resources at hand to ensure a secure infrastructure. Further users can use it as a third-party verification that we didn't put secret code into the binaries before we uploaded them (see below). To obtain the Freenet URI of an uploaded WoT JAR search the raw Travis job log for "CHK@". Jobs are at: https://travis-ci.org/xor-freenet/plugin-WebOfTrust/builds https://travis-ci.org/freenet/plugin-WebOfTrust/builds For every build there are usually different jobs for various Java versions. For WoT testing releases I will be using the CHK as compiled by OpenJDK in the version which is the minimal requirement of Freenet at time of the release. For this WoT release that is OpenJDK 7. The JAR will only be uploaded if compiling and testing did succeed, otherwise there will be no CHK@ in the output. If Freenet release managers choose to use Travis CI for compiling then the CHK@ URI which they put into fred will match what Travis CI has published in the build job for the git tag build0020 (= commit "Version 0.4.5 build0020") of this release. As the CHK contains a strong hash of the file it being identical proves that it indeed serves the same file as Travis CI produced. Once testing of this build is finished and it has been released as part of a regular Freenet release you can check the CHK@ in the following file of the fred repository: src/freenet/pluginmanager/OfficialPlugins.java which is hosted at https://git.io/vhcsV While this release is still in testing verification works differently because the testing URI is an USK: Download the Travis CHK and the testing USK (listed in the above file) and compare a checksum of the resulting file, e.g. with the sha256sum command. - 0006737: [Code quality] Prepare for fred's removal of freenet-ext.jar and db4o Freenet build01482 is scheduled to split up freenet-ext.jar into multiple JARs, and during this procedure remove db4o from the set of JARs which Freenet ships. To prepare for that, the following has been done: - the WoT build script has been amended to bundle db4o in WoT's own JAR. To achieve that a db4o-7.4 repository has been extracted from the Freenet "contrib" repository. WoT's build script has been amended to automatically clone that git repository and compile db4o locally. To ensure the least possible effort is required by Freenet release managers in validating that these db4o binaries match what had been deployed as part of freenet-ext previously, the following steps were taken: - the db4o-7.4 repository is pulled by the build script using a "git submodule" which ensures that the specific commit which is pulled is hardcoded into the WoT repository and can only be changed manually by whoever has push access to the WoT repository hosted by Freenet. This means that I can host the db4o-7.4 repository on my own GitHub account (xor-freenet) WITHOUT being able to change what people get when they pull WoT from Freenet's GitHub. In turn Freenet does not have to host a clone of the db4o repository on their own. - the db4o-7.4 repository was extracted from Freenet's contrib repository, not from the original db4o SVN. This allows Freenet release managers to simply use: $ git verify-tag v29 $ gitk v29..HEAD to see what has changed between the deployed freenet-ext and the new db4o-7.4 repository. Keeping the db4o version which WoT uses as close to what it has been using previously is critically important because my test runs have shown that newer db4o versions trigger severe bugs. That is also why the repository is called "db4o-7.4" instead of just "db4o" - to document that WoT currently needs precisely that version. Once I am ready to deal with fixing the bugs triggered by newer versions I will publish the original db4o SVN repository as a git repository called "db4o" and switch WoT to use that. So the db4o commit history is not lost yet :) - WoT's build script has been changed to not hardcode any specific filenames of JARs which fred and thereby WoT needs as dependencies. Instead all JARs in the directory ../fred/build/output/ are included on the classpath when building WoT. That directory is populated if fred is built with: $ ./gradlew jar copyRuntimeLibs To allow you to test whether the WoT builder finds all JARs a new Ant task was introduced: $ ant print-libs It should show all runtime dependencies of fred (see its Gradle build file), plus "junit4.jar", "hamcrest-core.jar" and optionally "cobertura.jar" if using "ant -Dtest.coverage=true". (The db4o JAR won't be shown as that is not an external dependency, i.e. WoT obtains it automatically, you don't have to provide it.) If compiling fails due to missing classes see the "Configuration" section at top of "build.xml" for the configured locations - all external JARs have been moved to that section. - 0006929: [Bugs] Travis CI builds fail due to changes at fred (xor) The .travis.yml configuration of Travis CI has been fixed to be compatible to the huge changeset of the Freenet core fred which is being developed on fred's next branch. Specifically it was adapted to be compatible with the new Gradle builder of fred which has replaced Ant there - while keeping Ant at WoT so we don't have too much migration efforts going on for now. The unit tests of WoT do succeed against the new fred code so hopefully WoT won't delay the release of the fred changes any further than they have already been delayed. This also the first time where the "CI" = continuous integration in Travis CI is really happening to meet its definition: I don't have a Gradle build setup for fred branch next yet so it is of true benefit to be able to test it elsewhere :) - 0007016: [Bugs] Fix unit test failure on Java 9 (xor) Fixes a bug in the unit tests, not in the actual code. Symptom was: junit.framework.AssertionFailedError: static final boolean plugins.WebOfTrust.Persistent.$assertionsDisabled expected not same - 0007017: [Bugs] Fix unit test failure on Java 9 (xor) Fixes a bug in the unit tests, not in the actual code. Symptom was: testExportIntroduction() would fail due to the Java 9 XML generation code changing the way indentation was produced. THANKS TO - ArneBab for being willing to review and deploy WoT. - ArneBab for providing pyFreenet which greatly helped in automatically uploading the WoT JAR using Travis CI.
Web of Trust Version 0.4.5 build0019 ------------------------------------------------------------------------ SUMMARY AND OUTLOOK (detailed changelog is below): As the previous build was mostly new code to improve performance, the main focus of this release shall be bugfixes, UI improvements and many new tools for developers. Alternating between new code and maintenance hopefully guarantees the quality of the codebase to flourish. To not give you a boring release without apparent changes, the build also ships a bugfix / optimization of startup time and a minor general performance improvement. There are 4 new translations and 12 updated ones. For developers, there is a special goodie available: I have released my bachelor's thesis about last year's WoT performance work. The thesis contains a large introduction which explains how WoT works. As it is intended for an audience with no prior Freenet-related knowledge, it shall be considered as the WoT core developer's manual. There are also open algorithmic questions outlined in the thesis - so if you want to do some computer science, please have a look at it, feedback is welcome. There are also: - new command line operations. - unit test framework improvements, most notably code to analyze test coverage. - a simplification of an on-disk fileformat. Outlook for the next build: The missing piece in the "first round" of the most promising performance improvements is planned to ship in the next build: Currently, WoT creates an USK subscription for all of the known identities, which amounts to over 14 000. This puts very heavy load on a typical Freenet node. The fix will address this by only subscribing to the identities which the user has assigned trust to. The other identities will only be fetched "on-demand" when WoT learns by gossip that they surely have published a new edition. For details of the planned algorithm, see: https://bugs.freenetproject.org/view.php?id=3816 In other news: Please excuse that this release took very long to be developed. I've been a volunteer since December and thus currently cannot provide as much time as I could when I was being an employee. This is because Freenet had run out of money. Luckily we've recently got a donation which will fund 1 year of development. I have not got a different job at the hope of being able to return to Freenet, so if the community decides to allow me to continue being an employee, I'll do my best to provide more frequent WoT releases again :) And even if not, I'll cut down the target size for the upcoming release: This release was planned to resolve 12 issues, and it ended up having to resolve 31. So the next release will be aimed at ~ 6 issues to take less time. If you want to help the project to avoid a half-a-year funding gap from happening again, please consider donating: https://freenetproject.org/ Scroll down to see the fundraising progress bar. Bitcoin is also accepted in case you want to stay semi-anonymous! HOW TO INSTALL: While this release has not yet been bundled with a new Freenet release, it can be acquired a lot easier than previous non-bundled ones: 1) Unload the "WebOfTrust" plugin and load the "WebOfTrust Testing Versions" plugin. Please do first read the description for the security implications! WARNING: When sending data to the network, WoT will include which version you are using. Thus, if you are a very early tester, it may become easier for attackers to correlate which identities belong to you, as there may be very few people using the new release. 2) If you had already done step 1 previously, i.e. are already running a previous testing release, you need to restart Freenet or unload and re-load the plugin for getting the update: Freenet will only check for updates of the testing version when the plugin is restarted. You will also get this release someday even if you do not switch to the testing versions. It will just take longer: Non-testing releases are shipped together with regular Freenet releases; and Freenet releases do not happen as often as WoT releases. CHANGELOG - prefixed with the bugtracker issue number: - 0006653: [Bugs] IdentityFileProcessor can get triggered during early startup, slows it down (xor) The previous build's changelog promised an improved startup time of WoT of 2 minutes. Unfortunately, in practice, a bug could cause this to become over 20 minutes. This bug has been fixed, and startup times of 2 minutes should in fact be possible now. This is of course still too much. The next build's work shall reduce this to almost zero startup time. See https://bugs.freenetproject.org/view.php?id=3816 - 0006663: [Performance] Enforce maximum numer of trust values when importing trust lists (xor) When WoT uploads trust lists to the network, there is a limit of 512 trust values. If you set more trust values than 512, only 512 of them will be exported. Unfortunately, this limit was not enforced when importing trust lists which were downloaded from the network. Thus, a malicious user could upload trust lists with much more trust values to conduct a denial of service attack on the network. Luckily, this was not a real security issue, since there was still an indirect limit: The file size of trust lists was properly limited to 1 MB. Still, a lot of trust values can fit in 1 MB, which is why this caused noticeable performance issues. This build fixes import to ignore trust lists with more than 512 entries. This could have an impact on 3rd party clients such as LCWoT: If they only ever produce trust lists that large, their identities will be ignored by WoT. Thus, if you're an author of a 3rd party client, please fix it to limit the amount of exported trust values to 512. - 0006648: [Performance] StatisticsPage: Add statistics to monitor worst case runtime of new distrust computation (xor) The previous release delivery a rewrite of the core WoT algorithm to be fully incremental. This was subject of the bachelor's thesis which this build ships as PDF. While the algorithm shows a sufficiently quick average runtime, the mathematical theory indicates a potential worst case runtime which would not be acceptable. It is unclear whether and how often the worst case will happen with the real network - this depends a lot on the network topology and usage patterns. To investigate whether further optimization of the algorithm would be worthwhile, this build adds statistics to measure how often the worst case happens during normal operation of WoT. If you want to help this investigation, please provide the following statistics from the "Community / Statistics" menu: - Number of incremental trust value re-computations due to new distrust - Number of incremental trust value re-computations due to new distrust - only of those which took more than 10 seconds - Average seconds for incremental trust value re-computation due to new distrust - Average seconds for incremental trust value re-computation due to new distrust - only of those which took more than 10 seconds Please ensure your WoT has been running for at least a day before you provide the stats. Ideally keep it running for as long as you can. Thank you! - 0006722: [Security] Improve captchas by randomization (bertm) Existing captchas were insecure: cracking was possible in under 1 second on an average PC, without the use of any advanced techniques, by means of best pattern match incremental generation. This fix introduces a new WordRenderer and GimpyEngine for the Kaptcha library that randomize: - Horizontal inter-glyph separation - Vertical glyph alignment - Glyph rotation - Twirl filter parameters (angle) - Ripple filter parameters (amplitude, wavelength) - Water filter parameters (centre, radius) Efforts have been made to keep the captchas readable by humans. The resulting captchas should not be much harder to read, but are far more difficult to crack by means of simple pattern matching. Also, for developers, code has been added to create arbitrary amounts of captcha JPG files on disk to examine what the captchas look like. See JavaDoc of function main() of class CaptchaFactory1. - 0006816: [Usability] Update translations from Transifex, add new ones (xor) 0006817: [Bugs] Fix Norwegian Bokmål and Portuguese (Brazil) translations to work (xor) New translations: Croatian Bokmål (Norwegian) Portuguese as spoken in Brazil Portuguese as spoken in Portugal Updated translations: lang_de.l10n | 30 ++--- lang_el.l10n | 126 ++++----------------- lang_en.l10n | 32 +++--- lang_es.l10n | 271 +++++++++++++++++++++++---------------------- lang_fi.l10n | 102 ----------------- lang_fr.l10n | 206 +++++++++++++++++----------------- lang_hu.l10n | 72 ------------ lang_it.l10n | 127 +++++---------------- lang_nl.l10n | 109 ++---------------- lang_pl.l10n | 97 ---------------- lang_ru.l10n | 65 ++--------- lang_zh-cn.l10n | 225 +++++++++++++++++++------------------ Huge thanks to the many translators on Transifex! - 0006569: [Usability] L10n: Backport improvements at German l10n to English (xor) 0006570: [Usability] L10n: Instead of "Trust" / "Score", use "Trust" and "Computed trust" (xor) 0006214: [Usability] L10n: Get rid of technical terms "Insert / Request URI" (xor) 0006568: [Usability] L10n: Instead of "contexts", use "services" (xor) 0006565: [Usability] Statistics page l10n: "Average time for full score re-computation" string lacks unit (seconds) (xor) 0004901: [Usability] L10n: Fix glitches (xor) 0006409: [Usability] L10n: when logged in the option “Create new Identity” seems strange. (xor) 0006818: [Usability] Process issues on Transifex (xor) In the last release, the German translation had received a full review & revamp. As I'm German, this means that it can be considered a reference translation now. Due to the large amount of improvements, it was decided to not immediately port them to English in the previous release. Thus this WoT release now ships the same improvements to the English l10n. The amount of improvements would be too large to explain here. You can check the differences on GitHub: freenet:build0018...freenet:3aff914346079760fa2e5921c30633e6f44c49a0 Click "Files changed", then "39 changed files", then "src/plugins/WebOfTrust/l10n/lang_en.l10n". - 0006812: [Features] wotutil.sh: Allow usage of FCP API (xor) build0018 had added "wotutil.sh" as a new command line user interface for offline use of WoT databases, i.e. without running WoT in a Freenet node. It allows analytics and maintenance operations. The tool now supports using most of the WoT FCP API on the terminal. As FCP is the intended main API for building applications on top of WoT, this allows usage of most of the WoT API on the terminal. "Most of" means that the event-notifications FCP message "Subscribe" is not supported. This is necessary because "Subscribe" is asynchronous, i.e. the replies will be delivered in the future. wotutil.sh bypasses the network layer, so asynchronous replies are not possible. Please do notice that this does *not* allow connecting to a WoT running inside of Freenet. It instead allows simulating FCP by directly loading the data from a closed WoT database on disk. - 0006651: [Bugs] "wotutil.sh -testAndRepair" says "Database OK!" if the database file did not exist, should fail (xor) build0018 had added "wotutil.sh" as a new command line user interface for offline use of WoT databases, i.e. without running WoT in a Freenet node. It allows analytics and maintenance operations. The "-testAndRepair" command checks the database for inconsistencies. Unfortunately, it had a bug which caused it to consider the database file not existing as "the database is healthy!". This has been fixed to show an error message now. - 0006640: [Bugs] KnownIdentitiesPage: IndexOutOfBoundsException when searching (xor) On the 'Other people's identities' page, the search function could cause an error due to an IndexOutOfBoundsException. This has been fixed. - 0006596: [Bugs] Add workaround for db4o bug (xor) This fixes potentially showing wrong information on the last page of the "Other people's identities" menu (previously "Known anonymous identities"). It is difficult to say whether this did happen in practice, it was triggered by a db4o bug which is too complex to justify doing an analysis of when it would happen. If it did happen, it should at least have only affected the UI, not the actual database contents. If you are a developer and have assertions enabled, it also fixes AssertionError happening on that page. Notice that assertions slow down WoT very much, so you should only enable them for test runs. CHANGELOG about stuff only interesting for developers: - 0006840: [Code quality] Provide xor's WoT bachelor thesis as a developer's manual (xor) My past year's Freenet work of improving the core algorithm of WoT had been recycled into my bachelor's thesis. It did pass examination by the university, and now is contained in the WoT source code at: developer-documentation/core-developers-manual/ OadSFfF-version1.2-non-print-edition.pdf Besides explaining last year's work, it has another purpose: Since the thesis aims to require no prior Freenet-related knowledge, it provides a pretty thorough explanation of how WoT works. Thus, you may consider it as the WoT core developer's manual. Beyond getting to know how WoT works, it would be of scientific benefit for the project if you do read it: The end of the thesis describes how the algorithm might be further improved by investigating what can be considered as a whole class of algorithms. I have not heard about such a class of algorithms being identified and named by science yet. But this might be merely due to lack of my knowledge. So I hope that someone knows if this class of algorithms has already been discovered and analyzed by other people. This could help us a lot to further improve WoT with algorithms which are specially crafted to be fast at this type of problem. EDIT: Meanwhile, Bert Massop already remarked that this may very well be the class to solve the "dynamic single-source shortest-paths problem"! There may still be some benefit in thinking about this though, as the paper he cited sounds like there could still be better solutions to the problem than what it provides. The paper is called: Roditty, L., & Zwick, U. (2011). On dynamic shortest paths problems. Algorithmica, 61(2), 389-401. Feedback for improvement of the document is welcome: Besides some layout changes it is still identical to what I handed in at university. Thus, it may contain many stupid mistakes caused by pre-deadline caffeine intoxication. - 0006849: [Code quality] Allow Java 7 features to be used in the code (xor) Freenet build 1473 switched from Java 6 to Java 7 as the minimal required Java version. As a consequence, this now also applies to WoT. As a positive consequence, developers can now benefit from the new language features and API of Java 7. - 0006861: [Code quality] Ant builder: Support analyzing test coverage with Cobertura (xor) Cobertura is a tool which injects its own code during the unit tests to analyze which code of the WoT core is called during test runs. This yields HTML output which contains things such as: - a view of the source code with the lines being color-coded as: red = untested, green = tested. - a percentage of how many lines of code are covered by the tests. - a percentage of how many code paths (if() etc) are covered by the tests. - bonus (static?) analysis: the "average McCabe's cyclomatic code complexity for all methods." = average number of code paths. This can be used to find giant functions which need to be split up. - the above statistics both for the whole repository and also for each package and class. I've amended the Ant builders of fred and Web of Trust to optionally use Cobertura and provide the HTML output at ./test-coverage/html. As dependency you only need to install the standard Ubuntu "cobertura" package. Alternatively, put a cobertura.jar into /usr/share/java/ It can then be executed by: ant -Dtest.coverage=true You can also compute test coverage as caused by only a specific single unit test class: ant -Dtest.coverage=true -Dtest.class=packageName.ClassName For fred, the code is pending as a pull request: hyphanet/fred#553 Here are the current analysis results for fred and WoT: Summary of analysis: Freenet core: http://i.imgur.com/xTuZwBv.png Web of Trust: http://i.imgur.com/tdfcM0J.png Detailed HTML output (you need Freenet for those links): Freenet core: http://127.0.0.1:8888/CHK@ukoL~NGt6t0p9VRJ7RATO0vxlGVr0mBvL0qB164A4Y8,DbjfwdzSP7hDN11-s20k79tRURobCRtyZyVGppVH1ZY,AAMC--8/test-coverage-fred_testing-build-1474-pre1-122-g5baa142/index.html Web of Trust: http://127.0.0.1:8888/CHK@vw18zCPqOdqfT5yLekpJjGVJlEzMLgCYJdLf8w34Cmw,bfQZ1je7uBpP1BzJqPYC7kr7oZ7KV5FrVk40Vw967cs,AAMC--8/test-coverage-WebOfTrust-build0019/index.html - 0006760: [Code quality] Use Travis CI (nextgens, xor) Travis CI is a cloud service which does the following when a developer files a new pull request: - It runs the unit tests on the branch the developer submitted. - It does a temporary merge of the submitted branch into the target branch and also runs the unit tests there. This is the "CI" part: Continous integration means continous testing of whether the branches of multiple developers integrate cleanly into the main repository. - The test runs are done on multiple JREs: * Oracle Java 7 * Oracle Java 8 * OpenJDK 7 - The success/failure of the test runs are shown at GitHub in the pull request. The log files can also be viewed. - The tests are repeated if you push new commits to the branch. Not only does this enforce that unit tests are monitored, but it can also help developers with slow computers or lack of JUnit: If you cannot run the tests yourself, you may submit a pull request marked as "don't merge yet" just to get the test results. In my experience, GitHub even triggers Travis CI as soon as you open the site for creating a pull request. Thus you don't need to submit it immediately if you first want to check for test failures. Travis CI notably is a widely used service, so Google may help you with plenty of documentation about things such as how to trigger more test runs. - 0006815: [Code quality] l10n: Add config for Transifex command line tool "tx" (xor) Translations nowadays are being managed on a collaborative website: https://www.transifex.com/ They offer a command line tool called "tx" for downloading and uploading translations. The tool is open source and available in package management of Ubuntu for example. Hence a configuration file for the tool was added to the WoT repository. It can be found in the ".tx" directory where tx expects it. To use the tool, it is recommended to use the "transifex-pull" script from our scripts repository: https://github.com/freenet/scripts Please make sure to first read the manual at the beginning of the tool's Python source code. - 0006846: [Code quality] Use a human readable file format for serializing Identity XML files to disk (ArneBab, xor) The core network traffic of WoT is upload and download of "Identity XML files". These contain things such as the name of your identity, the applications it uses (Sone, Freemail, etc.), and the trust values you have chosen to publish. WoT downloads those files for all known and trusted identities. As download from the network is usually faster than processing the files, they will be queued to disk before processing. Class IdentityFileDiskQueue can be configured to archive the files after processing so they can be used for scientific or debugging purposes. For example, you can run an offline instance of WoT and feed its Identity file queue with previously archived files to reproduce bugs. To ease those developer purposes, the format of the files has been changed from Java serialization to a human readable file format. The file format is based on fred's class SimpleFieldSet, which is the foundation of the Freenet client protocol (FCP). Thus, if a FCP library is available in your favorite programming language, it shall be easy for you to write a parser for the new file format. For an explanation of the file format, see: https://github.com/freenet/plugin-WebOfTrust/blob/3aff914346079760fa2e5921c30633e6f44c49a0/src/plugins/WebOfTrust/IdentityFile.java#L22-L70 For an explanation of how to repeat test runs using the queue, see: https://bugs.freenetproject.org/view.php?id=6873 For FCP libraries in various programming languages, see: https://wiki.freenetproject.org/Projects Thanks to ArneBab for making me realize this feature is a good idea! - 0006557: [Code quality] Ant: Builder silently won't run unit tests if there is no JUnit installed, should instead abort with error message (xor) When compiling WoT with the standard command line "ant", it will not only compile, but also by default run the unit tests. As new developers typically have different machines than the existing ones, this gives us a good feedback on whether the tests succeed in different environments. Unfortunately, new developers are also likely to forget installing JUnit 4 and Hamcrest, which are the requirement for the tests. If the dependencies were not found, the old code just didn't run the unit tests then, there was no error message - a rather severe breakage :( It will abort the whole compilation now. Thus, if you want to ensure good testing of WoT, please consider checking whether the unit tests do run now. If you want to skip the tests, use "ant -Dtest.skip=true". - 0006588: [Code quality] Unit tests: Check WebOfTrust.verifyDatabaseIntegrity() when finished (xor) This is a test which checks whether the contents of the database match the database schema. It verifies not only structural integrity, but also semantic correctness. This can be things such as "is the modification date of this Identity *before* the creation date?". The test is implemented by calling the startupDatabaseIntegrityTest() functions on all objects which extend class Persistent. See those functions for what is tested. By calling the database integrity test from the base-classes for all unit tests, it is now run at the end of every unit test - even those which are not related to the database schema at first glance. This should ensure quite a bit of general shotgun testing. That's a necessary insurance due to the general complexity of enforcing a database schema in db4o, which does not have any schema checks of its own: The only thing which ensures the schema is obeyed is WoT code, so it's easy for new developers to cause damage. - 0006686: [Code quality] Unit tests: Check WebOfTrust.verifyAndCorrectStoredScores() when finished (xor) One of the main jobs of WoT is to accumulate the trust ratings which identities give to each other. For identities which have not received a direct trust rating of the user, it will use the trust values of remote peers to calculate a Score value. The Score is a weighted average which acts as a replacement for missing direct trust ratings of the user. Thus, Scores are the central "output" of WoT, and their correctness is very important. The function verifyAndCorrectStoredScores() validates the correctness of the whole Score database using the reference algorithm. By calling this function from the base-classes of all unit tests, it is now run at the end of every test. As the reference implementation is not used in practice, but was replaced by a faster algorithm in build0018, this kind of shotgun testing ensures that the optimized algorithm works properly. - 0006682: [Code quality] Unit tests: Check WebOfTrust.isTerminated() when finished (xor) WoT has many different subsystems which run on a thread of their own. During shutdown, hypothetical threading issues such as deadlocks could prevent shutdown of some subsystems. This would be difficult to notice by the user: To make sure that terminating Freenet is always possible, shutting down WoT may not abort with an error message. So to ensure shutdown works properly, code was added to the base-classes of all unit tests. At the end of every test, it will check whether the subsystems reported successful sutdown. - 0006594: [Bugs] Invalid usage of HashMap/*Set with objects of subtypes of Persistent (xor) Child classes of Persistent, such as Identity / Trust / Score, typically have a hashCode() and equals() function, so they are eligible for being used with hash tables. However, there is a pitfall: The equals() function does not only compare whether the entity behind the object is the same. It also compares the current state of the object. For example, an Identity objects represents the entity of a single person. This person is always the same. But the state of an Identity object also includes things such as the list of WoT applications the identity uses. These things may change over time. Hence, equals() may return false if it is used to compare multiple different versions of the same identity. The consequence of that is that code which uses HashSets together with multiple versions of identities will cause multiple entries in the HashSets when there should only be one. The same applies to class HashMap when objects of class Persistent are used as key, their ID should be used instead. To fix this, class IdentifierHashSet was implemented. This class only considers the ID of the objects for determining equality. All code which uses HashSet was reviewed and fixed to use the new class instead where necessary. The code which uses class HashMap was reviewed to use the ID of Persistent objects as key where necessary. - 0006671: [Code quality] XMLTransformer: Log parsing failures as warning instead of normal so developers notice them (xor) WoT transfers its data using XML across the network. As we already have the experimental alternative implementation of LCWoT available thanks to digger3, it's possible that there are incompatibilities between the XML which WoT and LCWoT produce. Thus, the log level of XML parsing failures was raised from NORMAL to WARNING. This ensures they will be noticed during test runs: The developer documentation recommends to search the logs for messages of priority WARNING and ERROR after test runs. - 0006814: [Code quality] Identity.setNickname(): Provide more descriptive Exception message if given nickname is null (Ademan) The message was misleading previously, as we did not actually check whether the given nickname is null. The NullPointerException was merely being thrown implicitly due to nickname.trim() failing. Thanks to Ademan for this fix! Thanks to: - Ademan - ArneBab - Bert Massop - nextgens - The many translators on Transifex
Web of Trust Version 0.4.4 build0018 ------------------------------------------------------------------------ TLDR: Measurements [1] of removing Trust values show an average execution time of 1.7 seconds, which previously was 49 seconds = a speed improvement of factor 28. IMPORTANT STUFF: NOTICE: According to TheSeeker, his machines have been seized by the US government. This gives the government access to his WoT / FMS identities, his freesites, etc. Please update your Trust values. He was a seed-identity until ~2 years ago, and thus might have received a Trust value of 100 from you automatically if you created your identities back then. (He says this was due to stuff *not* related to Freenet, and that he was neither intentionally committing a crime nor being aware of unintentionally having illegal things on his computers.) NOTICE: While this release has not yet been bundled with a new Freenet release, it can be acquired a lot easier than previous non-bundled ones: 1) Update your Freenet to the testing version using the shell command "./update.sh testing" on Linux or "update.cmd testing" on Windows. 2) Unload the "WebOfTrust" plugin and load the "WebOfTrust Testing Versions" plugin. Please do first read the description for the security implications! 3) If you had already done step 2 previously, i.e. are already running the previous testing release build0017, you need to restart Freenet or unload and re-load the plugin for getting the update: Freenet will only check for updates of the testing version when the plugin is reloaded. Once Freenet build 1471 is released, step 1 will not be necessary anymore. You will also get this release someday even if you do not switch to the testing versions. It will just take longer: Non-testing releases are shipped together with regular Freenet releases; and Freenet releases do not happen as often as WoT releases. SUMMARY AND OUTLOOK (detailed changelog is below): This WoT version finally ships some of the long awaited core performance improvements: - Previous builds would fall back to "full Score re-computation" upon Trust changes which cause "distrust": Removal of a Trust value, changing a Trust value from above zero to zero or negative, or adding a zero or below zero Trust value. The full re-computation was a *very* slow piece of code as it basically recalculates the whole Score database. It usually took ~ 1 minute, or even 10 minutes on some machines. This was made even worse by the fact that it happened not only for local Trust changes but also for remote ones. Also, due to unfortunate current limitations of the database code [4], it would block the web interface from responding during the whole time. This build ships an incremental Score re-computation algorithm for these situations - which should be a lot faster: Measurements [1] show an average execution time of 1.7 seconds, which previously was 49 seconds = a speed improvement of factor 28. As there was an incremental re-computation code for non-distrust Trust changes already, Score re-computation is fully incremental now. - In previous versions, restarting WoT usually caused: 1) a defragmentation of the database. 2) a full Score re-computation (to prevent bugs in the existing incremental Score computation code from causing wrong values to exist for a long time). This has been changed to: 1) Defrag only happens once a week. 2) Full Score re-computation only happens once every 4 weeks. As a consequence, startup should be a lot faster. It now takes ~2 minutes on my machine. NOTICE: WoT client applications which were not yet updated to use the new "event-notifications" API (see build0015 changelog) can put a very high load on WoT. For example Sone currently downloads almost the whole WoT database every 60 seconds [8]. So if you want to get a realistic grasp of whether this WoT release is faster, please test without any client applications. Next steps of development will be: - While the average execution time of 1.7 seconds of the new incremental Score computation algorithm is acceptable, its measured worst case execution time of above 60 seconds is not. It is difficult to judge how frequently the worst case happens, as the benchmark which was conducted had to be synthetic: The operations which cause the new code to run are quite rare in network dumps, so real world Trust changes couldn't be used as a benchmarks. It thus can be hoped that the worst case is a "worst case of a worst case" and very rarely happens on the real network. To determine how often it happens, the next build will ship additions to the "Statistics" page to monitor the worst case [2]. (The average case was already added to the statistics, search for "distrust".) If it happens too often, there is still hope: During the months of development of the new code, quite a few ideas for further improvement arose. Part of those are completely new algorithms, and thus they're not implemented yet but will have to wait until we have measurements to show whether the effort of rewriting this yet once more is indicated. - The improved startup time of ~2 minutes is still too much. It is caused by the fact that WoT will currently subscribe to the USKs of *all* trusted Identities - currently over 11 000! This is not only unacceptable from a startup point of view, but also from a network-load perspective - it causes a O(N²) network-wide load; and also slows down each individual node significantly. Hence, the algorithm will be changed to only subscribe to the USKs of directly trusted identities; and to fetch non-directly-trusted ones in a more sparse, opportunistic approach. See [3]. This is likely the primary goal of the next release. - The new incremental distrust computation algorithm was the subject of my bachelor's thesis. Therefore not only the new code was written, but also a very long document which describes the old algorithm and the steps towards the new one. This thesis' document has turned out to be well-suited to become a "WoT developer's manual", and I plan to release it as such soon. I will wait with that until university has graded it (ETA: October) for slightly selfish reasons: I don't want to risk getting negative opinions on it before I know for sure whether I passed - that'd only cause needless panic. I hope this is only "slightly" selfish as a not-in-panic xor is a happy xor and a happy xor will produce more WoT code :) I will announce the document on FMS / IRC / my flog / the mailing list once it is available. CHANGELOG - prefixed with the bug tracker issue number: - 0006605: [Performance] Prevent using ObjectContainer.activate() if already activated to sufficient depth (xor) "Activation" is a technical term of the db4o database which WoT uses. It is similar to a "join" in relational databases: For example, activating a Trust object will load the identities which gave / received the Trust value. This involves disk IO, and therefore is an expensive operation. When an object is already activated, and then db4o's activate() function is called again, db4o should do nothing in theory. Profiling has shown that this is not the case unfortunately, it takes db4o quite a bit of time to determine that nothing needs to be done. Thus code has been added to WoT to prevent activating stuff twice. The performance impact of this has not been measured, but it should help quite a bit: All getter functions of WoT classes have activation code; and when doing stuff with objects one often calls more than one getter on each object. - 0006621: [Usability] Update translations (operhiem1) 10 files changed, 106 insertions(+), 55 deletions(-) de | 22 +++++++++++----------- el | 28 +++++++++++++++++++++++----- es | 25 +++++++++++++++++++++++-- fi | 2 -- fr | 28 ++++++++++++++++++++-------- hu | 2 -- nl | 2 -- pt_BR | 14 +++++++++----- ru | 22 ++++++---------------- zh-cn | 16 ++++++++++++++-- (The fact that some languages received more removals than additions is due to changed English source strings which were not translated yet.) Huge thanks to the volunteers on Transifex for providing these, and to operhiem1 for managing Transifex. - 0006610: [Code quality] Add command line utility for dumping Trust histogram (xor) WoT now has another user interface: The "wotutil.sh" terminal tool. One of its features is to gather statistics about the Trust network. This shall be of use for statistical evaluations. It already yielded a discovery [5]: The Trust value distribution is not "smooth". There are certain discrete steps of Trust values such as 75 which occur a lot more often than ones which would seem less "natural" to humans. It can be speculated that a reason for this is UI design in WoT client applications such as Sone which encourages discrete steps instead of small +/- 1 changes. If you are a client application developer, please consider changing your UI to encourage +/- 1 steps. The WoT algorithm supports the full input range of [-100, +100], and it would be usage below its capabilities if we only used a few of those 201 values. - 0006617: [Code quality] WOTUtil: Add feature for doing database integrity test and recomputing all Scores (xor) Since Score computation is fully incremental now (as explained in the summary at the beginning of the changelog), wrong Scores caused by bugs will persist for a long time. To be able to detect and fix this, the aforementioned "wotutil.sh" has an option "-testAndRepair" which does a full re-computation (and an integrity test of the database schema). You shouldn't need to run this in normal operation as WoT will do a full re-computation every 4 weeks. It is meant mostly as a tool for developers to be able to validate success of test runs. Nevertheless, users might help testing by doing this sometimes. - 0006618: [Code quality] StatisticsPage: Show date of last defrag / verification of Scores (xor) Since full Score re-computation and database defragmentation have been changed from happening every startup to every few days (as explained in the summary at the beginning of the changelog), the new code to only run them every few days might have bugs. To assist users with noticing these bugs, the dates when those maintenance operations last happened are displayed on the "Statistics" page. - 0006627: [Bugs] Trying to load new databases with old WOT versions can break them (xor) The structure of WoT databases ("database schema") is versioned. The version is changed if the structure changes. To ensure that old versions of WoT do not load databases with a newer structure version, there has always been code in WoT which makes it refuse loading a newer database. Unfortunately, in WoT versions prior to this one, this code was bugged. Even though WoT refused to start and showed an error message, the database *was* being loaded, which could cause corruption. This has now been fixed, so WoT versions starting from this one will correctly refuse to load newer databases. However, old versions such as the previous one cannot be fixed as they're out in the public already. And since this new version does change the database structure, the previous one *will* cause corruption. (You'll at least notice if your database is corrupted, WoT will crash at startup. I can write code to repair such databases if you don't have a backup; please ask me then.) Thus, please do *NOT* try to downgrade this release to build0017 or before! - 0005994: [Security] Schedule defragmentation after deletion of an OwnIdentity (xor) Databases are complex structures. Thus, deleting data from them might not actually erase it from disk. The free space might be kept as-is until something else fills it. To prevent leaking of deleted user identities, a defragmentation is scheduled after the user deletes an identity. This will happen at the next restart of WoT. NOTICE: In general, WoT is not yet safe against leaking data to disk. It for example does not yet encrypt its database if Freenet is configured for encryption. Freenet will hopefully soon notice the user about this, see [6]. - 0006607: [Security] deleteOwnIdentity() will cause the replacement non-own Identity to be fetched even if it is distrusted (xor) When the user deletes one of his local identities, it is not actually deleted but converted to a non-local one. (This is for security reasons: Other local identities might have assigned Trust to it, which should not get deleted.) This code had a bug which would cause the deleted identity to continue to be downloaded no matter whether it was trusted or not. While distrusting one of your own identities after deleting it for sure is a rare use case, this nevertheless was fixed. - 0005757: [Performance] Get rid of using computeAllScoresWithoutCommit whereever possible (xor) This is the new incremental distrust computation code which was described in the summary at the beginning of the changelog. - 0006636: [Performance] computeRankFromScratch() should opportunistically compute ranks and put them into a cache (xor) As explained in the summary at the beginning of the changelog, the new incremental distrust computation code has a high worst-case runtime. To alleviate the worst-case, this algorithmic optimization has been applied already. It is of a complex nature and thus beyond explaining here. Once my aforementioned bachelor's thesis has been published, you will be able to realize that this optimization is one of those suggested in the "Outlook" section of the thesis. Reading the thesis will then explain this optimization. The measurements at [1] aim to show how much this optimization helps. It is labeled as "revision 2" there; the version before it (which is the thesis' original code) is labeled as "revision 1". - 0005962: [Performance] Don't defragment at every startup, defragment every 7 days (xor) As explained in the summary at the beginning of the changelog. - 0006616: [Performance] Don't run verifyAndCorrectStoredScores() at every startup, run it once every 28 days (xor) As explained in the summary at the beginning of the changelog. - 0006631: [Code quality] Provide development versions of WOT via fred USK plugin updater (xor) As explained in the summary at the beginning of the changelog. Some more information: These are compiled by myself instead of being compiled by the release manager. So instead of on a dedicated high-security system which is only booted for releases, they're compiled on a regular development machine which is in daily online use. They're downloaded from my main USK which is used for my flog and WoT identity, and thus hooked up to the network quite often. These both are inherently less secure. With regards to quality testing, I plan to keep the same standards as with regular releases. I will test new code before deployment just as I always did. The releases will be packaged with a "buildXXXX" number as regular releases, and also be announced as such. So basically, this release channel has the main goal of allowing me to do releases myself without paying the security price of giving me access to the main Freenet release keys. This is necessary as nowadays Freenet releases happen less often than WoT releases. It provides the side effect of allowing many users to provide testing without big hassle so the release gets well-tested before it is put onto the main network. CHANGELOG about stuff only interesting for developers: - 0006620: [Code quality] Add Hamcrest to junit-build classpath (pull request 32) (operhiem1) Hamcrest is a framework complementary to JUnit [7]. It may be used for easing unit test development. Thanks to operhiem1 for wiring it in to WoT! - 0006609: [Performance] Add synthetic benchmark for improvements of issue 5757 (xor) Multiple attempts have been conducted to benchmark the aforementioned Score computation improvements: - Class WOTUtil's function benchmarkRemoveTrustDestructive(): This can be used upon a regular WoT database, i.e. a dump of the real network, to measure the performance of the function for removing Trust values. It does so by removing random Trust values one-by-one, and measuring the time it takes for each. This specifically tests the new code as described in the changelog's summary, and thus is what was used for producing the benchmarks [1] which were cited there. - Class ScoreComputationBenchmark: It aims to simulate the topology of the Trust graph as measured on the real network using WOTUtil's new histogram features. This was not completed to full extent yet: It does follow the Trust value distribution, and the trustee count distribution (thanks ArneBab!), but not the received-Trust distribution. Future development might complete this. It then could be used to make the unit tests more realistic: They do use random Trust graphs already, but the probability distribution was chosen arbitrarily, not from measurements. A more detailed elaboration of the benchmarks can be studied in my bachelor's thesis once it is published. - 0005882: [Core] computeAllScoresWithoutCommit() return value should be false when the IdentityFetcher state was wrong (xor) computeAllScoresWithoutCommit() is the full Score re-computation function which the summary of the changelog talks about. It not only serves the purpose of re-computing stuff, but also of *validating* correctness of the existing database contents. This is used heavily in assert()s, and thus also in unit tests. It previously did not check the correctness of the instructions given to the IdentityFetcher of whether it should fetch each of the known Identities or not. Now it also validates whether the IdentityFetcher has been correctly told to fetch/ignore Identities. This means that the new Score computation code is also tested to correctly feed the IdentityFetcher with commands. - 0006608: [Code quality] IdentityFetcher should have a "network dump mode" where it will also download old editions (xor) By default, the IdentityFetcher will try to only fetch the latest edition of each known Identity. A network dump using the IdentityFileDiskQueue thus is a snapshot of a possibly infinitesimally short timespan. It will therefore lack the temporal nature of Trust values being changed around by users. This is a problem if you want to test/benchmark things which happen across longer timespans. Hence the new flag IdentityFetcher.DEBUG__NETWORK_DUMP_MODE can be set to true to make the IdentityFetcher try to download some *old* editions as well. Unfortunately, my test run of a handful of hours showed that most old editions could have fallen out of the network. IIRC, less than 100 Identities were discovered, even though we have over 11 000 nowadays. I did not have very much patience though (less than half a day), so you might give this a try yourself. - 0006596: [Bugs] Add workaround for db4o bug (xor) This one is too boring to explain in detail here. What can be said is that assert()s have been added to make developer's notice if they use programming patterns which might trigger this db4o bug. - 0006647: [Code quality] Provide changelog template (xor) See file "developer-documentation/Changelog-template.txt". Thanks to: - All volunteers on Transifex for updating many translations. - ArneBab for the ideas about graph topology modeling in ScoreComputationBenchmark. - operhiem1 for coordinating Transifex. Links: [1] http://localhost:8888/CHK@DPyogjdlfKp1rUavVANbwRH2NTM7Anq~7dpFA3azdqo,CJ968vmM890poA1FNi7MXlB3-r6zMxv6fytmXPlf7d4,AAMC--8/WoT-benchmark-build0016-commit-3edbad7a70-vs-build0018.png Produced using this GnuPlot script: https://gist.github.com/xor-freenet/33b7a17db0d3b80b842e [2] https://bugs.freenetproject.org/view.php?id=6648 [3] https://bugs.freenetproject.org/view.php?id=3816 [4] https://bugs.freenetproject.org/view.php?id=5748 https://bugs.freenetproject.org/view.php?id=6247 [5] http://localhost:8888/CHK@iCe9WZKq52Esq73iRePhdQiu63nyKrHC7RwS8pP1TaA,sh0ODWAyYOuoHd85Llfr7pF2Sy-mnZwavkx3Lz7puuQ,AAMC--8/WoT-trust-value-histogram-2015-07-23.png [6] https://bugs.freenetproject.org/view.php?id=6559 [7] https://en.wikipedia.org/wiki/Hamcrest [8] Bombe/Sone#11
Web of Trust Version 0.4.3 build0017 ------------------------------------------------------------------------ This is merely a quick hotfix release for build0016 which could not be deployed to due a bug: build0016 was incompatible to Java 6, but Freenet policy still is to be compatible to it. Sorry. I did not notice this since my development system is on Java 7 already for security reasons. For the "real" changelog, please read the announcement of build0016. CHANGELOG - prefixed with the bugtracker issue number: 0006629: [Bugs] Remove usage of Java 7 API so WOT works on Java 6 (xor) As aforementioned. Thanks to operhiem1 for noticing this!
Web of Trust Version 0.4.3 build0016 --------------------------------------------------------- SUMMARY AND OUTLOOK - detailed changelog is below: The main goal of this build is to improve the very poor latency of the web interface. This is done by redesigning the way fetched Identity XML files are processed. Previously, each file would be processed in a thread of its own, even though the actual processing requires a single lock which can only be taken by one thread at once. This did cause processing to be effectively single-threaded, while hundreds or even thousands of threads would be piling up when competing for the single lock. With this build, processing is redesigned to queue fetched XML files to disk and process them one-by-one on a single thread. As a result, the web interface thread now does not have to compete against hundreds of threads who want the same lock. This will result in much lower latency of the web interface, especially if your WOT is not permanently running and thus at startup has to process many identity files in bulk. There are some more benefits which are listed in the detailed changelog below. It is still possible for the web interface to stall for minutes due to the fact that the computations to import a trust list are still implemented with a very poor algorithm. The goal of the next build is to improve this algorithm significantly. The next build is feature-complete already and will be available as release candidate soon. Also, please notice that usage of client applications which have not been adapted to use the new WOT event-notifications API yet can slow down the web interface significantly. This applies to especially Sone which downloads almost the whole WOT database every 60 seconds [1]. If you are a client application developer and want to make usage of event-notifications, please contact me on IRC (nick "xor-freenet"), I am willing to help by explaining how to use the reference implementation FCP client. Some builds later on, the locking on the web interface will be removed completely to allow it to always render instantaneously. This will take a lot of time to implement due to issues with the database and thus has been postponed to after the more easy algorithm improvements. In addition to the performance fixes, many new translations of the UI are included: 9 new languages, and 3 updated ones. Very very many thanks to the contributors on Transifex, and to operhiem1 for mentoring them! CHANGELOG - prefixed with the bugtracker issue number: - 0006244: [Performance] Queue fetched trust lists instead of processing them immediately (xor / nextgens) This is what has been described in the introduction. Beyond the latency improvement of the web interface, this also: - Helps plugins such as FlogHelper which have not been adapted to event-notifications yet. They should lose connection to WOT less frequently, which fixes issues such as FlogHelper saying that WOT is not loaded even though it is. - Helps users of machines with little memory such as Raspberry Pi. Less threads = less memory usage. - Speeds up shutdown - Reduces total CPU/IO usage by identity file deduplication: If an identity file is queued for processing with edition N, and the node fetches edition N+1 of that identity before edition N has been processed, then the old edition N will be dropped. Only the newest will be processed. It is possible that deduplication does not yield a good ratio yet. To fix this, more measurements will have to be done in the future to tweak the time window for which we wait before processing a queued file. Generally, on slower machines there will be more deduplication, which is a nice side effect as deduplication reduces load and slow machines can use that. To compute the deduplication ratio, go to the "Statistics" page and divide "Deduplicated files" by "Total ever enqueued (= fetched) files". Feel welcome to share this number with the team. - Helps make WOT look less buggy to new users: They were the ones where the most new XML files would be fetched at once, and thus did suffer the most severe web interface lockup. - Persists fetch-progress across restart: Previously, the progress of the stalling threads would be lost at shutdown. The new queue code will keep the queued files across restarts. - Allows creation of "bootstrap" bundles of a dump of the XML files of all Identities to help new users with slow connections. See below at issue 006597. (Please do read this, there is a security pitfall with this feature!) - Helps developers to deterministically test changes to WOT, see below at issue 0006597. You can view statistics about the new identity file queue and identity file processor on WOT's "Statistics" page. In particular, the two "Failed files" counters are of interest. Consider looking at your log files if they are non-zero. - 0006561: [Usability] L10n: Add new translations from Transifex, update existing ones (operhiem1, and many Transifex contributors) 0006560: [Security] L10n: Review German translation on Transifex (xor) 0006495: [Usability] L10n: Update Transifex, new l10n strings have been added (operhiem1) New translations: Brazilian Portugese Finnish French Greek Hungarian Norwegian Polish Simplified Chinese Spanish Updated translations: Dutch German (also fully reviewed and improved compared to English, now can be considered as reference translation) Italian - 0006499: [Security] Web interface: Allow sorting identities by edition count (xor) This allows you to check for identities which insert an excessive amount of trust lists, which can be considered as a denial of service attack. If an identity does this, please try to contact them to find out whether it is a bug in their WOT; and distrust them if do not cooperate to fix this behavior. Also, report this to the Freenet team please so we can check for bugs. - 0006554: [Code quality] Web interface / Statistics: Add "Number of unfetched identities" (xor) For first time users or people whose WOT has been offline for a long time, this number can be used to find out how far WOT is with downloading all identities. - 0006230: [Security] Make sure to always display the logged-in identity in the web interface (xor) This is to prevent you from accidentally revealing which of your identities are owned by the same person by setting a trust value or trust comment with the wrong identity. For now, it will show the full ID of the identity, which is very long and thus ugly. Fixing this is a bit of work and thus it will have to wait until after performance is fixed. See https://bugs.freenetproject.org/view.php?id=6072 - 0006493: [Usability] Web interface: Change page title from "Web of Trust" to "Community"; allow translating (xor) This is to ensure that the page title matches the name of the "Community" menu through which WOT is accessed. - 0006491: [Performance] Update seed identity edition numbers immediately before releasing (xor / ArneBab) This will speed up downloading all identities for new users. CHANGELOG about stuff only interesting for developers: - 0006597: [Code quality] IdentityFileDiskQueue: Support deterministic repeat by dequeuing sorted by filename (xor) The new disk queue for identity files, which was described above at issue 0006244 already, has a feature for repeating test runs: - If DEBUG logging is enabled, it will archive files after processing - to the directory "IdentityFileQueue/Finished". The filenames will be prefixed with an index number to indicate the order in which they were processed. - If DEBUG logging is enabled, it will process files in the "Queued" directory sorted by filename instead of in random order. This means you can copy files from the "Finished" to the "Queued" directory to repeat a test run in a deterministic way - the index number prefix together with the sorted import will guarantee that the files will be imported the same order as in the original run. Please shutdown WOT before doing this. The deterministic test runs are a useful feature for: - Reproducing bugs in a debugger. - Benchmarks: The statistics page shows the total time the identity file processor spent importing files. - Creating "bootstrap" dumps of Identity files of all Identities so new users can add them to their queue instead of waiting for them to be fetched. Please be aware that the cryptographic signatures of the files are NOT verified and thus you put a lot of trust in the person who provides the bundle - the files can be modified in any way. - 0006531: [Code quality] Unit tests which can fail on slow machines should only be executed if "test.unreliable=true" (xor) This applies to the tests in package "plugins.WebOfTrust.util.jobs". They test a timer class, and thus are timing-sensitive and fail randomly on slow or heavily loaded machines. - 0006523: [Code quality] Ant build improvements (xor) The "-projecthelp" parameter now lists all build targets. It was missing "junit". It now also lists the various "-D" options for junit. The "-D" options for junit now all start with "test.". Namely, "-Dbenchmark=true" is now "-Dtest.benchmark". - 0006522: [Code quality] Merge PR 28: Ant build: allow for -Dtest.class=... unit test selection (bertm / xor) This allows running only a single unit test class from Ant. It's useful if you are working on a new test class for example. Thanks to bertm for implementing this! - 0006368: [Bugs] Identicon leaks an InputStream (xor) This memory leak would only apply when using a WOT-based application which showed Identicons via WOT. This probably applied to Sone. Thanks to: - nextgens for motivating me to implement the performance optimization which is the main feature of this build. - operhiem1 for administrating Transifex which has proven to be a huge source of new Translations; and for committing the new l10n files. - ArneBab, stanley and probably some more people I forgot about for testing - bertm for the Ant improvements - you, the users, for having patience while performance optimizations are implemented one after another. Links: [1] Bombe/Sone#11
For changelog, see tag build0016-rc02. The only change after that is code which is only enabled if DEBUG logging is enabled. See https://bugs.freenetproject.org/view.php?id=6597
Web of Trust Version 0.4.3 build0016 release candidate 02 --------------------------------------------------------- SUMMARY AND OUTLOOK - detailled changelog is below: (Changes compared to release candidate 01 are only visible to developers, and not interesting to discuss.) The main goal of this build is to improve the very poor latency of the web interface. This is done by redesigning the way fetched Identity XML files are processed. Previously, each file would be processed in a thread of its own, even though the actual processing requires a single lock which can only be taken by one thread at once. This did cause processing to be effectively single-threaded, while hundreds or even thousands of threads would be piling up when competing for the single lock. With this build, processing is redesigned to queue fetched XML files to disk and process them one-by-one on a single thread. As a result, the web interface thread now does not have to compete against hundreds of threads who want the same lock. This will result in much lower latency of the web interface, especially if your WOT is not permanently running and thus at startup has to process many identity files in bulk. There are some more benefits which are listed in the detailled changelog below. It is still possible for the web interface to stall for minutes due to the fact that the computations to import a trust list are still implemented with a very poor algorithm. The goal of the next build is to improve this algorithm significantly. Also, please notice that usage of client applications which have not been adapted to use the new WOT event-notifications API yet can slow down the web interface significantly. This applies to especially Sone which downloads almost the whole WOT database every 60 seconds. If you are a client application developer and want to make usage of event-notifications, please contact me on IRC, I am willing to help by explaining how to use the reference implementation FCP client. Some builds later on, the locking from the web interface will be removed completely to allow it to always render instantaneously. This will take some time to implement due to issues with the database and thus has been postponed to after the more easy algorithm improvements. In addition to the performance fixes, many new translations of the UI are included: 9 new languages, and 3 updated ones. Very very many thanks to the contributors on Transifex! CHANGELOG - prefixed with the bugtracker issue number: - 0006244: [Performance] Queue fetched trust lists instead of processing them immediately (xor / nextgens) This is what has been described in the introduction. Beyond the latency improvement of the web interface, this also: - Helps plugins such as FlogHelper which have not been adapted to event-notifications yet. They should lose connection to WOT less frequently, which fixes issues such as FlogHelper saying that WOT is not loaded even though it is. - Helps users of machines with little memory such as Raspberry Pi. Less threads = less memory usage. - Speeds up shutdown - Reduce total CPU/IO usage by identity file deduplication: If an identity file is queued for processing with edition N, and the node fetches edition N+1 of that identity before edition N has been processed, then the old edition N will be dropped. Only the newest will be processed. It is possible that deduplication does not yield a good ratio yet. To fix this, more measurements will have to be done in the future to tweak the time window for which we wait before processing a queued file. Generally, on faster machines there will be less deduplication. To compute the deduplication ratio, go to the "Statistics" page and divide "Deduplicated files" by "Total ever enqueued (= fetched) files". Feel welcome to share this number with the team. - Helps make WOT look less buggy to new users: They were the ones where the most new XML files would be fetched at once, and thus did suffer the most severe web interface lockup. You can view statistics about the new identity file queue and identity file processor on WOT's "Statistics" page. In particular, the two "Failed files" counters are of interest. Consider looking at your log files if they are non-zero. Instructions for logging are in "Testing instructions.txt" - 0006561: [Usability] L10n: Add new translations from Transifex, update existing ones (operhiem1, and many Transifex contributors) 0006560: [Security] L10n: Review German translation on Transifex (xor) 0006495: [Usability] L10n: Update Transifex, new l10n strings have been added (operhiem1) New translations: Brazilian Portugese Finnish French Greek Hungarian Norwegian Polish Simplified Chinese Spanish Updated translations: Dutch German (also fully reviewed and improved compared to English, can be considered as reference translation) Italian - 0006499: [Security] Web interface: Allow sorting identities by edition count (xor) This allows you to check for identities which insert an excessive amount of trust lists, which can be considered as a denial of service attack. If an identity does this, please try to contact them to find out whether it is a bug in their WOT; and distrust them if do not cooperate to fix this behavior. Also, report this to the Freenet team please so we can check for bugs. - 0006554: [Code quality] Web interface / Statistics: Add "Number of unfetched identities" (xor) For first time users or people whose WOT has been offline for a long time, this number can be used to find out how far WOT is with downloading all identities. - 0006230: [Security] Make sure to always display the logged-in identity in the web interface (xor) This is to prevent you from accidentally revealing which of your identities are owned by the same person by setting a trust value or trust comment with the wrong identity. For now, it will show the full ID of the identity, which is very long and thus ugly. Fixing this is a bit of work and thus it will have to wait until after performance is fixed. See https://bugs.freenetproject.org/view.php?id=6072 - 0006493: [Usability] Web interface: Change page title from "Web of Trust" to "Community"; allow translating (xor) This is to ensure that the page title matches the name of the "Community" menu through which WOT is accessed. - 0006491: [Performance] Update seed identity edition numbers immediately before releasing (xor / ArneBab) This will speed up downloading all identities for new users. CHANGELOG about stuff only interesting for developers: - 0006531: [Code quality] Unit tests which can fail on slow machines should only be executed if "test.unreliable=true" (xor) This applies to the tests in package "plugins.WebOfTrust.util.jobs". They test a timer class, and thus are timing-sensitive and fail randomly on slow or heavily loaded machines. - 0006523: [Code quality] Ant build improvements (xor) - resolved. The "-projecthelp" parameter now lists all build targets. It was missing "junit". It now also lists the various "-D" options for junit. The "-D" options for junit now all start with "test.". Namely, "-Dbenchmark=true" is now "-Dtest.benchmark". - 0006522: [Code quality] Merge PR 28: Ant build: allow for -Dtest.class=... unit test selection (bertm / xor) This allows running only a single unit test class from Ant. It's useful if you are working on a new test class for example. Thanks to bertm for implementing this! - 0006368: [Bugs] Identicon leaks an InputStream (xor) This memory leak would only apply when using a WOT-based application which showed Identicons via WOT. This probably applied to Sone. Thanks to: - nextgens for motivating me to implement the performance optimization which is the main feature of this build. - operhiem1 for administrating Transifex which has proven to be a huge source of new Translations; and for committing the new l10n files. - ArneBab, stanley and probably some more people I forgot about for testing - bertm for the Ant improvements - you, the users, for having patience while performance optimizations are implemented one after another.
Web of Trust Version 0.4.3 build0016 release candidate 01 --------------------------------------------------------- SUMMARY AND OUTLOOK - detailled changelog is below: The main goal of this build is to improve the very poor latency of the web interface. This is done by redesigning the way fetched Identity XML files are processed. Previously, each file would be processed in a thread of its own, even though the actual processing requires a single lock which can only be taken by one thread at once. This did cause processing to be effectively single-threaded, while hundreds or even thousands of threads would be piling up when competing for the single lock. With this build, processing is redesigned to queue fetched XML files to disk and process them one-by-one on a single thread. As a result, the web interface thread now does not have to compete against hundreds of threads who want the same lock. This will result in much lower latency of the web interface, especially if your WOT is not permanently running and thus at startup has to process many identity files in bulk. There are some more benefits which are listed in the detailled changelog below. It is still possible for the web interface to stall for minutes due to the fact that the computations to import a trust list are still implemented with a very poor algorithm. The goal of the next build is to improve this algorithm significantly. Also, please notice that usage of client applications which have not been adapted to use the new WOT event-notifications API yet can slow down the web interface significantly. This applies to especially Sone which downloads almost the whole WOT database every 60 seconds. If you are a client application developer and want to make usage of event-notifications, please contact me on IRC, I am willing to help by explaining how to use the reference implementation FCP client. Some builds later on, the locking from the web interface will be removed completely to allow it to always render instantaneously. This will take some time to implement due to issues with the database and thus has been postponed to after the more easy algorithm improvements. In addition to the performance fixes, many new translations of the UI are included: 9 new languages, and 3 updated ones. Very very many thanks to the contributors on Transifex! CHANGELOG - prefixed with the bugtracker issue number: - 0006244: [Performance] Queue fetched trust lists instead of processing them immediately (xor / nextgens) This is what has been described in the introduction. Beyond the latency improvement of the web interface, this also: - Helps plugins such as FlogHelper which have not been adapted to event-notifications yet. They should lose connection to WOT less frequently, which fixes issues such as FlogHelper saying that WOT is not loaded even though it is. - Helps users of machines with little memory such as Raspberry Pi. Less threads = less memory usage. - Speeds up shutdown - Reduce total CPU/IO usage by identity file deduplication: If an identity file is queued for processing with edition N, and the node fetches edition N+1 of that identity before edition N has been processed, then the old edition N will be dropped. Only the newest will be processed. It is possible that deduplication does not yield a good ratio yet. To fix this, more measurements will have to be done in the future to tweak the time window for which we wait before processing a queued file. Generally, on faster machines there will be less deduplication. To compute the deduplication ratio, go to the "Statistics" page and divide "Deduplicated files" by "Total ever enqueued (= fetched) files". Feel welcome to share this number with the team. - Helps make WOT look less buggy to new users: They were the ones where the most new XML files would be fetched at once, and thus did suffer the most severe web interface lockup. You can view statistics about the new identity file queue and identity file processor on WOT's "Statistics" page. In particular, the two "Failed files" counters are of interest. Consider looking at your log files if they are non-zero. Instructions for logging are in "Testing instructions.txt" - 0006561: [Usability] L10n: Add new translations from Transifex, update existing ones (operhiem1, and many Transifex contributors) 0006560: [Security] L10n: Review German translation on Transifex (xor) 0006495: [Usability] L10n: Update Transifex, new l10n strings have been added (operhiem1) New translations: Brazilian Portugese Finnish French Greek Hungarian Norwegian Polish Simplified Chinese Spanish Updated translations: Dutch German (also fully reviewed and improved compared to English, can be considered as reference translation) Italian - 0006499: [Security] Web interface: Allow sorting identities by edition count (xor) This allows you to check for identities which insert an excessive amount of trust lists, which can be considered as a denial of service attack. If an identity does this, please try to contact them to find out whether it is a bug in their WOT; and distrust them if do not cooperate to fix this behavior. Also, report this to the Freenet team please so we can check for bugs. - 0006554: [Code quality] Web interface / Statistics: Add "Number of unfetched identities" (xor) For first time users or people whose WOT has been offline for a long time, this number can be used to find out how far WOT is with downloading all identities. - 0006230: [Security] Make sure to always display the logged-in identity in the web interface (xor) This is to prevent you from accidentally revealing which of your identities are owned by the same person by setting a trust value or trust comment with the wrong identity. For now, it will show the full ID of the identity, which is very long and thus ugly. Fixing this is a bit of work and thus it will have to wait until after performance is fixed. See https://bugs.freenetproject.org/view.php?id=6072 - 0006493: [Usability] Web interface: Change page title from "Web of Trust" to "Community"; allow translating (xor) This is to ensure that the page title matches the name of the "Community" menu through which WOT is accessed. - 0006491: [Performance] Update seed identity edition numbers immediately before releasing (xor / ArneBab) This will speed up downloading all identities for new users. CHANGELOG about stuff only interesting for developers: - 0006531: [Code quality] Unit tests which can fail on slow machines should only be executed if "test.unreliable=true" (xor) This applies to the tests in package "plugins.WebOfTrust.util.jobs". They test a timer class, and thus are timing-sensitive and fail randomly on slow or heavily loaded machines. - 0006523: [Code quality] Ant build improvements (xor) - resolved. The "-projecthelp" parameter now lists all build targets. It was missing "junit". It now also lists the various "-D" options for junit. The "-D" options for junit now all start with "test.". Namely, "-Dbenchmark=true" is now "-Dtest.benchmark". - 0006522: [Code quality] Merge PR 28: Ant build: allow for -Dtest.class=... unit test selection (bertm / xor) This allows running only a single unit test class from Ant. It's useful if you are working on a new test class for example. Thanks to bertm for implementing this! - 0006368: [Bugs] Identicon leaks an InputStream (xor) This memory leak would only apply when using a WOT-based application which showed Identicons via WOT. This probably applied to Sone. Thanks to: - nextgens for motivating me to implement the performance optimization which is the main feature of this build. - operhiem1 for administrating Transifex which has proven to be a huge source of new Translations; and for committing the new l10n files. - ArneBab, stanley and probably some more people I forgot about for testing - bertm for the Ant improvements - you, the users, for having patience while performance optimizations are implemented one after another.
With 30 787 lines of code, this version includes 76% more lines of code than the previous version which was 17 494 LOC (source: see [1]). There is a very large amount of code quality improvements and bugfixes. This will pay out a lot during future development hopefully; for example the improved unit testing architecture may speed up debugging a lot. Even though the primary purpose of this build is to alleviate the performance issues, it will still be very slow until applications built on top of WOT have been adapted to use the new performance features. This has a technical reason: The main new performance improvement is the so-called "event-notifications" FCP API which enables apps to request notification of changes without having to download the whole WOT database every few minutes. This cannot be put to use without applications being changed to use it. A reallife example to understand why this is the case: If your mail service delivers letters by horse, they won't arrive any faster because the government has just build a railway system. The mail company needs to first start using trains instead of horses. If you are a developer of a WOT application, please read the "Changelog for developers" section for instructions how to wire it in to your app. There are at least 5 further strong algorithmic performance optimizations possible within WOT itself which I shall work on for the following releases. Those then should speed up WOT without requiring adaptions from client application developers. It was chosen to implement event-notifications before these so we can have a parallelization of development: Client app developers can start integrating event-notifications now, while I can work on the WOT core optimizations. Sorry that speeding up WOT takes so long. WOT is a major project of 30 000 lines of code already, and as such just takes its time to optimize. I am as annoyed as anyone else by the performance issues, and trying to do my best to get them fixed ASAP. Please stay patient :) What follows now are two changelogs of this build: - One of the new features which are primarily relevant to the users - One for new stuff which can be used by developers of applications which use WOT. The changelogs are briefly sorted by importance, most important stuff first. The number left to each change is the issue number in the bugtracker. Notice: The changelogs exclude a LOT of the actual changes. This is because the development of this build took a very long time, and thus to speed up the release, I produced the changelog from the bugtracker instead of reviewing the whole amount of the 1800 commits. So what had no bugtracker entry is likely not included. Also, I've removed anything which isn't "visible" to users or developers, i.e. anything which is not a new feature or bugfix. And bugfixes are also only listed if they are easy to explain/understand. As a conclusion, the very very many code quality improvements and core reliability bugfixes are not in the changelog. If you're interested in the unfiltered list of the 48 resolved bugtracker entries in this release, go to https://bugs.freenetproject.org/changelog_page.php?version_id=63. Sorry about the lack of a complete changelog due to the time pressure to release soon. I will aim for a MUCH shorter release cycle with following builds; and then you'll get full changelogs again. Changelog for users: - 0006039: [Web Interface] WebInterface "Log in" refactoring TODOs (xor) Operhiem1 helped a lot with dealing with the job of refactoring the general style of the web interface to allow users to "Log in" with a certain identity instead of having to chose which identity to work with at every page. This should greatly increase usability as it matches the general convention of how websites work. Also, this functionality can be used by authors of other plugins so there is one common WOT login for anything which uses WOT. - 0006027: [Web Interface] Backport LogInPage features from Freetalk to WOT (xor) The new "Log in" page features a wizard for creating and restoring identities. This wizard explains what WOT is, what an identity is, what trust values are, etc. It allows the user to chose preferences. Also, it fixes the issue of people mixing up the features for restoring an existing identity and creating a new one from a SSK, which could have caused their identity to be permanently broken. This is fixed by explaining the difference very clearly. Same as with the "Log in" page, the wizard can be used by other plugins. - 0006225: [Web Interface] Make the KnownIdentitiesPage split the list into pages (xor) Previously, the "Known identities" page would list ALL identites in a single page. This was obviously not acceptable anymore since we have > 11 000 identities nowadays. Now the list is split into pages. - [Not in bugtracker] [Web Interface]: Translation updates Add French and Simplified Chinese translations, and update Italian, Dutch, and Russian. Thanks to the volunteers on Transifex, and thanks to operhiem1 for putting this into a pull request. Notice: This is probably incomplete already because it was done before adding the "Create identity" wizard. AFAIK, the new stuff is up on Transifex already, so if you want to contribute with translating it, please contact me or operhiem1. Mail xor@freenetproject.org, or even better join the #freenet channel on Freenode IRC. - [No bugtracker entry] [Web Interface]: Tell the user whats invalid bout a chosen nickname When the user enters a nick which contains non-allowed characters, we now tell him which those characters are. - 0006136: [Web Interface] "Restore identity" is only accessible if the user creates an identity first (xor) Previously, the menu for restoring an old identity was only accessible if you actually had already created least one identity. The "Restore identity" feature can now be used through the "Create identity" wizard without creating a completely new identity first. - 0006148: [Web Interface] Randomly generated nicknames may exceed 30 character limit (xor) The nicknames now always fit the 30 character limit. - 0005354: [Web Interface] Usability of the web interface (xor) When clicking on "Community" or "Community -> Home", the "Own anonymous identities" page is now shown instead of the "Statistics" page. - 0006458: [Performance] Parallelize shutdown (xor) During shutdown, instead of waiting for each subsystem of WOT to have terminated before termining the next one, we now terminate them all in parallel using threads. This should speed up shutdown a lot (but please don't expect wonders, there are further issues with shutdown which the next release will improve). - 0006447: [Performance] IdentityInserter thread period constant is too low and causes inserts to never succeed (xor) This fixed bug would prevent your identities from ever being inserted unless you had very high uptime. - 0006001: [Core] Restore WebOfTrust.db4o.backup if WebOfTrust.db4o does not exist (xor) This fixed bug would happen if your computer crashed while WOT was starting up. The symptom would be loss of your database, i.e. all your identities would be gone. - 0004854: [Web Interface] Stacktrace when clicking 'switch to simple mode' while viewing an identity (xor) Clicking this web interface link of Freenet did cause WOT to display an error, which is now fixed. - [No bugtracker entry] [Performancde] New seed identity, thanks to operhiem1 Having another seed identity should speed up the bootstrapping for new users. Changelog for developers: - [Various bugtracker entries] Event-notifications: This new FCP API should provide a HUGE reduction of the load which client apps put onto WOT. What this changes is as follows: To get an up-to-date list of all known identities (or trusts/scores), you previously had to download ALL of them by FCP every X minutes. You would do this even if NONE of them changed - a huge CPU/IO load for nothing. Now you just "Subscribe" to the list of identities. As a result, WOT will send you all of them once as you subscribe, and then keep you up to date on its own by sending changed identities only when a change happens. Thus, instead of a constant load of O(N = number of identites) every X minutes, independent of the amount of changes, you now get a load of O(1) for each change as it happens. If you are a client application developer, and want to integrate event-notifications in your app, I strongly recommend you to 1) Use the new classes DebugFCPClient and FCPClientReferenceImplementation. The latter aims at being something which you can copy-paste to your client application plugin so you have a fully working FCP client which you can use to connect to WOT and obtain event-notifications. The DebugFCPClient uses the aforementioned reference FCP client to have WOT connect to itself and test whether event-notifications ships correct information (test runs of this have been done for several nights). For your purposes, it can also be considered as an example of how to actually use the FCPClientReferenceImplementation. 2) Contact me in case you need help in understanding how to use it! :) Mail xor@freenetproject.org, or even better join the #freenet channel on Freenode IRC. - 0006039: [Web Interface] WebInterface "Log in" refactoring TODOs (xor) (Even though this seems something for users, it IS interesting for developers, please continue reading) Operhiem1 helped a lot with dealing with the job of refactoring the general style of the web interface to allow users to "Log in" with a certain identity instead of having to chose which identity to work with at every page. And this functionality can be used by your plugins! You can redirect users of your plugin to the log in page of WOT, and have WOT redirect the users back to your plugin after they have logged in. For an example of how to use this, see https://github.com/Thynix/plugin-DVCS-WebUI/blob/4893cb22d919c0d6b9178711dbb4990f66a52232/src/main/java/org/freenetproject/plugin/dvcs_webui/ui/web/VelocityToadlet.java Please contact me or operhiem1 for further help. Mail xor@freenetproject.org, or even better join the #freenet channel on Freenode IRC. - 0006027: [Web Interface] Backport LogInPage features from Freetalk to WOT (xor) The new "Log in" page features a wizard for creating and restoring identities. Same as with the "Log in" page, the wizard can be used by other plugins. For an example of how to use this, contact me. Mail xor@freenetproject.org, or even better join the #freenet channel on Freenode IRC. - 0006231: [Core] Finish synchronous FCP API at fred (xor) The implementation of event-notification spawned a huge 569-commit side-project at Freenet itself aka "plugin-fcp-rewrite": The rewrite of the FCP API for communication with plugins, and especially for plugin-to-plugin FCP communication. This adds a lot of new features to replace the code duplication in existing plugins, as they had usually implemented their own extensions of the old API to fix its shortcomings. It also provides new stuff which was completely impossible with the previous API. Notably for example the ability for servers to send messages to the client on their own, without any recent message from the client. This allows plugin FCP server authors to deploy events to clients. Also it improves the code quality of the FCP plugin API a lot - the old API was completely undocumented, the new one has full JavaDoc. There is a much longer and more complete description of whats new in the new API at the pull request. Anyone who has written a plugin which offers an FCP server or uses FCP to connect to another plugin such as WOT, please read it: hyphanet/fred#319 Notice: WOT will for a while stay backwards-compatible to users of the old API. However, to use event-notifications, you must use the new API. I would strongly recommend you to just copypaste the WOT class FCPClientReferenceImplementation for communicating with WOT by FCP, it does use the new API alredy. - [Pull request 26] Background jobs Bertm did a great job with implementing a framework for replacing dangerous usage of the fred implementations of interface Ticker such as especially TrivialTicker. This might be interesting for fred code and other plugins which use those fred classes. The issue was that the existing fred classes provide a "no duplicates" flag which sounds like it ensures that a scheduled job will only ever be executed once. But under certain conditions, the job can actually be executed multiple times in parallel. The new interface "BackgroundJob" and its implementations provide a way to run a thread with a time delay and the guarantee that it will execute only once in parallel. This is a candidate for being moved to fred or a common library. Bertm saved me probably weeks of work with that, thanks a lot! - [No bugtracker entry]: Full-node unit tests WOT now contains class "AbstractFullNodeTest" which can be used as a base class for unit tests. This class starts a regular Freenet node, and loads the WOT plugin into it just as it would be loaded during normal operation of Freenet. From a quality point of view, this is a lot better since it provides a more real environment than the previous approach of manually creating certain WOT classes in a synthetic environment. Also, it allows unit tests to use the full capabilities of Freenet. This is a foundation for future tests which could do fancy stuff such as testing the Identity introduction code by doing real uploads / downloads of puzzles and their solutions. Notice: This might also be interesting to authors of other plugins. The code is pretty generic and should be easy to re-use there. - [No bugtracker entry]: Randomized unit tests which aim to cover all code paths of WOT Function doRandomChangesToWOT() in the new class AbstractJUnit4BaseTest is a framework for thorough, randomized unit tests. It simulates real usage by doing a random amount of changes chosen among many of the possible UI action: - WebOfTrust.createOwnIdentity() - WebOfTrust.deleteOwnIdentity() - WebOfTrust.restoreOwnIdentity() - WebOfTrust.restoreOwnIdentity() with existing colliding non-own Identity - WebOfTrust.addIdentity() - WebOfTrust.addContext() (adds context to identity) - WebOfTrust.setProperty (adds property to identity) - Add/change/Remove trust values This can help with testing stuff for which it is difficult to say what might cause problems. It is already used for testing-event notifications: Any of the mentioned user actions are sources of events, so it is a good idea to test a random permutation of them for whether event-notifications ships their events correctly. Thanks to: - operhiem1 for various pull requests, the great amount of web interface work, and insane amount of code review. - bertm for the giant background-job pull request which probably saved me weeks worth of work - ArneBab for various pull requests and code review - TheSeeker for testing - anyone who I forgot about - and basically EVERYONE in the project for having the great great patience of awating this release for over a year. Sorry again that it took so long! The only explanation I can offer of why this happend is because I wanted to not release half-finished stuff anymore given that the whole WOT peformance disaster was a result of deploying half-finished code as an official plugin. Nevertheless, I hope to have learned something from all the complaints about the huge delay of the release, and will try to have a MUCH shorter release cycle in the future. [1] Lines of code were computed excluding the third-party CAPTCHA library "kaptcha" using the command: find src/ test/ -iname '*.java' | fgrep -vi kaptcha | xargs wc -l NOTICE: This use of xargs can be dangerous if used upon directories which contain files with the newline character in their name.
PreviousNext