Merge pull request #2 from CKingX/v0.5.0-alpha

V0.5.0

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "haveibeenpwned"
-version = "0.4.0-alpha"
+version = "0.5.0"
 edition = "2021"
 license = "AGPL-3.0-only"
 authors = ["CKingX"]
@@ -28,8 +28,12 @@ crossbeam-channel = "0.5.4"
 xorf = {version = "0.8.0", features = ["serde"]}
 siphasher = "0.3.*"
 serde = "1.0.*"
-rmp-serde = "1.1.0"
+bincode = "1.3.3"
+directories = "2.0.2"
+bitvec = {version = "1.0.0", features = ["serde"]}
+update-informer = "0.5.0"
 
 [profile.release]
 overflow-checks = true
-lto = "thin"
+lto = "thin"
+strip = true

README.md

@@ -1,13 +1,58 @@
+[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2FCKingX%2Fhaveibeenpwned.svg?type=shield)](https://app.fossa.com/projects/git%2Bgithub.com%2FCKingX%2Fhaveibeenpwned?ref=badge_shield)
+
 # haveibeenpwned
 
 haveibeenpwned is a command-line application that uses [HaveIBeenPwned](https://haveibeenpwned.com/) service and can create and use Binary Fuse filter (which is smaller than Bloom filter or Cuckoo filter for the same false positive ratio) for efficient query at cost of false positives. This is still WIP.
 
-## Roadmap
-- [x] Interactively check compromised password using HIBP API (requires internet)
-- [x] Download password file using HaveIBeenPwned queries. This can be more up to date than downloading passwords directly from HaveIBeenPwned website. According to Troy Hunt, passwords from ingestions are not included since a password version release in the download version. However, querying the password does contain the ingested passwords
-- [x] Interactively check compromised password using filter
-- [x] Create filter (of 3 sizes) that allows you to query offline while consuming a fraction of the space. Does require existing downloaded password file (either from website or by using this tool) to create. However, downloadable filter files will eventually be provided.
-- [ ] Check list of passwords in a file (using a filter) to see how many are compromised
+## Features
+- Interactively check compromised password using HIBP API (requires internet) with `haveibeenpwed interactive-online`
+- Download password file using HaveIBeenPwned queries. This can be more up to date than downloading passwords directly from HaveIBeenPwned website. According to Troy Hunt, passwords from ingestions are not included since a password version release in the download version. However, querying the password does contain the ingested passwords. In practice, this contained 3 more passwords than version 8 as of June 7 (847,223,405 vs Version 8's 847,223,402). You can download with `haveibeenpwned downloader [path to output file]`
+- Downloads can be resumed if given the same argument
+- Can interactively check compromised password using filter with `haveibeenpwned interactive-filter [path to filter file]`
+- Can create filter (of 3 sizes) that allows you to query offline while consuming a fraction of the space. Does require existing downloaded password file (either from website or by using this tool) to create with. Filters can be created with `haveibeenpwned create-filter [path to password file] [output path for filter file]` Existing filters are available ([small](https://mega.nz/file/l5JwgTgR#fUtrkSzuItzO_ED_WWxAJOfvld9TnuHrDhEwW2ToMcg), [medium](https://mega.nz/file/wgYUiQwQ#JJLJ-QPLdJ0YCRXulLPjq0tVQG69kMQ8IkEIjdZYllk), [large](https://mega.nz/file/ApZVXRxL#PUSdijeY1wyQdyBHLqWtZ2yB0PfnNZLwTX-VhTew9HU))
+- Check list of passwords in a file (using a filter) to see how many are compromised with `haveibeenpwned file-check [path to file with passwords to test] [path to filter]` (with optional -p command to print compromised passwords from the file)
 
 ## Compatibility
-As haveibeenpwned is still in alpha, the design of the filter is not final.  Therefore, filter file compatibility will **not** be maintained between versions until haveibeenpwned is no longer an alpha version.
+As haveibeenpwned was in alpha, the design of the filter wasis not final at the time. Therefore, filter file compatibility was not maintained between versions until now. Filter created by version 0.4.0-alpha is not compatible with 0.5.0 (and version 0.5.0 has smaller filters than version 0.4.0). However, compatibility from v0.5.0 onwards (current version) will be maintained.
+
+## Install
+haveibeenpwned can be downloaded from [Releases](https://github.com/CKingX/haveibeenpwned/releases) page for Ubuntu .deb package for 18.04 and later, generic linux executable for 64-bit Intel systems (You may need to run `chmod +x <path to binary>`), and Windows releases. If you have rustup installed (see Build Guide), you can install by running:
+```
+cargo intall haveibeenpwned
+```
+
+Currently, macOS builds are not provided as I do not have a Mac. I will also work on creating a flatpak version of haveibeenpwned
+
+## Upgrade Instructions
+If you use the deb file on Ubuntu, uninstall the deb package with:
+```
+sudo apt remove haveibeenpwned
+```
+Finally, install with the newer deb file.
+
+For Windows, just replace the older haveibeenpwned.exe with the newer version.
+
+If you used the haveibeenpwned linux binary, just replace it with newer one (you may need to run `chmodm +x <path to haveibeenpwned>` again)
+
+## Build Guide
+We can use cargo to build haveibeenpwned. We first need to install rustup and build tools (instructions for those can be found [here](https://www.rust-lang.org/tools/install)). Then, we can build with:
+```
+git clone https://github.com/CKingX/haveibeenpwned.git
+cd ./haveibeenpwned
+cargo install --path ./
+```
+Now you can run by typing haveibeenpwned in terminal. Upgrading can be done with cargo install command again. If you would just like to build the binary, you can build the debug binary with:
+```
+cargo build
+```
+Release binary can be built with:
+```
+cargo build --release
+```
+
+The output of the build command will be in ./target/{debug/release}/haveibeenpwned
+
+## License
+haveibeenpwned is licensed as AGPL 3.0. However, there will eventually be an MPL library that can use a filter to check passwords in other programs.
+
+[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2FCKingX%2Fhaveibeenpwned.svg?type=large)](https://app.fossa.com/projects/git%2Bgithub.com%2FCKingX%2Fhaveibeenpwned?ref=badge_large)

src/arguments.rs

@@ -24,10 +24,13 @@ pub enum Commands {
     },
     /// Check all passwords in a file to see if they are compromised
     FileCheck {
+        /// Path to the file containing passwords to check
         password_file: OsString,
-        file: OsString,
+        /// Path to the filter file
+        filter: OsString,
+        /// Use -p if you want to print compromised passwords
         #[clap(short, long)]
-        print_passwords: bool,
+        print_compromised_passwords: bool,
     },
     /// Create an efficient filter that allows you to check passwords offline
     /// However, while significantly smaller, it can result in false positives

src/config.rs

@@ -0,0 +1,64 @@
+use bitvec::prelude::*;
+use directories::ProjectDirs;
+use serde::{Deserialize, Serialize};
+use std::{
+    ffi::OsString,
+    io::{Read, Write},
+    path::PathBuf,
+};
+
+#[derive(Serialize, Deserialize, Default)]
+pub struct Config {
+    pub password_filter: Option<OsString>,
+    pub resume_token: Option<Resume>,
+}
+
+#[derive(Serialize, Deserialize)]
+pub struct Resume {
+    pub resume: BitBox,
+    pub download_file: OsString,
+}
+
+impl Config {
+    pub fn load() -> Self {
+        let config_file = Self::get_config_file();
+        let mut file = std::fs::File::options()
+            .read(true)
+            .write(true)
+            .create(true)
+            .open(config_file)
+            .expect("Unable to open config file");
+
+        let mut file_contents = Vec::new();
+        file.read_to_end(&mut file_contents).expect("Unable to read config file");
+        let result: Result<Self, _> = bincode::deserialize(&file_contents);
+        match result {
+            Ok(result) => result,
+            Err(_) => Config {
+                ..Default::default()
+            },
+        }
+    }
+
+    pub fn store(self) {
+        let config_file = Self::get_config_file();
+        let mut file = std::fs::File::options()
+            .write(true)
+            .truncate(true)
+            .create(true)
+            .open(config_file)
+            .expect("Unable to open config file");
+        let serialized = bincode::serialize(&self).expect("Unable to create configuration");
+        file.write_all(&serialized).expect("Unable to write configuration file");
+    }
+
+    fn get_config_file() -> PathBuf {
+        let mut result = ProjectDirs::from("rs", "", "haveibeenpwned")
+            .unwrap()
+            .config_dir()
+            .to_owned();
+        std::fs::create_dir_all(&result).unwrap();
+        result.push("config");
+        result
+    }
+}

src/downloader.rs

@@ -1,14 +1,19 @@
+use crate::config::Config;
+use crate::config::Resume;
 use crate::error;
 use crate::password;
+use bitvec::bitbox;
 use crossbeam_channel::{bounded, select};
 use indicatif::{ProgressBar, ProgressStyle};
 use rayon::prelude::*;
 use std::ffi::OsString;
 use std::io::Write;
+use std::path::Path;
+use std::sync::RwLock;
 use std::sync::{Arc, Mutex};
 use std::thread;
 
-const HIBP_TOTAL: u64 = 16u64.pow(5);
+const HIBP_TOTAL: u64 = 16u64.pow(5) - 1;
 
 #[derive(Copy, Clone)]
 enum Message {
@@ -18,33 +23,74 @@ enum Message {
 }
 
 pub fn downloader(output: OsString) {
+    let output: &Path = output.as_ref();
+    let output = output.canonicalize();
+
+    if output.is_err() {
+        eprintln!("Unable to use output file");
+        return;
+    }
+    let output = output.unwrap().as_os_str().to_owned();
+
     let (sender, receiver) = bounded::<Message>(128);
     let sender = Arc::new(sender);
     let progress_bar = ProgressBar::new(HIBP_TOTAL);
     let mut progress = 0;
+    let output_file = output.clone();
+
+    if rayon::ThreadPoolBuilder::new()
+        .num_threads(6)
+        .build_global()
+        .is_err()
+    {
+        eprintln!("Could not configure parallel downloading");
+        return;
+    }
 
     progress_bar.set_style(
         ProgressStyle::template(
             ProgressStyle::default_bar(),
-            "[{elapsed_precise}] [{wide_bar:.cyan/blue}] {pos:>7}/{lens:7} ({eta})",
+            "[{elapsed_precise}] [{wide_bar:.cyan/blue}] {pos:>7}/{len:7} ({eta})",
         )
         .progress_chars("#>-"),
     );
 
+    let config = Config::load();
+    let mut resume_flag = false;
+    let resume_file = if let Some(resume) = config.resume_token {
+        if resume.download_file == output_file {
+            Arc::new(RwLock::new(resume.resume))
+        } else {
+            resume_flag = true;
+            Arc::new(RwLock::new(bitbox![0;HIBP_TOTAL as usize + 1]))
+        }
+    } else {
+        Arc::new(RwLock::new(bitbox![0;HIBP_TOTAL as usize + 1]))
+    };
+
+    let resume = Arc::clone(&resume_file);
+
     let thread = thread::spawn(move || {
         let file = std::fs::File::options()
             .write(true)
             .create(true)
-            .truncate(true)
-            .open(output);
+            .truncate(!resume_flag)
+            .open(output_file);
 
         if let Ok(file) = file {
-            let file = Arc::new(Mutex::new(std::io::BufWriter::new(file)));
+            let file = Mutex::new(std::io::BufWriter::new(file));
+            let agent = ureq::agent();
+            let resume = Arc::clone(&resume);
 
             _ = (0..=HIBP_TOTAL).into_par_iter().try_for_each(|n| {
-                let sender = Arc::clone(&sender);
-                let file = Arc::clone(&file);
-                let result = password::download_range(n);
+                {
+                    let status = resume.read().unwrap();
+                    if *status.get(n as usize).unwrap() {
+                        sender.send(Message::Progress).unwrap();
+                        return Ok(());
+                    }
+                }
+                let result = password::download_range(&agent, n);
                 match result {
                     Ok(range) => {
                         let mut file = file.lock().unwrap();
@@ -61,6 +107,9 @@ pub fn downloader(output: OsString) {
                         return Err(n);
                     }
                 }
+                let mut token = resume.write().unwrap();
+                token.get_mut(n as usize).unwrap().set(true);
+                drop(token);
                 match sender.send(Message::Progress) {
                     Ok(_) => Ok(()),
                     Err(_) => Err(n),
@@ -86,11 +135,21 @@ pub fn downloader(output: OsString) {
                 },
                 Message::Done => {
                     progress_bar.finish_with_message("downloaded");
+                    let mut config = Config::load();
+                    config.resume_token = None;
+                    config.store();
                     break;
                 },
                 Message::Error(n) => {
                     progress_bar.abandon_with_message("⚠️");
+                    let mut config = Config::load();
+                    config.resume_token = Some(Resume {
+                        resume: resume_file.read().unwrap().clone(),
+                        download_file: output,
+                    });
+                    config.store();
                     error::download_error(n);
+                    eprintln!("You can resume the download by running with the same command");
                     break;
                 }
             }

src/file_check.rs

@@ -0,0 +1,59 @@
+use rayon::iter::ParallelIterator;
+use std::sync::atomic::{AtomicI32, Ordering};
+use std::{ffi::OsString, io::BufRead};
+
+use rayon::iter::ParallelBridge;
+
+use crate::filter::Filter;
+use crate::password::Password;
+
+pub fn file_check(password_file: OsString, filter: OsString, print_passwords: bool) {
+    println!("Loading filter...");
+    let filter = if let Some(filter) = Filter::open_filter(filter) {
+        filter
+    } else {
+        return;
+    };
+    println!("Filter loaded");
+
+    let file = std::fs::File::options().read(true).open(&password_file);
+
+    if let Err(error) = file {
+        eprintln!("Unable to open password file: {}", error.kind());
+        return;
+    }
+    let file = file.unwrap();
+
+    let file = std::io::BufReader::new(file);
+
+    let total_count = AtomicI32::new(0);
+    let compromised_count = AtomicI32::new(0);
+
+    let result = file.lines().par_bridge().try_for_each(|password| {
+        if password.is_err() {
+            eprintln!("unable to read password from password file");
+            return Err(());
+        }
+        total_count.fetch_add(1, Ordering::Relaxed);
+
+        let password = password.unwrap();
+        if let Password::CompromisedPassword = filter.check_password(&password) {
+            compromised_count.fetch_add(1, Ordering::Relaxed);
+            if print_passwords {
+                println!("{password}");
+            }
+        }
+
+        Ok(())
+    });
+
+    if result.is_err() {
+        return;
+    }
+
+    println!(
+        "Out of {}, there were {} compromised passwords",
+        total_count.load(Ordering::Relaxed),
+        compromised_count.load(Ordering::Relaxed)
+    );
+}