Skip to content

Commit 633478a

Browse files
HandyGuySoftwareHandyGuySoftware
committed
Merge branch 'Issue_177'
2 parents 28320ec + 8df8f81 commit 633478a

File tree

8 files changed

+174
-77
lines changed

8 files changed

+174
-77
lines changed

‎changelog

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,9 @@
1+
3.0.8
2+
-----
3+
- Fix crashing error when subject line parsing fails (Issue #177)
4+
- Rewrote documentation to describe advanced regex options for source, destination, and delimiter
5+
- Updated GitHub home page
6+
17
3.0.7
28
-----
39
- Fixed parsing routines to allow for spaces in the source/destination/delimiter specification (Issue #174 - Thanks ekutner)

‎docs/Config-EmailsofInterest.md

Lines changed: 0 additions & 20 deletions
This file was deleted.

‎docs/Config-SrcDestPairs.md

Lines changed: 139 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -2,46 +2,170 @@
22

33
# Source-Destination Pairs
44

5-
dupReport uses "Source-Destination Pairs" to identify the source and destination systems for each backup job. The default dupReport configuration requires that jobs be named in a way that indicates what is being backed up and where it is going. For instance, a job named: “Fred_Home_Desktop-Homers_Minio would show up in the dupReport as:
5+
The heart of dupReport's functionality is the use of "Source-Destination Pairs" to identify the source and destination systems for each backup job. The default dupReport configuration requires that Duplicati backup jobs be named in a way that indicates what is being backed up and where it is going. For instance, a Duplicati backup job named: “MediaServer-NAS" would show up in dupReport as:
66

7-
> **Source:** Fred_Home_Desktop **Destination:** Homers_Minio
7+
> **Source:** MediaServer **Destination:** NAS
88
9-
Note that spaces in job names are not supported, at least by the default pattern matching.
10-
11-
Source-Destination pairs are specified in dupReport in the following format:
9+
Source-Destination pairs are used in dupReport in the following format:
1210

1311
```
14-
<Source><delimiter><Destination>
12+
<Source><Delimiter><Destination>
1513
```
1614

1715

1816
Where:
1917

2018
- \<Source\> is a series of alphanumeric characters
21-
- \<delimiter\> is a single character (typically one of the "special" characters) and **CAN NOT** be a character you use in any of your Source-Destination pairs
19+
- \<Delimiter\> is a single character (typically one of the "special" characters like !,@,#,$,-,*, etc.) and **CAN NOT** be a character you use in any of your Source or Destination names
2220
- \<Destination\> is a series of alphanumeric characters
23-
- Spaces are allowed between the \<Source>, \<delimiter>, and \<Destination> specifications, *though they are not recommended*
2421

25-
dupReport allows you to define the format specification of the Source, Destination, and Delimiter in the [main] section of the dupReport.rc file. Each specification is the regular expression definition of that element. The defaults are:
22+
Spaces are allowed in the \<Source>, \<Delimiter>, and \<Destination> if you define these specifications carefully, *though they are not recommended*, at least when you are just starting out.
23+
24+
**<u>Regular Expressions</u>**
25+
26+
dupReport uses *Regular Expressions* (also known as "regex") to define the source, destination, and delimiter specifications. Regular expressions allow you to specify patterns of characters in a shorthand way so you can match those patterns against a variety of text. If you are not familiar with regular expressions, you have two options:
27+
28+
1. **Accept the defaults**. In the default dupReport configuration the Source and Destination each consist of a single string of characters without spaces and the delimiter is the '-' character (for example, "MediaServer-NAS".) If you name all your backup jobs this way your Duplicati emails should be processed properly. No additional regular expression knowledge is needed on your part.
29+
2. **Learn about regular expressions**. Building your skill set is always a good thing. You can start at [RegexOne](https://regexone.com/) (not an endorsement, just a good site), though there are lots of good regular expressions tutorials on the Internet. dupReport is written in Python, if you are looking for a language-specific tutorial.
30+
31+
<u>**Specifying Source, Destination, and Delimiter**</u>
32+
33+
The full source-destination regex specification is:
34+
35+
```
36+
<Source Regex><Delimiter Regex><Destination Regex>
37+
```
38+
39+
dupReport allows you to define the Source, Destination, and Delimiter regular expressions in the [main] section of the dupReport.rc file. The defaults are:
40+
41+
```
42+
[main]
43+
srcregex=\w+
44+
destregex=\w+
45+
srcdestdelimiter=-
46+
```
47+
48+
| Option | Defintion | Meaning |
49+
| ---------------- | --------- | ------------------------------------------------------------ |
50+
| srcregex | \w+ | a single string of one or more alphanumeric characters (i.e., A-Z, a-z, 0-9, and the underscore character'_') |
51+
| destregex | \w+ | a single string of one or more alphanumeric characters (i.e., A-Z, a-z, 0-9, and the underscore character'_') |
52+
| srcdestdelimiter | - | The '-' character |
53+
54+
Note that whitespace characters (space, tab, newlines, etc.) are **not** allowed in Source or Destination names using this default definition. If the Source or Destination systems in your Duplicati backup job names contain white space characters they will not match the expression.
55+
56+
Using these definitions, the full Backup name dupReport will look for (\<Source Regex>\<Delimiter Regex>\<Destination Regex>) is:
57+
58+
(\w+)-(\w+)
59+
60+
With this specification, the following list explains what will match and not match:
61+
62+
**Will Match**
63+
64+
- MediaServer-NAS
65+
- WebServer-GDrive
66+
- Development-B2
67+
68+
**Will Not Match**
69+
70+
- Media Server-NAS (space in Source name)
71+
- WebServer-G Drive (space in Destination name)
72+
- Development - B2 (space surrounding the delimiter character)
73+
- System Backup (no delimiter character to define the Destination system)
74+
75+
76+
77+
If you want to get a bit more creative and allow space characters in your Source or Destination systems you can change the Source or destination to the following:
2678

2779
```
2880
[main]
29-
srcregex=\w\*
30-
destregex=\w\*
81+
srcregex=[^-]+
82+
destregex=.+
3183
srcdestdelimiter=-
3284
```
3385

34-
Together the full source-destination regex is:
86+
| Option | Defintion | Meaning |
87+
| ---------------- | --------- | ------------------------------------------------------------ |
88+
| srcregex | [^-]+ | One or more characters up to (but not including) the '-' character |
89+
| destregex | .+ | All characters up to the end of the line |
90+
| srcdestdelimiter | - | The '-' character |
91+
92+
Using these definitions, the full Backup name dupReport will look for (\<Source Regex>\<Delimiter Regex>\<Destination Regex>) is:
93+
94+
(\[^-]\+)-(.+)
95+
96+
With this specification, the following list explains what will match and not match:
97+
98+
**Match**
99+
100+
- Media Server-NAS
101+
- WebServer-G Drive
102+
- Computer Under This Desk-Computer Under Other Desk
103+
104+
**Will Not Match**
105+
106+
- Media Server - NAS (OK, this will *technically* work, but your Source name will have an extra space at the end and your Destination system will have an extra space at the beginning. This has the potential to confuse things down the road.)
107+
108+
109+
110+
If you want to allow spaces anywhere in the Source-Destination specification, you can try something like the following:
111+
112+
```
113+
[main]
114+
srcregex=[^-]+
115+
destregex=.+
116+
srcdestdelimiter= \s*-\s*
117+
```
118+
119+
| Option | Defintion | Meaning |
120+
| ---------------- | --------- | ------------------------------------------------------------ |
121+
| srcregex | [^-]* | One or more characters up to (but not including) the '-' character |
122+
| destregex | .+ | One or more characters up to the end of the line |
123+
| srcdestdelimiter | \s\*-\s\* | An arbitrary number of spaces, followed by the '-' character, followed by an arbitrary number of spaces |
124+
125+
126+
127+
**<u>A WORD OF CAUTION</u>**
128+
129+
Using advanced regex patterns to match your Duplicati backup job names can get extremely tricky and the results can be unexpected. The best advice is to keep your naming convention as simple as possible (i.e., "Source-Destination") and things will work much better. dupReport relies on the Source-Destination pair format for all of its operations. If you do not properly specify your Source-Destination pair formats in both the program (through the dupReport.rc file) and in Duplicati (through proper job naming) none of this will work for you.
130+
131+
If you want to proceed with using advanced regular expressions in dupReport, you should use a good regular expression testing program to thoroughly test your regexes and understand how they work before using them in the program. [Regex101](https://regex101.com/) is a good site to use, though there are many others available on the Internet.
132+
133+
134+
135+
# Identifying Emails of Interest
136+
137+
dupReport scans the incoming mailbox looking for backup job emails. However, there may be hundreds (or thousands) of emails in the inbox, only a few of which contain information about Duplicati backup jobs. dupReport identifies "Emails of Interest" by matching the email's subject line against a pattern defined in the dupReport.rc file. If the pattern matches, the email is analyzed. If the pattern does not match, the email is ignored.
138+
139+
You can specify the text that dupReport tries to match by adjusting the subjectregex= option in the [main] section of the dupReport.rc file. subjectregex is the regular expression definition for the desired phrase.
140+
141+
The default for this option is:
142+
143+
```
144+
[main]
145+
subjectregex=^Duplicati Backup report for
146+
```
147+
148+
This instructs dupReport to look for emails whose subject line start with the phrase, "Duplicati backup report for", which is the default used in Duplicati's “send-mail-subject” advanced option.
149+
150+
If you change the subjectregex option in the dupReport.rc file, or change the "send-mail-subject" advanced option in Duplicati, be sure that the two patterns match or you will not be able to properly match incoming emails.
151+
152+
For example, Duplicati allows you to change the "send-mail-subject" to:
35153

36154
```
37-
<srcregex><srcdestdelimiter><destregex>
155+
Duplicati %PARSEDRESULT%, %OPERATIONNAME% report for %backup-name%
38156
```
39157

40-
You can modify the specification of these elements by replacing each with a regular expression defining how dupReport can find that element in a email's subject line.
158+
This can result in an email with the subject line:
41159

42-
***WARNING!*** *dupReport relies on the Source-Destination pair format for all of its operations. If you do not properly specify your Source-Destination pair formats in both the program (through the dupReport.rc file) and in Duplicati (through proper job naming) none of this will work for you. In particular (and repeating what's already been stated) make sure that you **DO NOT INCLUDE ANY SPACES** in or between the \<Source>, \<delimiter>, and \<Destination> specifications in your Duplicati job names.*
160+
```
161+
Duplicati Warning, Backup report for FileServer-B2
162+
```
43163

164+
In this case, you would change the subjectregex option in the dupReport.rc file to:
44165

166+
```
167+
subjectregex=^Duplicati ([\w ]*, |)Backup report for
168+
```
45169

46170

47171

‎docs/readme.md

Lines changed: 18 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -2,19 +2,26 @@
22

33
# WELCOME TO dupReport
44

5-
dupReport is an email-based reporting system for Duplicati. It will gather all your Duplicati backup status emails and produce a summary report on what Duplicati backup jobs were run and their success or failure.
5+
dupReport is an Python-based email collection and reporting system for Duplicati. It will gather all your Duplicati backup status emails and produce a summary report on what Duplicati backup jobs were run and their success or failure.
66

77
Here is a list of some of dupReport's most important features:
88

99
- Collects all your Duplicati result emails and produces easy-to-understand status reports
1010
- Runs on multiple operating systems. dupReport has been tested on Linux (Debian 8 & 9) and Windows 10, but users have reported it working on a wide variety of operating systems
11-
- Support for IMAP and POP3 email services (we recommend IMAP for better results)
12-
- Supports both text and JSON status emails from Duplicati
13-
- SSL/TLS support for incoming/outgoing email transmissions.
14-
- Output report supports HTML, Text, CSV, and JSON formats
15-
- User-defined reporting formats with configurable column options
16-
- No limit to the number of different backup jobs it can track
17-
- Support for Apprise push notification service (<https://github.com/caronc/apprise>)
11+
- A Guided Setup for new users. If there is no configuration (.rc) file when the program runs, the Guided Setup will take the user through the most common configurable options.
12+
- Support for IMAP and POP3 email services (IMAP is recommended for better results)
13+
- Support for using multiple inbound (IMAP/POP3) and outbound (SMTP) email servers.
14+
- Supports both text and JSON formatted status emails from Duplicati
15+
- Supports SSL/TLS support for incoming/outgoing email transmissions.
16+
- Output report in HTML, Text, CSV, and JSON formats
17+
- Send results to email or local files (or both)
18+
- User-defined reporting formats with configurable column and organization options
19+
- Syslog-style logging format for easier searching and organization.
20+
21+
- Ability to send log output to an external syslog server or log aggregator.
22+
- Support for the Apprise push notification service (<https://github.com/caronc/apprise>)
23+
24+
Please see the (new, updated, and reorganized) documentation on these and all the dupReport features.
1825

1926
------
2027

@@ -26,37 +33,20 @@ There are usually only two branches in the dupReport repository:
2633

2734
| Branch Name | Current Version | Purpose |
2835
| ------------ | --------------- | ------------------------------------------------------------ |
29-
| **master** | 3.0.7 | This is the Release branch, which should contain <u>completely stable</u> code. If you want the latest and greatest release version, get it here. If you are looking for an earlier release, tags in this branch with the name "Release_x.x.x" will point you there. |
36+
| **master** | 3.0.8 | This is the Release branch, which should contain <u>completely stable</u> code. If you want the latest and greatest release version, get it here. If you are looking for an earlier release, tags in this branch with the name "Release_x.x.x" will point you there. |
3037
| **pre_prod** | \<None\> | The Pre-Production branch. This is a late-stage beta branch where code should be mostly-stable, but no guarantees. Once final testing of code in this branch is complete it will be moved to master and released to the world. If you want to get a peek at what's coming up in the next release, get the code from here. **If you don't see a pre_prod branch in the repository, that means there isn't any beta code available for testing.** |
3138

3239
If you see any additional branches in the repository, they are there for early-stage development or bug fix testing purposes. Code in such branches should be considered **<u>highly unstable</u>**. Swim here at your own risk. Void where prohibited. Batteries not included. Freshest if eaten before date on carton. For official use only. Use only in a well-ventilated area. Keep away from fire or flames. May contain peanuts. Keep away from pets and small children.
3340

34-
Bug reports and feature requests can be made on GitHub in the [Issues Section](https://github.com/HandyGuySoftware/dupReport/issuesdupReport). <u>Please do not issue pull requests</u> before discussing any problems or suggestions as an Issue.
41+
Bug reports and feature requests can be made on GitHub in the [dupReport Issues Section](https://github.com/HandyGuySoftware/dupReport/issuesdupReport). <u>Please do not issue pull requests</u> before discussing any problems or suggestions as an Issue.
3542

3643
The discussion group for dupReport is on the Duplicati Forum in [this thread](https://forum.duplicati.com/t/announcing-dupreport-a-duplicati-email-report-summary-generator/1116).
3744

3845
The program is released under an MIT license. Please see the LICENSE file for more details.
3946

40-
Please follow dupReport on Twitter [@dupReport](https://twitter.com/DupReport)
41-
4247
Enjoy!
4348

44-
------
45-
46-
# What's New in Version 3.0
4749

48-
dupReport Version 3.0 includes some new and advanced features for analyzing and reporting on Duplicati emails:
49-
50-
- The reporting engine has been re-built to allow for user-created reporting formats (see the new reporting documentation for more details).
51-
- Added a Guided Setup for new users. If there is no .rc file when the program runs the guided Setup will take the user through the most common configurable options.
52-
- Added the ability to specify multiple inbound (IMAP/POP3) and outbound (SMTP) email servers.
53-
- Standardized log format for easier searching and organization.
54-
- Added ability to send output to syslog server or log aggregator.
55-
- Can now send output to a JSON file.
56-
- Added ability to rollback (-b and -B) to a relative time (e.g., 1w,3h) instead of an absolute date time (i.e., "04/11/2020 8:00:00").
57-
- Re-structured the documentation for better readibility and made it easier to find specific settings.
58-
59-
Please see the (new, updated, and reorganized) documentation on these and all the dupReport features.
6050

6151
------
6252

@@ -68,10 +58,7 @@ Please see the (new, updated, and reorganized) documentation on these and all th
6858

6959
[Getting Started (Quickly)](QuickStart.md)
7060

71-
Understanding dupReport
72-
73-
- [Source-Destination Pairs](Config-SrcDestPairs.md)
74-
- [Identifying Emails of Interest](Config-EmailsofInterest.md)
61+
[Understanding Source-Destination Pairs and Identifying Emails of Interest](Config-SrcDestPairs.md)
7562

7663
Running dupReport
7764

0 commit comments

Comments
 (0)