Skip to content

[ML] Change point detection and prediction #92

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 41 commits into from
May 11, 2018
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
b8fa21c
Better naming of decomposition methods and related variables
tveasey Mar 9, 2018
91c0057
More consistent naming in CTimeSeriesModel and avoid lots of overload…
tveasey Mar 9, 2018
eac04e9
[ML] First pass implementation of support functionality for change de…
tveasey Mar 12, 2018
4052390
Factor out some convenience functionality. Switch to using std versio…
tveasey Mar 12, 2018
10d4912
[ML] Wire in change detection/modelling to our univariate time series…
tveasey Mar 15, 2018
7202a17
Less confusing naming of the anomaly score calculation
tveasey Mar 15, 2018
6b420dc
Fix name change fallout
tveasey Mar 20, 2018
69a96ef
Switch to (more standard) logistic function
tveasey Mar 20, 2018
4bc1e53
Tidy up expectation w.r.t. marginal likelihood
tveasey Mar 21, 2018
1bc86c6
Merge branch 'master' into feature/forecast-enhancements-part-2
tveasey Mar 23, 2018
893f0b2
Bad merge
tveasey Mar 23, 2018
447f31a
[ML] Implements an absolute goodness-of-fit test to accept a change …
tveasey Mar 26, 2018
e763bc3
[ML] Linear scaling change detection (#25)
tveasey Apr 4, 2018
96bcb62
Fix windows build issue
tveasey Apr 4, 2018
3ec6deb
Reference to weight styles outlives the object
tveasey Apr 4, 2018
3872822
Merge commit 'd4e4cca70edae4500cc1535a2da582935074a25b' into feature/…
tveasey Apr 13, 2018
57b51c2
Reformat
tveasey Apr 13, 2018
a0d0836
Merge commit '2580b4fc3068699fc8f2c61fbb7b85125f756e0b' into feature/…
tveasey Apr 13, 2018
6debc50
Merge commit '1347a65c807f8e9e7d80c5af8e6d2fd6bb176e1f' into feature/…
tveasey Apr 13, 2018
d616779
Reformat
tveasey Apr 13, 2018
b26e68d
Merge commit '3dadf1c2fe15507fde675ac64e7ff668ef9a014c' into feature/…
tveasey Apr 13, 2018
f09fcb9
Merge branch 'master' into feature/forecast-enhancements-part-2
tveasey Apr 13, 2018
77c6fd8
Fixing format merge
tveasey Apr 13, 2018
8665f50
More fixing of the format merge
tveasey Apr 13, 2018
9a6e471
Switch to std shared pointers
tveasey Apr 13, 2018
3fb0911
Fix fallout from merge
tveasey Apr 16, 2018
5a250d1
Merge master
tveasey Apr 23, 2018
8c64117
Merge branch 'master' into feature/forecast-enhancements-part-2
tveasey Apr 23, 2018
705a6a4
C++11 style changes for clustering code
tveasey Apr 24, 2018
ff82513
Merge branch 'master' into feature/forecast-enhancements-part-2
tveasey Apr 26, 2018
2aa457b
Fix unit test
tveasey Apr 26, 2018
e478892
Merge branch 'master' into feature/forecast-enhancements-part-2
tveasey May 3, 2018
bb83c5e
Improve anomaly sign calculation
tveasey May 4, 2018
d0ae51c
Improve anomaly sign calculation
tveasey May 4, 2018
5fb47b3
Merge branch 'master' into feature/forecast-enhancements-part-2
tveasey May 9, 2018
c08506a
Merge branch 'feature/forecast-enhancements-part-2' of github.com:ela…
tveasey May 9, 2018
0597495
Merge branch 'master' into feature/forecast-enhancements-part-2
tveasey May 9, 2018
0229047
Standardise on CTools pow2
tveasey May 9, 2018
f3e0372
Fix tests and formatting
tveasey May 9, 2018
b54c408
Add change log entry
tveasey May 11, 2018
1ae79ab
Formatting fix
tveasey May 11, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
Prev Previous commit
Next Next commit
Reformat
  • Loading branch information
tveasey committed Apr 13, 2018
commit 57b51c2c59c71b0d67c32b32c5b9f8e456d0ced5
140 changes: 55 additions & 85 deletions bin/autoconfig/CCmdLineParser.cc
Original file line number Diff line number Diff line change
Expand Up @@ -12,137 +12,107 @@

#include <iostream>

namespace ml
{
namespace autoconfig
{
namespace ml {
namespace autoconfig {

const std::string CCmdLineParser::DESCRIPTION =
"Usage: autoconfig [options]\n"
"Options";
const std::string CCmdLineParser::DESCRIPTION = "Usage: autoconfig [options]\n"
"Options";

bool CCmdLineParser::parse(int argc,
const char * const *argv,
std::string &logProperties,
std::string &logPipe,
char &delimiter,
bool &lengthEncodedInput,
std::string &timeField,
std::string &timeFormat,
std::string &configFile,
std::string &inputFileName,
bool &isInputFileNamedPipe,
std::string &outputFileName,
bool &isOutputFileNamedPipe,
bool &verbose,
bool &writeDetectorConfigs)
{
try
{
const char* const* argv,
std::string& logProperties,
std::string& logPipe,
char& delimiter,
bool& lengthEncodedInput,
std::string& timeField,
std::string& timeFormat,
std::string& configFile,
std::string& inputFileName,
bool& isInputFileNamedPipe,
std::string& outputFileName,
bool& isOutputFileNamedPipe,
bool& verbose,
bool& writeDetectorConfigs) {
try {
boost::program_options::options_description desc(DESCRIPTION);
desc.add_options()
("help", "Display this information and exit")
("version", "Display version information and exit")
("logProperties", boost::program_options::value<std::string>(),
"Optional logger properties file")
("logPipe", boost::program_options::value<std::string>(),
"Optional log to named pipe")
("delimiter", boost::program_options::value<char>(),
"Optional delimiter character for delimited data formats - default is ',' (comma separated)")
("lengthEncodedInput",
"Take input in length encoded binary format - default is delimited")
("timefield", boost::program_options::value<std::string>(),
"Optional name of the field containing the timestamp - default is 'time'")
("timeformat", boost::program_options::value<std::string>(),
"Optional format of the date in the time field in strptime code - default is the epoch time in seconds")
("config", boost::program_options::value<std::string>(),
"Optional configuration file")
("input", boost::program_options::value<std::string>(),
"Optional file to read input from - not present means read from STDIN")
("inputIsPipe", "Specified input file is a named pipe")
("output", boost::program_options::value<std::string>(),
"Optional file to write output to - not present means write to STDOUT")
("outputIsPipe", "Specified output file is a named pipe")
("verbose", "Output information about all detectors including those that have been discarded")
("writeDetectorConfigs",
"Output the detector configurations in JSON format")
;
desc.add_options()("help", "Display this information and exit")("version", "Display version information and exit")(
"logProperties", boost::program_options::value<std::string>(), "Optional logger properties file")(
"logPipe", boost::program_options::value<std::string>(), "Optional log to named pipe")(
"delimiter",
boost::program_options::value<char>(),
"Optional delimiter character for delimited data formats - default is ',' (comma separated)")(
"lengthEncodedInput", "Take input in length encoded binary format - default is delimited")(
"timefield",
boost::program_options::value<std::string>(),
"Optional name of the field containing the timestamp - default is 'time'")(
"timeformat",
boost::program_options::value<std::string>(),
"Optional format of the date in the time field in strptime code - default is the epoch time in seconds")(
"config", boost::program_options::value<std::string>(), "Optional configuration file")(
"input", boost::program_options::value<std::string>(), "Optional file to read input from - not present means read from STDIN")(
"inputIsPipe", "Specified input file is a named pipe")(
"output", boost::program_options::value<std::string>(), "Optional file to write output to - not present means write to STDOUT")(
"outputIsPipe", "Specified output file is a named pipe")(
"verbose", "Output information about all detectors including those that have been discarded")(
"writeDetectorConfigs", "Output the detector configurations in JSON format");

boost::program_options::variables_map vm;
boost::program_options::store(boost::program_options::parse_command_line(argc, argv, desc), vm);
boost::program_options::notify(vm);

if (vm.count("help") > 0)
{
if (vm.count("help") > 0) {
std::cerr << desc << std::endl;
return false;
}
if (vm.count("version") > 0)
{
if (vm.count("version") > 0) {
std::cerr << ver::CBuildInfo::fullInfo() << std::endl;
return false;
}
if (vm.count("logProperties") > 0)
{
if (vm.count("logProperties") > 0) {
logProperties = vm["logProperties"].as<std::string>();
}
if (vm.count("logPipe") > 0)
{
if (vm.count("logPipe") > 0) {
logPipe = vm["logPipe"].as<std::string>();
}
if (vm.count("delimiter") > 0)
{
if (vm.count("delimiter") > 0) {
delimiter = vm["delimiter"].as<char>();
}
if (vm.count("lengthEncodedInput") > 0)
{
if (vm.count("lengthEncodedInput") > 0) {
lengthEncodedInput = true;
}
if (vm.count("timefield") > 0)
{
if (vm.count("timefield") > 0) {
timeField = vm["timefield"].as<std::string>();
}
if (vm.count("timeformat") > 0)
{
if (vm.count("timeformat") > 0) {
timeFormat = vm["timeformat"].as<std::string>();
}
if (vm.count("config") > 0)
{
if (vm.count("config") > 0) {
configFile = vm["config"].as<std::string>();
}
if (vm.count("input") > 0)
{
if (vm.count("input") > 0) {
inputFileName = vm["input"].as<std::string>();
}
if (vm.count("inputIsPipe") > 0)
{
if (vm.count("inputIsPipe") > 0) {
isInputFileNamedPipe = true;
}
if (vm.count("output") > 0)
{
if (vm.count("output") > 0) {
outputFileName = vm["output"].as<std::string>();
}
if (vm.count("outputIsPipe") > 0)
{
if (vm.count("outputIsPipe") > 0) {
isOutputFileNamedPipe = true;
}
if (vm.count("verbose") > 0)
{
if (vm.count("verbose") > 0) {
verbose = true;
}
if (vm.count("writeDetectorConfigs") > 0)
{
if (vm.count("writeDetectorConfigs") > 0) {
writeDetectorConfigs = true;
}
}
catch (std::exception &e)
{
} catch (std::exception& e) {
std::cerr << "Error processing command line: " << e.what() << std::endl;
return false;
}

return true;
}

}
}
57 changes: 26 additions & 31 deletions bin/autoconfig/CCmdLineParser.h
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,8 @@
#include <string>
#include <vector>

namespace ml
{
namespace autoconfig
{
namespace ml {
namespace autoconfig {

//! \brief Very simple command line parser.
//!
Expand All @@ -24,34 +22,31 @@ namespace autoconfig
//! IMPLEMENTATION DECISIONS:\n
//! Put in a class rather than main to allow testing.
//!
class CCmdLineParser
{
public:
using TStrVec = std::vector<std::string>;

public:
//! Parse the arguments and return options if appropriate.
static bool parse(int argc,
const char * const *argv,
std::string &logProperties,
std::string &logPipe,
char &delimiter,
bool &lengthEncodedInput,
std::string &timeField,
std::string &timeFormat,
std::string &configFile,
std::string &inputFileName,
bool &isInputFileNamedPipe,
std::string &outputFileName,
bool &isOutputFileNamedPipe,
bool &verbose,
bool &writeDetectorConfigs);

private:
static const std::string DESCRIPTION;
class CCmdLineParser {
public:
using TStrVec = std::vector<std::string>;

public:
//! Parse the arguments and return options if appropriate.
static bool parse(int argc,
const char* const* argv,
std::string& logProperties,
std::string& logPipe,
char& delimiter,
bool& lengthEncodedInput,
std::string& timeField,
std::string& timeFormat,
std::string& configFile,
std::string& inputFileName,
bool& isInputFileNamedPipe,
std::string& outputFileName,
bool& isOutputFileNamedPipe,
bool& verbose,
bool& writeDetectorConfigs);

private:
static const std::string DESCRIPTION;
};


}
}

Expand Down
42 changes: 15 additions & 27 deletions bin/autoconfig/Main.cc
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,8 @@
//! Standalone program.
//!
#include <core/CLogger.h>
#include <core/CoreTypes.h>
#include <core/CProcessPriority.h>
#include <core/CoreTypes.h>

#include <ver/CBuildInfo.h>

Expand All @@ -37,23 +37,21 @@

#include <stdlib.h>


int main(int argc, char **argv)
{
int main(int argc, char** argv) {
// Read command line options
std::string logProperties;
std::string logPipe;
char delimiter(',');
bool lengthEncodedInput(false);
char delimiter(',');
bool lengthEncodedInput(false);
std::string timeField("time");
std::string timeFormat;
std::string configFile;
std::string inputFileName;
bool isInputFileNamedPipe(false);
bool isInputFileNamedPipe(false);
std::string outputFileName;
bool isOutputFileNamedPipe(false);
bool verbose(false);
bool writeDetectorConfigs(false);
bool isOutputFileNamedPipe(false);
bool verbose(false);
bool writeDetectorConfigs(false);
if (ml::autoconfig::CCmdLineParser::parse(argc,
argv,
logProperties,
Expand All @@ -68,20 +66,15 @@ int main(int argc, char **argv)
outputFileName,
isOutputFileNamedPipe,
verbose,
writeDetectorConfigs) == false)
{
writeDetectorConfigs) == false) {
return EXIT_FAILURE;
}

// Construct the IO manager before reconfiguring the logger, as it performs
// std::ios actions that only work before first use
ml::api::CIoManager ioMgr(inputFileName,
isInputFileNamedPipe,
outputFileName,
isOutputFileNamedPipe);
ml::api::CIoManager ioMgr(inputFileName, isInputFileNamedPipe, outputFileName, isOutputFileNamedPipe);

if (ml::core::CLogger::instance().reconfigure(logPipe, logProperties) == false)
{
if (ml::core::CLogger::instance().reconfigure(logPipe, logProperties) == false) {
LOG_FATAL("Could not reconfigure logging");
return EXIT_FAILURE;
}
Expand All @@ -93,20 +86,16 @@ int main(int argc, char **argv)

ml::core::CProcessPriority::reducePriority();

if (ioMgr.initIo() == false)
{
if (ioMgr.initIo() == false) {
LOG_FATAL("Failed to initialise IO");
return EXIT_FAILURE;
}

typedef boost::scoped_ptr<ml::api::CInputParser> TScopedInputParserP;
TScopedInputParserP inputParser;
if (lengthEncodedInput)
{
if (lengthEncodedInput) {
inputParser.reset(new ml::api::CLengthEncodedInputParser(ioMgr.inputStream()));
}
else
{
} else {
inputParser.reset(new ml::api::CCsvInputParser(ioMgr.inputStream(), delimiter));
}

Expand All @@ -125,8 +114,7 @@ int main(int argc, char **argv)
0, // no persistence at present
*inputParser,
configurer);
if (skeleton.ioLoop() == false)
{
if (skeleton.ioLoop() == false) {
LOG_FATAL("Ml autoconfig failed");
return EXIT_FAILURE;
}
Expand Down
Loading