Enrich factory (#128)

* Fixed #120 - we now better store the results of SDK-declared entries. Also made the documentation better for the SDK
* Allowing for storing just part of a path. By setting the ParseDefinition property storePathFrom, we keep only the trailing data
* Working on the extraction of the path

README.md

@@ -11,23 +11,43 @@ The basic method for using this library is, that you create a definition for you
 ![The Processes](diagrams/Log_Parser-Processes.png)
 
 # Table of contents
-
-- [Installation](#installation)
-  - [Maven](#maven)
-- [Parse Definitions](#parse-definitions)
-  - [Defining a Parsing](#defining-a-parsing)
-  - [Defining an entry](#defining-an-entry)
-  - [How parsing works](#how-parsing-works)
-  - [Code Example](#code-example)
-  - [Import and Export](#import-and-export)
-- [Using the Standard Method](#using-the-standard-method)
-- [Using the SDK](#using-the-sdk)
-- [Code Structure](#code-structure)
-- [Searching a organizing log data](#searching-a-organizing-log-data)
-  - [Search and Filter Mechanisms](#search-and-filter-mechanisms)
-  - [GroupBy Mechanisms](#groupby-mechanisms)
-- [Assertions and LogDataAssertions](#assertions-and-logdataassertions)
-- [Release Notes](#release-notes)
+<!-- TOC -->
+  * [Installation](#installation)
+    * [Maven](#maven)
+  * [Parse Definitions](#parse-definitions)
+    * [Defining a Parsing](#defining-a-parsing)
+    * [Defining an entry](#defining-an-entry)
+    * [How parsing works](#how-parsing-works)
+    * [Code Example](#code-example)
+    * [Import and Export](#import-and-export)
+  * [Extracting Data from Logs](#extracting-data-from-logs)
+    * [Using the Standard Method](#using-the-standard-method)
+    * [Using the SDK](#using-the-sdk)
+      * [Writing your own SDK](#writing-your-own-sdk)
+        * [Declaring a Default and Copy Constructor](#declaring-a-default-and-copy-constructor)
+        * [Declaring the transformation Rules in setValuesFromMap](#declaring-the-transformation-rules-in-setvaluesfrommap)
+        * [Declaring the Key](#declaring-the-key)
+        * [Declare the HeaderMap, and ValueMap](#declare-the-headermap-and-valuemap)
+  * [Code Structure](#code-structure)
+  * [Searching and organizing log data](#searching-and-organizing-log-data)
+    * [Search and Filter Mechanisms](#search-and-filter-mechanisms)
+    * [GroupBy Mechanisms](#groupby-mechanisms)
+      * [Passing a list](#passing-a-list)
+      * [Chaining GroupBy](#chaining-groupby)
+  * [Assertions and LogDataAssertions](#assertions-and-logdataassertions)
+  * [Exporting Results to a CSV File](#exporting-results-to-a-csv-file)
+  * [Release Notes](#release-notes)
+    * [1.11.0 (next version)](#1110--next-version-)
+    * [1.0.10](#1010)
+    * [1.0.8.2](#1082)
+    * [1.0.8](#108)
+    * [1.0.7](#107)
+    * [1.0.6](#106)
+    * [1.0.5](#105)
+    * [1.0.4](#104)
+    * [1.0.3](#103)
+    * [1.0.1](#101)
+<!-- TOC -->
 
 ## Installation
 For now we are using this library with maven, in later iteration we will publish other build system examples:
@@ -114,16 +134,38 @@ At the end we can see that each data is stored in a map with the parse defnition
 ### Import and Export
 You can import or store a Parse Definition to or from a JSON file.
 
-## Using the Standard Method
+## Extracting Data from Logs
+
+### Using the Standard Method
 By default each entry for your lag parsing will be stored as a Generic entry. This means that all values will be stored as Strings. Each entry will have a :
 - Key
 - A set of values
 - The frequence of the key as found in the logs
 
-## Using the SDK
+### Using the SDK
 Using the log parser as an SDK allow you to define your own transformations and also to override many of the behaviors.
 
-In order to use this feature you need to define a class that extends the class StdLogEntry
+#### Writing your own SDK
+In order to use this feature you need to define a class that extends the class StdLogEntry.
+
+You will often want to transform the parsed information into a more manageable object by defining your own fields in the SDK class.
+
+##### Declaring a Default and Copy Constructor
+You will need to declare a default constructor and a copy constructor. The copy constructor will allow you to copy the values from one object to another.
+
+##### Declaring the transformation Rules in setValuesFromMap
+You will need to declare how the parsed variables are transformed into your SDL. This is done in the method `setValuesFromMap()`.
+
+In there you can define a fine-grained extraction of the variables. This could be extracting hidden data in strings of the extracted data, or simple data transformations such as integer or dates.
+
+##### Declaring the Key
+You will need to define how a unique line will look like. Although this is already done in the Definition Rules, you may want to provide more precisions. This is doen in the method `makeKey()`.
+
+##### Declare the HeaderMap, and ValueMap
+Depending on the fields you have defined, you will want to define how the results are represented when they are stored in your system.
+
+You will need to give names to the headers, and provide a map that extracts the values.
+
 
 ## Code Structure
 Below is a diagram representing the class structure:
@@ -248,7 +290,8 @@ We now have the possibility to export the log data results into a CSV file. The
 - [#68](https://github.com/adobe/log-parser/issues/68) We now present a report of the findings at the end of the analysis.
 - [#55](https://github.com/adobe/log-parser/issues/55) We can now export the log parsing results into a CSV file.
 - [#102](https://github.com/adobe/log-parser/issues/102) Corrected bug where Log parser could silently stop with no error when confronted with CharSet incompatibilities.
-
+- [#120](https://github.com/adobe/log-parser/issues/120) Corrected the export system as it did not work well with SDK defined entries.
+- Removed ambiguities in the methods for StdLogEntry. For example "fetchValueMap" is no longer abstract, but it can be overriden.
 ### 1.0.8.2
 - Building with java8. 
 - Upgraded Jackson XML to remove critical version

src/main/java/com/adobe/campaign/tests/logparser/core/GenericEntry.java

@@ -46,10 +46,11 @@ public String makeKey() {
                 .map(e -> valuesMap.get(e).toString()).collect(Collectors.toList()));
     }
 
+
     @Override
     public Set<String> fetchHeaders() {
 
-        return getParseDefinition().fetchHeaders();
+        return super.fetchStoredHeaders();
 
     }
 
@@ -59,7 +60,7 @@ public Set<String> fetchHeaders() {
     @Override
     public Map<String, Object> fetchValueMap() {
 
-        return getValuesMap();
+        return super.fetchValueMap();
     }
 
     @Override

src/main/java/com/adobe/campaign/tests/logparser/core/LogData.java

@@ -13,6 +13,7 @@
 import java.io.IOException;
 import java.lang.reflect.InvocationTargetException;
 import java.util.*;
+import java.util.stream.Collectors;
 
 import com.adobe.campaign.tests.logparser.exceptions.LogDataExportToFileException;
 import org.apache.commons.csv.CSVFormat;
@@ -418,7 +419,7 @@ public File exportLogDataToCSV() throws LogDataExportToFileException {
         Optional<T> l_firstEntry = this.getEntries().values().stream().findFirst();
 
         if (l_firstEntry.isPresent()) {
-            return exportLogDataToCSV(l_firstEntry.get().fetchStoredHeaders(), l_firstEntry.get().getParseDefinition()
+            return exportLogDataToCSV(l_firstEntry.get().fetchHeaders(), l_firstEntry.get().getParseDefinition()
                     .fetchEscapedTitle()
                     + "-export.csv");
         } else {
@@ -451,7 +452,8 @@ public File exportLogDataToCSV(Set<String> in_headerSet, String in_csvFileName)
             printer.printRecord(in_headerSet);
 
             for (StdLogEntry lt_entry : this.getEntries().values()) {
-                printer.printRecord(lt_entry.fetchValuesAsList());
+                Map lt_values = lt_entry.fetchValueMap();
+                printer.printRecord(in_headerSet.stream().map(h -> lt_values.get(h)).collect(Collectors.toList()));
             }
 
         } catch (IOException ex) {

src/main/java/com/adobe/campaign/tests/logparser/core/LogDataFactory.java

@@ -209,4 +209,27 @@ public static List<String> findFilePaths(String in_rootDir, String in_fileFilter
         l_foundFilesList.forEach(log::info);
         return l_foundFilesList;
     }
+
+    /**
+     * A factory method for LogData. By default we create GenricEntries. Given a root directory path and a wildcard for
+     * finding files, it generates a LogDataObject containing all the data the log parser finds in the files matching
+     * the search query
+     * <p>
+     * Author : gandomi
+     *
+     * @param in_rootDir         A list of file paths containing log/generated data
+     * @param in_fileFilter      A wildcard to be used for filtering the files
+     * @param in_parseDefinition A ParseDefinition Object defining the parsing rules
+     * @param in_logEntryClass               A log entry class that defines how the found data is to be transformed
+     * @param <T>                            The type of entry we want to be generated while parsing logs. The type
+     *                                       should be a child of {@link StdLogEntry}
+     * @return A LogData Object containing the found entries from the logs
+     * @throws ParseDefinitionImportExportException Thrown if there is a problem with the given parseDefinition file
+     * @throws StringParseException                 When there are logical rules when parsing the given string
+     */
+    public static <T extends StdLogEntry> LogData<T> generateLogData(String in_rootDir, String in_fileFilter, ParseDefinition in_parseDefinition, Class<T> in_logEntryClass)
+            throws StringParseException {
+        List<String> l_foundFilesList = findFilePaths(in_rootDir, in_fileFilter);
+        return generateLogData(l_foundFilesList, in_parseDefinition, in_logEntryClass);
+    }
 }

src/main/java/com/adobe/campaign/tests/logparser/core/ParseDefinition.java

@@ -33,9 +33,11 @@
 public class ParseDefinition {
 
     protected static final String TITLE_PLACEHOLDER = "parseDefinitionResult";
+
     private String title;
     private boolean storeFileName = false;
     private boolean storeFilePath = false;
+    private String storePathFrom = "";
     private String keyPadding = "#";
     private List<String> keyOrder;
     private String printOutPadding = ";";
@@ -69,6 +71,9 @@ public ParseDefinition(ParseDefinition in_oldParseDefinition) {
         this.keyPadding = in_oldParseDefinition.keyPadding;
         this.keyOrder = in_oldParseDefinition.keyOrder;
         this.printOutPadding = in_oldParseDefinition.printOutPadding;
+        this.setStoreFileName(in_oldParseDefinition.isStoreFileName());
+        this.setStoreFilePath(in_oldParseDefinition.storeFilePath);
+        this.setStorePathFrom(in_oldParseDefinition.getStorePathFrom());
     }
 
     /**
@@ -264,4 +269,12 @@ public boolean isStoreFilePath() {
     public void setStoreFilePath(boolean storeFilePath) {
         this.storeFilePath = storeFilePath;
     }
+
+    public String getStorePathFrom() {
+        return storePathFrom;
+    }
+
+    public void setStorePathFrom(String storePathFrom) {
+        this.storePathFrom = storePathFrom;
+    }
 }

src/main/java/com/adobe/campaign/tests/logparser/core/StdLogEntry.java

@@ -33,7 +33,6 @@ public abstract class StdLogEntry {
 
 
     protected static final String STD_DATA_KEY = "key";
-
     protected static final String STD_DATA_FREQUENCE = "frequence";
     protected static final String STD_DATA_FILE_NAME = "fileName";
     protected static final String STD_DATA_FILE_PATH = "filePath";
@@ -87,29 +86,11 @@ public Map<String, Object> getValuesMap() {
      * @return a String for the print out
      */
     public String fetchPrintOut() {
-        List<String> l_printOutList = fetchValuesAsList();
-        return StringUtils.join(l_printOutList, this.getParseDefinition().getPrintOutPadding());
+        return StringUtils.join(fetchValuesAsList(), this.getParseDefinition().getPrintOutPadding());
     }
 
     protected List<String> fetchValuesAsList() {
-        List<String> l_printOutList = new ArrayList<>();
-
-        final Map<String, Object> l_valueMap = this.fetchValueMap();
-
-        l_printOutList.add(makeKey());
-        if (getParseDefinition().isStoreFileName()) {
-            l_printOutList.add(getFileName());
-        }
-
-        if (getParseDefinition().isStoreFilePath()) {
-            l_printOutList.add(getFilePath());
-        }
-        for (String lt_header : this.fetchHeaders()) {
-
-            l_printOutList.add(l_valueMap.get(lt_header).toString());
-        }
-        l_printOutList.add(getFrequence().toString());
-        return l_printOutList;
+        return this.fetchHeaders().stream().map(e -> fetchValueMap().get(e).toString()).collect(Collectors.toList());
     }
 
     /**
@@ -125,12 +106,27 @@ protected List<String> fetchValuesAsList() {
     /**
      * Returns a set of objects you have defined for your log class. When using Generic Object no changes are made to
      * it.
-     * <p>
+     * When defining an SDK you should override this method.
      * Author : gandomi
      *
-     * @return A Maps of extentions of StdLogEntry
+     * @return A Maps of values for the LogEntry
      */
-    public abstract Map<String, Object> fetchValueMap();
+    public Map<String, Object> fetchValueMap() {
+        final Map<String, Object> l_valueMap = this.getValuesMap();
+
+        l_valueMap.put(STD_DATA_KEY, makeKey());
+        if (getParseDefinition().isStoreFileName()) {
+            l_valueMap.put(STD_DATA_FILE_NAME, getFileName());
+        }
+
+        if (getParseDefinition().isStoreFilePath()) {
+            l_valueMap.put(STD_DATA_FILE_PATH, getFilePath());
+        }
+
+        l_valueMap.put(STD_DATA_FREQUENCE , getFrequence().toString());
+
+        return valuesMap;
+    };
 
     /**
      * Increments the frequence
@@ -205,12 +201,12 @@ public void setValuesFromMap(Map<String, String> in_valueMap) {
     }
 
     public void put(String in_dataTitle, String in_value) {
-        this.fetchValueMap().put(in_dataTitle, in_value);
+        this.getValuesMap().put(in_dataTitle, in_value);
 
     }
 
     public Object get(String in_dataTitle) {
-        return this.fetchValueMap().get(in_dataTitle);
+        return this.getValuesMap().get(in_dataTitle);
     }
 
     /**
@@ -229,7 +225,7 @@ public boolean matches(Map<String, Object> in_filterMap) {
                 log.warn("The filter key {} could not be found among the log entry headers.", lt_filterKey);
                 return false;
             }
-            if (!this.get(lt_filterKey).equals(in_filterMap.get(lt_filterKey))) {
+            if (!this.fetchValueMap().get(lt_filterKey).equals(in_filterMap.get(lt_filterKey))) {
                 return false;
             }
         }
@@ -295,5 +291,17 @@ public void setLogFileName(String in_logFile) {
     public void setFilePath(String in_logFile) {
         this.filePath = in_logFile;
     }
+
+    /**
+     * Updates the store path of the log entry. We also remove training "/" to have a clean path
+     * @param in_newPath The path we want to store
+     */
+    public void updatePath(String in_newPath) {
+        var l_pathDelta = StringUtils.difference(this.getParseDefinition().getStorePathFrom(),
+                in_newPath);
+        l_pathDelta = (l_pathDelta.startsWith("/")) ? l_pathDelta.substring(1) : l_pathDelta;
+        l_pathDelta = (l_pathDelta.endsWith("/")) ? l_pathDelta.substring(0, l_pathDelta.length() - 1) : l_pathDelta;
+        setFilePath(l_pathDelta);
+    }
 }