feat(data-format): add support for MANUAL data format (in addition to current DUMP format)

closes #5

Author: tinesoft

README.md

@@ -14,6 +14,7 @@ The library reads all the metadata it needs from the entity classes (index name,
 * Support  for  **JUnit 4** via `LoadEsDataRule`, `DeleteEsDataRule`
 * Support  for  **JUnit  Jupiter** via `@LoadEsDataConfig` / `@LoadEsDataExtension` or `@DeleteEsDataConfig` / `@DeleteEsDataExtension`
 * Built-in support for **gzipped data**
+* Multiple data formats(dump, manual)
 * Written  in  **Java  8**
 * Based on **Spring (Data, Test)**
 
@@ -26,7 +27,7 @@ The library reads all the metadata it needs from the entity classes (index name,
 
 ## Installation & Usage
 
-The library is splitted into 2 independent sub-modules, both are available on [JCenter](https://bintray.com/bintray/jcenter?filterByPkgName=spring-esdata-loader) and [Maven Central](https://search.maven.org/search?q=spring-esdata-loader):
+The library is split into 2 independent sub-modules, both are available on [JCenter](https://bintray.com/bintray/jcenter?filterByPkgName=spring-esdata-loader) and [Maven Central](https://search.maven.org/search?q=spring-esdata-loader):
 
 * `spring-esdata-loader-junit4` for testing with **JUnit 4**
 * `spring-esdata-loader-junit-jupiter` for testing with **JUnit Jupiter**
@@ -80,25 +81,28 @@ To get started,
 * [junit4](/junit4) - if your are using **JUnit 4**
 * [junit-jupiter](/junit-jupiter) - if you are using **JUnit Jupiter**
 
-## Data Format
+## Supported Data Formats
+
+`spring-esdata-loader` currently supports 2 formats to load data into Elasticsearch: **DUMP** and **MANUAL**.
+
+### Dump data format
 
-The data to be loaded must follow the appropriate format.
 Here is an example:
 ```json
-{"_index":"author","_type":"Author","_id":"5","_score":1,"_source":{"id":"5","firstName":"firstName5","lastName":"lastName5"}}
-{"_index":"author","_type":"Author","_id":"8","_score":1,"_source":{"id":"8","firstName":"firstName8","lastName":"lastName8"}}
-{"_index":"author","_type":"Author","_id":"9","_score":1,"_source":{"id":"9","firstName":"firstName9","lastName":"lastName9"}}
-{"_index":"author","_type":"Author","_id":"10","_score":1,"_source":{"id":"10","firstName":"firstName10","lastName":"lastName10"}}
+{"_index":"author","_type":"Author","_id":"1","_score":1,"_source":{"id":"1","firstName":"firstName1","lastName":"lastName1"}}
 {"_index":"author","_type":"Author","_id":"2","_score":1,"_source":{"id":"2","firstName":"firstName2","lastName":"lastName2"}}
+{"_index":"author","_type":"Author","_id":"3","_score":1,"_source":{"id":"3","firstName":"firstName3","lastName":"lastName3"}}
 {"_index":"author","_type":"Author","_id":"4","_score":1,"_source":{"id":"4","firstName":"firstName4","lastName":"lastName4"}}
+{"_index":"author","_type":"Author","_id":"5","_score":1,"_source":{"id":"5","firstName":"firstName5","lastName":"lastName5"}}
 {"_index":"author","_type":"Author","_id":"6","_score":1,"_source":{"id":"6","firstName":"firstName6","lastName":"lastName6"}}
-{"_index":"author","_type":"Author","_id":"1","_score":1,"_source":{"id":"1","firstName":"firstName1","lastName":"lastName1"}}
 {"_index":"author","_type":"Author","_id":"7","_score":1,"_source":{"id":"7","firstName":"firstName7","lastName":"lastName7"}}
-{"_index":"author","_type":"Author","_id":"3","_score":1,"_source":{"id":"3","firstName":"firstName3","lastName":"lastName3"}}
+{"_index":"author","_type":"Author","_id":"8","_score":1,"_source":{"id":"8","firstName":"firstName8","lastName":"lastName8"}}
+{"_index":"author","_type":"Author","_id":"9","_score":1,"_source":{"id":"9","firstName":"firstName9","lastName":"lastName9"}}
+{"_index":"author","_type":"Author","_id":"10","_score":1,"_source":{"id":"10","firstName":"firstName10","lastName":"lastName10"}}
 
 ```
 You can use a tool like [elasticdump](https://npmjs.com/package/elasticdump) (*requires* [NodeJS](https://nodejs.org/)) to extract existing data
-from your Elasticseach server, and them dump them into a JSON file.
+from your Elasticsearch server, and them dump them into a JSON file.
 
 ```
 $ npx elasticdump --input=http://localhost:9200/my_index --output=my_index_data.json
@@ -108,11 +112,32 @@ $ npx elasticdump --input=http://localhost:9200/my_index --output=my_index_data.
 
 > If you change the `--output` part above into `--output=$ | gzip my_data.json.gz` the data will be automatically gzipped
 
+### Manual data format
+
+In this format, you specify your target data directly (no metadata like `_index`, `_source`, ...), as an Array of JSON objects.
+
+This is more suitable when you create test data from scratch (as opposed to dumping existing ones from a ES server) because it is easier to tweak later on to accommodate future modifications in tests. _(Thanks to @DPorcheron for the idea 💡!)_
+
+Here is an example:
+```json
+[
+    {"id":"1","firstName":"firstName1","lastName":"lastName1"},
+    {"id":"2","firstName":"firstName2","lastName":"lastName2"},
+    {"id":"3","firstName":"firstName3","lastName":"lastName3"},
+    {"id":"4","firstName":"firstName4","lastName":"lastName4"},
+    {"id":"5","firstName":"firstName5","lastName":"lastName5"},
+    {"id":"6","firstName":"firstName6","lastName":"lastName6"},
+    {"id":"7","firstName":"firstName7","lastName":"lastName7"},
+    {"id":"8","firstName":"firstName8","lastName":"lastName8"},
+    {"id":"9","firstName":"firstName9","lastName":"lastName9"},
+    {"id":"10","firstName":"firstName10","lastName":"lastName10"}
+]
+```
 ## Contributing
 
 Contributions are always welcome! Just fork the project, work on your feature/bug fix, and submit it.
 You can also contribute by creating issues. Please read the [contribution guidelines](.github/CONTRIBUTING.md) for more information.
 
 ## License
 
-Copyright (c) 2019  Tine  Kondo. Licensed under the MIT License (MIT)
+Copyright (c) 2019  Tine Kondo. Licensed under the MIT License (MIT)

core/src/main/java/com/github/spring/esdata/loader/core/EsDataFormat.java

@@ -0,0 +1,17 @@
+package com.github.spring.esdata.loader.core;
+
+/**
+ * Enum defining the data formats supported by the tool to load data into Elasticsearch.
+ */
+public enum EsDataFormat {
+  /***
+   * Format of data as dumped from an existing Elasticsearch server (using tools like `elasticdump` for e.g).
+   */
+  DUMP,
+  /**
+   * Format of data as manually created by User. Must be an array of JSON objects, each representing the actual content to put into related ES index.
+   */
+  MANUAL,
+
+  UNKNOWN
+}

core/src/main/java/com/github/spring/esdata/loader/core/IndexData.java

@@ -6,102 +6,111 @@
  * Data is basically represented by:
  * <ul>
  *      <li><code>esEntityClass</code>: mapping class, used to create the corresponding index and mapping</li>
- *      <li><code>location</code>: path to the JSON file that contains the actual data to import</li>
+ *      <li><code>location</code>: path to the JSON file that contains the actual data to import (can be gzipped)</li>
  *      <li><code>nbMaxItems</code> (<i>optional</i>): how many max items to load (<code>all</code> <i>by default</i> )</li>
  *      <li><code>nbSkipItems</code> (<i>optional</i>): how many items to skip (<code>0</code> <i>by default</i> )</li>
+ *      <li><code>format</code> (<i>optional</i>): format of the data to import (<code>null</code> <i>by default</i>, will be detected from JSON file content )</li>
  * </ul>
  *
  * @author tinesoft
- *
  */
 public class IndexData {
 
-	final Class<?> esEntityClass;
-	final String location;
-	final boolean gzipped;
-	final Long nbMaxItems;
-	final Long nbSkipItems;
+  final Class<?> esEntityClass;
+  final String location;
+  final boolean gzipped;
+  final Long nbMaxItems;
+  final Long nbSkipItems;
+  final EsDataFormat format;
+
+  /**
+   * @param esEntityClass mapping class of the data to be indexed in ES
+   * @param location      path to the file that contains data (as JSON) to be indexed
+   * @param gzipped       whether or not the data is gzipped (true by default)
+   * @param nbMaxItems    maximum number of items to load
+   * @param nbSkipItems   number of items to skip
+   * @param format        format of the data to load
+   */
+  public IndexData(final Class<?> esEntityClass, final String location, final boolean gzipped, final Long nbMaxItems,
+                   final Long nbSkipItems, final EsDataFormat format) {
+    this.esEntityClass = esEntityClass;
+    this.location = location;
+    this.gzipped = gzipped;
+    this.nbMaxItems = nbMaxItems;
+    this.nbSkipItems = nbSkipItems;
+    this.format = format;
+  }
+
+  /**
+   * Builds a new {@link IndexData} using provided parameters.
+   *
+   * @param esEntityClass mapping class of the data to be indexed in ES
+   * @param location      path to the file that contains data (as JSON) to be indexed
+   * @return a new {@link IndexData}
+   */
+  public static IndexData of(final Class<?> esEntityClass, final String location) {
+    return of(esEntityClass, location, Long.MAX_VALUE, 0L, null);
+  }
 
-	/**
-	 *
-	 * @param esEntityClass
-	 *            mapping class of the data to be indexed in ES
-	 * @param location
-	 *            path to the file that contains data (as JSON) to be indexed
-	 * @param gzipped
-	 *            whether or not the data is gzipped (true by default)
-	 * @param nbMaxItems
-	 *            maximum number of items to load
-	 * @param nbSkipItems
-	 *            number of items to skip
-	 */
-	public IndexData(final Class<?> esEntityClass, final String location, final boolean gzipped, final Long nbMaxItems,
-			final Long nbSkipItems) {
-		this.esEntityClass = esEntityClass;
-		this.location = location;
-		this.gzipped = gzipped;
-		this.nbMaxItems = nbMaxItems;
-		this.nbSkipItems = nbSkipItems;
-	}
+  /**
+   * Builds a new {@link IndexData} using provided parameters.
+   *
+   * @param esEntityClass mapping class of the data to be indexed in ES
+   * @param location      path to the file that contains data (as JSON) to be indexed
+   * @param nbMaxItems    maximum number of items to load
+   * @return a new {@link IndexData}
+   */
+  public static IndexData of(final Class<?> esEntityClass, final String location, final Long nbMaxItems) {
+    return of(esEntityClass, location, nbMaxItems, 0L, null);
+  }
 
-	/**
-	 * Builds a new {@link IndexData} using provided parameters.
-	 *
-	 * @param esEntityClass mapping class of the data to be indexed in ES
-	 * @param location      path to the file that contains data (as JSON) to be indexed
-	 * @return a new {@link IndexData}
-	 */
-	public static IndexData of(final Class<?> esEntityClass, final String location) {
-		return of(esEntityClass, location, Long.MAX_VALUE, 0L);
-	}
+  /**
+   * Builds a new {@link IndexData} using provided parameters.
+   *
+   * @param esEntityClass mapping class of the data to be indexed in ES
+   * @param location      path to the file that contains data (as JSON) to be indexed
+   * @param nbMaxItems    maximum number of items to load
+   * @param nbSkipItems   number of items to skip
+   * @param format        format of the data to load
+   * @return a new {@link IndexData}
+   */
+  public static IndexData of(final Class<?> esEntityClass, final String location, final Long nbMaxItems,
+                             final Long nbSkipItems, final EsDataFormat format) {
+    boolean gzipped = location.toLowerCase().endsWith(".gz");
+    return new IndexData(esEntityClass, location, gzipped, nbMaxItems, nbSkipItems, format);
+  }
 
-	/**
-	 * Builds a new {@link IndexData} using provided parameters.
-	 *
-	 * @param esEntityClass mapping class of the data to be indexed in ES
-	 * @param location      path to the file that contains data (as JSON) to be indexed
-	 * @param nbMaxItems    maximum number of items to load
-	 * @return a new {@link IndexData}
-	 */
-	public static IndexData of(final Class<?> esEntityClass, final String location, final Long nbMaxItems) {
-		return of(esEntityClass, location, nbMaxItems, 0L);
-	}
+  /**
+   * Builds a new {@link IndexData} using provided parameter.
+   *
+   * @param a {@link LoadEsData} to construct the data from
+   * @return a new {@link IndexData}
+   */
+  public static IndexData of(final LoadEsData a) {
+    return of(a.esEntityClass(), a.location(), a.nbMaxItems(), a.nbSkipItems(), a.format());
+  }
 
-	/**
-	 * Builds a new {@link IndexData} using provided parameters.
-	 *
-	 * @param esEntityClass mapping class of the data to be indexed in ES
-	 * @param location      path to the file that contains data (as JSON) to be indexed
-	 * @param nbMaxItems    maximum number of items to load
-	 * @param nbSkipItems   number of items to skip
-	 * @return a new {@link IndexData}
-	 */
-	public static IndexData of(final Class<?> esEntityClass, final String location, final Long nbMaxItems,
-							   final Long nbSkipItems) {
-		boolean gzipped = location.toLowerCase().endsWith(".gz");
-		return new IndexData(esEntityClass, location, gzipped, nbMaxItems, nbSkipItems);
-	}
+  public Class<?> getEsEntityClass() {
+    return this.esEntityClass;
+  }
 
-	/**
-	 * Builds a new {@link IndexData} using provided parameter.
-	 *
-	 * @param a {@link LoadEsData} to construct the data from
-	 * @return a new {@link IndexData}
-	 */
-	public static IndexData of(final LoadEsData a) {
-		return of(a.esEntityClass(), a.location(), a.nbMaxItems(), a.nbSkipItems());
-	}
+  public String getLocation() {
+    return this.location;
+  }
 
-	public Class<?> getEsEntityClass() {
-		return this.esEntityClass;
-	}
+  public boolean isGzipped() {
+    return this.gzipped;
+  }
 
-	public String getLocation() {
-		return this.location;
-	}
+  public Long getNbMaxItems() {
+    return this.nbMaxItems;
+  }
 
-	public boolean isGzipped() {
-		return this.gzipped;
-	}
+  public Long getNbSkipItems() {
+    return this.nbSkipItems;
+  }
 
-}
+  public EsDataFormat getFormat() {
+    return this.format;
+  }
+}

core/src/main/java/com/github/spring/esdata/loader/core/LoadEsData.java

@@ -1,10 +1,6 @@
 package com.github.spring.esdata.loader.core;
 
-import java.lang.annotation.ElementType;
-import java.lang.annotation.Repeatable;
-import java.lang.annotation.Retention;
-import java.lang.annotation.RetentionPolicy;
-import java.lang.annotation.Target;
+import java.lang.annotation.*;
 
 /**
  * {@code @LoadEsData} is a {@linkplain Repeatable repeatable} annotation
@@ -46,4 +42,11 @@
 	 */
 	long nbSkipItems() default 0;
 
+  /**
+   * Format of the data to load. If unspecified the library will try to guess the right format from the content of file at {@link #location()}.
+   *
+   * @return format of the data to load
+   */
+  EsDataFormat format() default EsDataFormat.UNKNOWN;
+
 }