-
Notifications
You must be signed in to change notification settings - Fork 15
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
109 changed files
with
2,147 additions
and
2,804 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,107 +1,57 @@ | ||
| # Extend Metrics | ||
|
|
||
| To implement a new metric, create a new class that extends the abstract class `AbstractMetric`: | ||
| To implement a new metric, create a new class that extends the abstract class `Metric`: | ||
|
|
||
| ```java | ||
| package org.benchmark.metric; | ||
|
|
||
| @Shorthand("MyMetric") | ||
| public class MyMetric extends AbstractMetric{ | ||
| public class MyMetric extends Metric { | ||
|
|
||
| @Override | ||
| public void receiveData(Properties p) { | ||
| // ... | ||
| } | ||
|
|
||
| @Override | ||
| public void close() { | ||
| callbackClose(); | ||
| super.close(); | ||
|
|
||
| } | ||
|
|
||
| protected void callbackClose() { | ||
| // your close method | ||
| } | ||
| public MyMetric() { | ||
| super("name", "abbreviation", "description"); | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ## Receive Data | ||
|
|
||
| This method will receive all the results during the benchmark. | ||
|
|
||
| You'll receive a few values regarding each query execution. Those values include the amount of time the execution took, if it succeeded, and if not, the reason why it failed, which can be either a timeout, a wrong HTTP Code or an unknown error. | ||
| Further on you also receive the result size of the query. | ||
|
|
||
| If your metric is a single value metric, you can use the `processData` method, which will automatically add each value together. | ||
| However, if your metric is query specific, you can use the `addDataToContainter` method. (Look at the [QPSMetric](https://github.com/dice-group/IGUANA/blob/master/iguana.resultprocessor/src/main/java/org/aksw/iguana/rp/metrics/impl/QPSMetric.java)) | ||
| You can then choose if the metric is supposed to be calculated for each Query, Worker | ||
| or Task by implementing the appropriate interfaces: `QueryMetric`, `WorkerMetric`, `TaskMetric`. | ||
|
|
||
| Be aware that both methods will save the results for each used worker. This allows the calculation of the overall metric, as well as the metric for each worker itself. | ||
| You can also choose to implement the `ModelWritingMetric` interface, if you want your | ||
| metric to create a special RDF model, that you want to be added to the result model. | ||
|
|
||
| We will stick to the single-value metric for now. | ||
|
|
||
|
|
||
| The following shows an example, that retrieves every possible value and saves the time and success: | ||
| The following gives you an examples on how to work with the `data` parameter: | ||
|
|
||
| ```java | ||
| @Override | ||
| public void receiveData(Properties p) { | ||
|
|
||
| double time = Double.parseDouble(p.get(COMMON.RECEIVE_DATA_TIME).toString()); | ||
| long tmpSuccess = Long.parseLong(p.get(COMMON.RECEIVE_DATA_SUCCESS).toString()); | ||
| long success = (tmpSuccess > 0) ? 1 : 0; | ||
| long failure = (success == 1) ? 0 : 1; | ||
| long timeout = (tmpSuccess == COMMON.QUERY_SOCKET_TIMEOUT) ? 1 : 0; | ||
| long unknown = (tmpSuccess == COMMON.QUERY_UNKNOWN_EXCEPTION) ? 1 : 0; | ||
| long wrongCode = (tmpSuccess == COMMON.QUERY_HTTP_FAILURE) ? 1 : 0; | ||
|
|
||
| if(p.containsKey(COMMON.RECEIVE_DATA_SIZE)) { | ||
| size = Long.parseLong(p.get(COMMON.RECEIVE_DATA_SIZE).toString()); | ||
| @Override | ||
| public Number calculateTaskMetric(StresstestMetadata task, List<QueryExecutionStats>[][] data) { | ||
| for (WorkerMetadata worker : task.workers()) { | ||
| for (int i = 0; i < worker.noOfQueries(); i++) { | ||
| // This list contains every query execution statistics of one query | ||
| // from the current worker | ||
| List<QueryExecutionStats> execs = data[worker.workerID()][i]; | ||
| } | ||
| } | ||
| return BigInteger.ZERO; | ||
| } | ||
|
|
||
| Properties results = new Properties(); | ||
| results.put(TOTAL_TIME, time); | ||
| results.put(TOTAL_SUCCESS, success); | ||
|
|
||
| Properties extra = getExtraMeta(p); | ||
| processData(extra, results); | ||
| } | ||
| ``` | ||
|
|
||
| ## Close | ||
|
|
||
| In this method you should calculate your metric and send the results. | ||
| An example: | ||
|
|
||
| ```java | ||
| protected void callbackClose() { | ||
| // create a model that contains the results | ||
| Model m = ModelFactory.createDefaultModel(); | ||
|
|
||
| Property property = getMetricProperty(); | ||
| Double sum = 0.0; | ||
|
|
||
| // Go over each worker and add metric results to model | ||
| for(Properties key : dataContainer.keySet()){ | ||
| Double totalTime = (Double) dataContainer.get(key).get(TOTAL_TIME); | ||
| Integer success = (Integer) dataContainer.get(key).get(TOTAL_SUCCESS); | ||
|
|
||
| Double noOfQueriesPerHour = hourInMS * success * 1.0 / totalTime; | ||
| sum += noOfQueriesPerHour; | ||
| Resource subject = getSubject(key); | ||
|
|
||
| m.add(getConnectingStatement(subject)); | ||
| m.add(subject, property, ResourceFactory.createTypedLiteral(noOfQueriesPerHour)); | ||
| @Override | ||
| public Number calculateWorkerMetric(WorkerMetadata worker, List<QueryExecutionStats>[] data) { | ||
| for (int i = 0; i < worker.noOfQueries(); i++) { | ||
| // This list contains every query execution statistics of one query | ||
| // from the given worker | ||
| List<QueryExecutionStats> execs = data[i]; | ||
| } | ||
| return BigInteger.ZERO; | ||
| } | ||
|
|
||
| // Add overall metric to model | ||
| m.add(getTaskResource(), property, ResourceFactory.createTypedLiteral(sum)); | ||
|
|
||
| // Send data to storage | ||
| sendData(m); | ||
| } | ||
| ``` | ||
|
|
||
| ## Constructor | ||
|
|
||
| The constructor parameters are provided the same way as for the tasks. Thus, simply look at the [Extend Task](../extend-task) page. | ||
| @Override | ||
| @Nonnull | ||
| public Model createMetricModel(StresstestMetadata task, Map<String, List<QueryExecutionStats>> data) { | ||
| for (String queryID : task.queryIDS()) { | ||
| // This list contains every query execution statistics of one query from | ||
| // every worker that executed this querys | ||
| List<QueryExecutionStats> execs = data.get(queryID); | ||
| } | ||
| } | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,47 +1,23 @@ | ||
| # Extend Result Storages | ||
|
|
||
| If you want to use a different storage other than RDF, you can implement a different storage solution. | ||
|
|
||
| The current implementation of Iguana is highly optimized for RDF, thus we recommend you to work on top of the `TripleBasedStorage` class: | ||
| If you want to use a different storage other than RDF, you can implement a different storage solution. | ||
|
|
||
| ```java | ||
| package org.benchmark.storage; | ||
|
|
||
| @Shorthand("MyStorage") | ||
| public class MyStorage extends TripleBasedStorage { | ||
|
|
||
| @Override | ||
| public void commit() { | ||
|
|
||
| } | ||
|
|
||
| @Override | ||
| public String toString(){ | ||
| return this.getClass().getSimpleName(); | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ## Commit | ||
| public class MyStorage implements Storage { | ||
|
|
||
| This method should take all the current results, store them, and remove them from the memory. | ||
|
|
||
| You can access the results at the Jena Model `this.metricResults`. | ||
|
|
||
| For example: | ||
|
|
||
| ```java | ||
| @Override | ||
| public void commit() { | ||
| try (OutputStream os = new FileOutputStream(file.toString(), true)) { | ||
| RDFDataMgr.write(os, metricResults, RDFFormat.NTRIPLES); | ||
| metricResults.removeAll(); | ||
| } catch (IOException e) { | ||
| LOGGER.error("Could not commit to NTFileStorage.", e); | ||
| @Override | ||
| public void storeResults(Model m) { | ||
| // method for storing model | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| The method `storeResults` will be called at the end of the task. The model from | ||
| the parameter contains the final result model for that task. | ||
|
|
||
| ## Constructor | ||
|
|
||
| The constructor parameters are provided the same way as for the tasks. Thus, simply look at the [Extend Task](../extend-task) page. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.