update doc

README.md

@@ -15,11 +15,12 @@
 ---
 
 ## libCacheSim features 
-* **high performance** - over 20M requests/sec for a realistic trace replay. 
-* **high memory efficiency** - predictable and small memory footprint. 
-* **State-of-the-art algorithms** - eviction algorithms, admission algorithms, sampling techniques, approximate miss ratio computation. 
-* **Simple API** - easy to build cache clusters, multi-layer caching, etc.
-* **extensible** - easy to support new trace types or eviction algorithms.
+* **High performance** - over 20M requests/sec for a realistic trace replay. 
+* **High memory efficiency** - predictable and small memory footprint. 
+* **State-of-the-art algorithms** - eviction algorithms, admission algorithms, sampling techniques, approximate miss ratio computation, see [here](/doc/quickstart_cachesim.md). 
+* **The ONLY feature-rich trace analyzer** - all types of trace analysis you need, see [here](/doc/quickstart_traceAnalyzer.md).
+* **Simple API** - easy to build cache clusters, multi-layer caching, etc, see [here](/doc/API.md).
+* **Extensible** - easy to support new trace types or eviction algorithms, see [here](/doc/advanced_lib_extend.md).
 ---
 
 ## Supported algorithms
@@ -46,13 +47,18 @@ cachesim supports the following algorithms:
 
 
 ## Build and Install libCacheSim
-### Install dependency
-libCacheSim uses [cmake](https://cmake.org/) build system and has a few dependencies: 
-[glib](https://developer.gnome.org/glib/)
-[tcmalloc](https://github.com/google/tcmalloc), 
-[zstd](https://github.com/facebook/zstd).
+### One-line install
+We provide some scripts for quick installation of libCacheSim. 
+```bash
+cd scripts && bash install_dependency.sh && bash install_libcachesim.sh;
+```
+If this does not work, please 
+1. let us know what system you are using and what error you get
+2. read the following sections for self-installation.
 
-Please see [install.md](doc/install.md) for how to install the dependencies. 
+### Install dependency
+libCacheSim uses [cmake](https://cmake.org/) build system and has a few dependencies: [glib](https://developer.gnome.org/glib/), [tcmalloc](https://github.com/google/tcmalloc), [zstd](https://github.com/facebook/zstd).
+Please see [install.md](/doc/install.md) for how to install the dependencies. 
 
 
 ### Build libCacheSim
@@ -133,7 +139,7 @@ See [trace analysis](/doc/quickstart_traceAnalyzer.md) for more details.
 
 ### Using libCacheSim as a library 
 libCacheSim can be used as a library for building cache simulators. 
-For example, you can build a cache cluster with consistent hashing, or a multi-layer caching simulator.
+For example, you can build a cache cluster with consistent hashing, or a multi-layer cache simulator.
 
 Here is a simplified example showing the basic APIs. 
 ```c 
@@ -150,63 +156,40 @@ common_cache_params_t cc_params = {.cache_size=1024*1024U};
 cache_t *cache = LRU_init(cc_params, NULL); 
 
 /* counters */
-uint64_t req_byte = 0, miss_byte = 0;
 uint64_t n_req = 0, n_miss = 0;
 
 /* loop through the trace */
 while (read_one_req(reader, req) == 0) {
-    if (cache->get(cache, req) == cache_ck_miss) {
-        miss_byte += req->obj_size;
+    if (!cache->get(cache, req)) {
         n_miss++;
     }
-    req_byte += req->obj_size; 
     n_req++;
 }
 
-printf("miss ratio: %.4lf, byte miss ratio %.4lf\n", 
-        (double)n_miss / n_req, (double)miss_byte / req_byte);
+printf("miss ratio: %.4lf\n", (double)n_miss / n_req);
 
 /* cleaning */
 close_trace(reader);
 free_request(req);
 cache->cache_free(cache);
 ```
+
 save this to `test.c` and compile with 
-```
+``` bash
 gcc test.c $(pkg-config --cflags --libs libCacheSim glib-2.0) -o test.out
 ```
 
-if you get `error while loading shared libraries`, run `sudo ldconfig`
-
-See [quickstart](doc/quickstart_lib.md) for more details. 
-And see [example folder](example) for examples on how to use libCacheSim, such as cache cluster with consistent hashing, multi-layer caching simulators. 
+See [here](/doc/advanced_lib.md) for more details, and see [example folder](/example) for examples on how to use libCacheSim, such as building a cache cluster with consistent hashing, multi-layer cache simulators. 
 
 ---
 
 
-### Extending libCacheSim 
-#### Adding new trace types
-libCacheSim supports txt, csv, and binary traces. We prefer binary traces because it allows libCacheSim to run faster, and the traces are more compact. 
-
-We also support zstd compressed traces with decompression first, this allows you to store the traces with less space.
-
-You should not need to add support to use a new trace if you follow the (cachesim quick start tutorial)[doc/quickstart_cachesim.md] and (libCacheSim quick start tutorial)[doc/quickstart_libcachesim].
-
-But if you ever need to add a new trace type, please see [`libCacheSim/traceReader/customizedReader/akamaiBin.h`](libCacheSim/traceReader/customizedReader/akamaiBin.h) for an example reader.
-
-#### Adding new eviction algorithms
-Adding eviction algorithm is easy, you can see [`libCacheSim/cache/eviction/LRU.c`](/libCacheSim/cache/eviction/LRU.c) for an example.
-Besides implementing the a new eviction algorithm in `libCacheSim/cache/eviction/myCache.c`, you also need to perform the following tasks.
-1. Add the `myCache_init()` function to [`libCacheSim/include/libCacheSim/evictionAlgo.h`](/libCacheSim/include/libCacheSim/evictionAlgo.h).
-2. Add the mycache.c to [`libCacheSim/cache/eviction/CMakeLists.txt`](/libCacheSim/cache/eviction/CMakeLists.txt) so that it can be compiled.
-3. Add the option to use mycache in `cachesim` in [`libCacheSim/bin/cachesim/cli.c`](/libCacheSim/bin/cachesim/cli.c).
-4. If you are creating a pull request, you would also need to add a test in [`test/test_evictionAlgo.c`](/test/test_evictionAlgo.c) and add the algorithm to this README. 
+### Extending libCacheSim (new algorithms and trace types)
+libCacheSim supports *txt*, *csv*, and *binary* traces. We prefer binary traces because it allows libCacheSim to run faster, and the traces are more compact. 
 
-#### Adding new eviction algorithms in C++
-You can also write your eviction algorithm in C++ and use it in libCacheSim.
-You can see [`libCacheSim/cache/eviction/cpp/LFU.cpp`](/libCacheSim/cache/eviction/cpp/LFU.cpp) for an example.
+We also support zstd compressed binary traces without decompression. This allows you to store the traces with less space.
 
-For further reading on how to use libCacheSim, please see the [quick start libCacheSim](/doc/quickstart_libcachesim.md).
+If you need to add a new trace type, please see [here](/doc/advanced_lib_extend.md) for details.
 
 
 ---

doc/lib.md → doc/advanced_lib.md

@@ -228,6 +228,11 @@ Such as in the `_build` directory:
 export PKG_CONFIG_PATH=$PWD
 ```
 
+#### Possible problems
+* if you get `error while loading shared libraries`, run `sudo ldconfig`
+
+
+
 ---
 
 

doc/advanced_lib_extend.md

@@ -0,0 +1,79 @@
+# Extend libCacheSim 
+
+
+## Add new eviction algorithms in C
+Implementing a new eviction algorithm is easy in libCacheSim. You can start from [LRU](/libCacheSim/cache/eviction/LRU.c) as the template, copy to `myCache.c` and change the functions. The cache interface requires the developer to implement the following functions: 
+
+```c
+/* initialize all the variables */
+cache_t *LRU_init(const common_cache_params_t ccache_params, const char *cache_specific_params);
+
+/* free the resources used by the cache */
+void LRU_free(cache_t *cache);
+
+/* get the object from the cache, it is find + on-demand insert/evict, return true if cache hit */
+bool LRU_get(cache_t *cache, const request_t *req);
+
+/* find an object in the cache, return the cache object if found, NULL otherwise, update_cache means whether update the cache state, e.g., moving object to the head of the queue */
+cache_obj_t *LRU_find(cache_t *cache, const request_t *req, const bool update_cache);
+
+/* insert an object to the cache, return the cache object, this assumes the object is not in the cache */
+cache_obj_t *LRU_insert(cache_t *cache, const request_t *req);
+
+/* find the object to be evicted, return the cache object, not used very often */
+cache_obj_t *LRU_to_evict(cache_t *cache, const request_t *req);
+
+/* evict an object from the cache, req should not be used */
+void LRU_evict(cache_t *cache, const request_t *req);
+
+/* remove an object from the cache, return true if the object is found and removed, note that this is used for user-triggered remove, eviction should use evict */
+bool LRU_remove(cache_t *cache, const obj_id_t obj_id);
+```
+
+Specifically, you can following the steps:
+1. Add a new file e.g., `mycache.c` to [cache/eviction/](/libCacheSim/cache/eviction/) for your cache eviction algorithm implementation. 
+2. If your cache eviction algorithm needs extra metadata, add a new object metadata struct in 
+   [include/libCacheSim/cacheObj.h](/libCacheSim/include/libCacheSim/cacheObj.h).
+3. Add `myCache_init()` function to [include/libCacheSim/evictionAlgo.h](/libCacheSim/include/libCacheSim/evictionAlgo.h).
+4. Add mycache.c to [CMakeLists.txt](/libCacheSim/cache/eviction/CMakeLists.txt) so that it can be compiled.
+5. Add command line option in [bin/cachesim/cli.c](/libCacheSim/bin/cachesim/cli.c) so that you can use `cachesim` binary.
+6. Remember to add a test in [test/test_evictionAlgo.c](/test/test_evictionAlgo.c) and add the algorithm to this [README](README.md). 
+
+---
+
+## Add new eviction algorithms in C++
+While most of the eviction algorithms in libCacheSim is written in C, there are a few written in C++, especially the ones that are ported from the original user, e.g., [LHD](/libCacheSim/cache/eviction/LHD/), [LRB](/libCacheSim/cache/eviction/LRB/).
+
+You can also write your eviction algorithm in C++ and use it in libCacheSim. We have provided a few C++ eviction algorithms as examples in [cache/eviction/cpp/](/libCacheSim/cache/eviction/cpp/), e.g., [LFU](/libCacheSim/cache/eviction/cpp/LFU.cpp), [LRU-K](/libCacheSim/cache/eviction/cpp/LRU_K.cpp).
+
+There are two steps you can follow, 
+1. implement most functions in C++ 
+2. implement the libCacheSim cache interface in C, e.g., `mycache_init()`, `mycache_get()`. 
+
+
+
+---
+
+## Add new trace readers 
+libCacheSim supports [txt](/libCacheSim/traceReader/generalReader/txt.c), [csv](/libCacheSim/traceReader/generalReader/csv.c), and binary traces. We prefer binary traces because it allows libCacheSim to run faster, and the traces are more compact. 
+For binary traces, libCacheSim also supports zstd compressed traces without decompression.
+
+But if you ever need to implement a new trace type, please see [here](/libCacheSim/traceReader/customizedReader/akamaiBin.h) for an example reader. 
+
+To implement a reader, you need to implement two functions:
+```c
+/* initialize the reader, return 0 if success, 1 otherwise */
+int myReader_setup(reader_t *reader);
+
+/* read one request from the trace, return 0 if success, 1 otherwise */
+int myReader_read_one_req(reader_t *reader, request_t *req);
+
+```
+
+Here are the steps to add a new trace reader:
+1. add a new new trace type, e.g., `MYREADER_TRACE` in `trace_type_e` in [include/libCacheSim/enum.h](/libCacheSim/include/libCacheSim/enum.h). 
+2. add a new reader file, e.g., `myReader.h` in [traceReader/customizedReader/](/libCacheSim/traceReader/customizedReader/) and implement the two functions.
+3. add `myReader_setup()` to `setup_reader()`and `myReader_read_one_req()` to `read_one_req()` in [traceReader/reader.c](/libCacheSim/traceReader/reader.c). 
+4. add `MYREADER_TRACE` to `trace_type_str_to_enum()` in [bin/cli_reader_utils.c](/libCacheSim/bin/cli_reader_utils.c)
+
+

doc/quickstart_cachesim.md

@@ -11,7 +11,7 @@ Meanwhile, cachesim has high-performance with low resource usages.
 ---
 
 ## Installation
-First, [build libCacheSim](../README.md). After building libCacheSim, `cachesim` should be in the build directory. 
+First, [build libCacheSim](/doc/install.md). After building libCacheSim, `cachesim` should be in the build directory. 
 
 ---
 
@@ -58,21 +58,21 @@ You can enable this feature by setting cache size to 0 or auto.
 
 ### Use different eviction algorithms
 cachesim supports the following algorithms:
-[FIFO](../libCacheSim/libCacheSim/cache/eviction/FIFO.c), 
-[LRU](../libCacheSim/libCacheSim/cache/eviction/LRU.c), 
-[Clock](../libCacheSim/libCacheSim/cache/eviction/Clock.c),
-[LFU](../libCacheSim/libCacheSim/cache/eviction/LFU.c), 
-[ARC](../libCacheSim/libCacheSim/cache/eviction/ARC.c), 
-[SLRU](../libCacheSim/libCacheSim/cache/eviction/SLRU.c), 
-[GDSF](../libCacheSim/libCacheSim/cache/eviction/GDSF.c), 
-[TinyLFU](../libCacheSim/libCacheSim/cache/eviction/TinyLFU.c), 
-[LeCaR](../libCacheSim/libCacheSim/cache/eviction/LeCaR.c), 
-[Cacheus](../libCacheSim/libCacheSim/cache/eviction/Cacheus.c), 
-[Hyperbolic](../libCacheSim/libCacheSim/cache/eviction/Hyperbolic.c), 
-[LHD](../libCacheSim/libCacheSim/cache/eviction/LHD/LHDInterface.cpp), 
-[GLCache](../libCacheSim/libCacheSim/cache/eviction/GLCache/GLCache.c),
-[Belady](../libCacheSim/libCacheSim/cache/eviction/Belady.c), 
-[BeladySize](../libCacheSim/libCacheSim/cache/eviction/BeladySize.c), 
+* [FIFO](/libCacheSim/cache/eviction/FIFO.c)
+* [LRU](/libCacheSim/cache/eviction/LRU.c)
+* [Clock](/libCacheSim/cache/eviction/Clock.c)
+* [LFU](/libCacheSim/cache/eviction/LFU.c)
+* [ARC](/libCacheSim/cache/eviction/ARC.c)
+* [SLRU](/libCacheSim/cache/eviction/SLRU.c)
+* [GDSF](/libCacheSim/cache/eviction/GDSF.c)
+* [TinyLFU](/libCacheSim/cache/eviction/TinyLFU.c)
+* [LeCaR](/libCacheSim/cache/eviction/LeCaR.c)
+* [Cacheus](/libCacheSim/cache/eviction/Cacheus.c)
+* [Hyperbolic](/libCacheSim/cache/eviction/Hyperbolic.c)
+* [LHD](/libCacheSim/cache/eviction/LHD/LHDInterface.cpp)
+* [GLCache](/libCacheSim/cache/eviction/GLCache/GLCache.c)
+* [Belady](/libCacheSim/cache/eviction/Belady.c)
+* [BeladySize](/libCacheSim/cache/eviction/BeladySize.c)
 
 You can just use the algorithm name as the eviction algorithm parameter, for example  
 

doc/quickstart_traceAnalyzer.md

@@ -2,7 +2,7 @@
 ## Trace analysis tool
 libCacheSim provides a set of tools to help you analyze traces. After building the project, you can find a binary called `traceAnalyzer`.
 This doc shows how to use the tool.
-If you are interested, the source code is located in the [libCacheSim/bin/traceAnalyzer/](/libCacheSim/bin/traceAnalyzer) and [libCacheSim/traceAnalyzer](/libCacheSim/traceAnalyzer) directory.
+If you are interested, the source code is located in the [bin/traceAnalyzer/](/libCacheSim/bin/traceAnalyzer) and [traceAnalyzer](/libCacheSim/traceAnalyzer) directory.
 
 ### Obtain trace statistics
 #### Usage: 

libCacheSim/traceReader/customizedReader/akamaiBin.h

@@ -5,7 +5,7 @@ extern "C" {
 
 /*
  * format
- * struct {
+ * struct req {
  *  uint32_t real_time;
  *  uint64_t obj_id;
  *  uint32_t size;
@@ -21,30 +21,59 @@ extern "C" {
 #include "../../include/libCacheSim/reader.h"
 #include "binaryUtils.h"
 
+/**
+ * @brief setup the reader for akamai trace
+ *
+ * @param reader
+ * @return int
+ */
 static inline int akamaiReader_setup(reader_t *reader) {
+  /* set the trace format */
   reader->trace_type = AKAMAI_TRACE;
+
+  /* most added traces are binary, the other format is TXT_TRACE_FORMAT  */
   reader->trace_format = BINARY_TRACE_FORMAT;
+
+  /* the size of one request in the trace, this is sizeof(struct req) */
   reader->item_size = 22; /* IqIhhh */
+
+  /* calculate the number of requests in the trace */
   reader->n_total_req = (uint64_t)reader->file_size / (reader->item_size);
+
+  /* the object id is numerical so do not convert it again, this is useful for
+   * TXT_TRACE_FORMAT */
   reader->obj_id_is_num = true;
+
   return 0;
 }
 
+/**
+ * @brief read one request from the trace
+ *
+ * @param reader
+ * @param req
+ * @return int
+ */
 static inline int akamai_read_one_req(reader_t *reader, request_t *req) {
+  /* read reader->item_size bytes from the trace */
   char *record = read_bytes(reader);
 
   if (record == NULL) {
+    /* end of the trace or error */
     req->valid = false;
     return 1;
   }
 
+  /* read the trace */
   req->clock_time = *(uint32_t *)record;
   req->obj_id = *(uint64_t *)(record + 4);
   req->obj_size = *(uint32_t *)(record + 12);
   req->tenant_id = *(uint16_t *)(record + 16);
   req->bucket_id = *(uint16_t *)(record + 18);
   req->content_type = *(uint16_t *)(record + 20);
 
+  /* if we read a request of size 0 and the trace is reading forward,
+     read the next request */
   if (req->obj_size == 0 && reader->ignore_size_zero_req &&
       reader->read_direction == READ_FORWARD)
     return akamai_read_one_req(reader, req);