Alpha

README.md

@@ -1,14 +1,16 @@
-# Titli
-A toolkit for hosting feature extraction, model training, model inference, and model evaluation of AI-based Intrusion Detection Systems
-<p align="center">
-	<img src="assets/images/pipeline-overview.jpg" alt="Pipeline Overview" width="800" />
-</p>
+<img src="assets/images/ titli-logo.png" alt="Titli Logo" width="150" />
+
 
 ![PyPI - Python Version](https://img.shields.io/pypi/pyversions/titli)
 ![PyPI - Version](https://img.shields.io/pypi/v/titli)
 ![GitHub License](https://img.shields.io/github/license/spg-iitd/titli)
 [![Documentation Status](https://readthedocs.org/projects/titli/badge/?version=latest)](https://titli.readthedocs.io/en/latest/?badge=latest)
 
+A toolkit for hosting feature extraction, model training, model inference, and model evaluation of AI-based Intrusion Detection Systems
+<p align="center">
+	<img src="assets/images/pipeline-overview.jpg" alt="Pipeline Overview" width="800" />
+</p>
+
 ## Documentation
 
 📚 **[Read the full documentation](docs/)** to get started with Titli.
@@ -35,10 +37,10 @@ pip install titli
 ```
 
 ### Usage
-- Step 1: Copy the ```examples/train_ids.py``` and ```examples/test_ids.py``` file from the repo to your local machine.
-- Step 2: Run both the files to train and test the Kitsune IDS respectively.
+- Step 1: Copy the ```test_titli.ipynb``` file and run it locally.
+- Step 2: Run both the files to train and test IDSs.
 
-### Todo (Developer Tasks)
+<!-- ### Todo (Developer Tasks)
 - [ ] Check if RMSE is used for loss or just the difference.
 - [ ] Put Kitsune code into the base IDS format.
 - [ ] Write code to evaluate the model and calculate all the metrics.  
@@ -47,5 +49,8 @@ pip install titli
 - [ ] Resolve BaseSKLearn class infer function's TODO
 - [ ] Make fit and predict function as private by putting "_" in the starting
 - [ ] Similar to PyTorch, define __call__ function for SkLearn models too
+- [ ] Investigate and modify the use of get_model in the Pytorch class.
+- [ ] Make internal methods private
+- [ ] Add docstrings in the class and functions
 
-- Write where the model is saved in the print statement!
+- Write where the model is saved in the print statement! -->

docs/source/api/fe.rst

@@ -63,17 +63,14 @@ Usage Examples
 Basic Feature Extraction
 ~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Extract features from a PCAP file:
+Extract features from a PCAP file and output to CSV for DataLoader usage:
 
 .. code-block:: python
 
    from titli.fe import AfterImage
 
-   fe = AfterImage(
-       file_path="traffic.pcap",
-       dataset_name="my_dataset"
-   )
-   features = fe.extract_features()
+   fe = AfterImage(file_path="traffic.pcap")
+   fe.extract_features(output_path="features.csv")
 
 With Custom Parameters
 ~~~~~~~~~~~~~~~~~~~~~~
@@ -84,37 +81,37 @@ With Custom Parameters
 
    fe = AfterImage(
        file_path="traffic.pcap",
-       dataset_name="my_dataset",
        decay_factors=[5, 3, 1, 0.1, 0.01],
        max_pkt=100000,
        limit=10000
    )
-   features = fe.extract_features()
+   fe.extract_features(output_path="features.csv")
 
-Stateful Extraction
-~~~~~~~~~~~~~~~~~~~
+Integration with DataLoader
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Save and reuse the feature extractor state:
+Use extracted features with DataLoader for model training:
 
 .. code-block:: python
 
-   import pickle
    from titli.fe import AfterImage
+   from titli.utils import StreamingCSVDataset
+   from torch.utils.data import DataLoader
 
-   # Training phase
-   train_fe = AfterImage(file_path="train.pcap", dataset_name="train")
-   train_features = train_fe.extract_features()
+   # Extract features
+   fe = AfterImage(file_path="traffic.pcap")
+   fe.extract_features(output_path="features.csv")
 
-   with open("state.pkl", "wb") as f:
-       pickle.dump(train_fe.state, f)
-
-   # Test phase
-   with open("state.pkl", "rb") as f:
-       state = pickle.load(f)
-
-   test_fe = AfterImage(
-       file_path="test.pcap",
-       dataset_name="test",
-       state=state
+   # Load with DataLoader
+   dataset = StreamingCSVDataset(
+       feature_csv_path="features.csv",
+       label_csv_path="labels.csv",
+       label_column=0
    )
-   test_features = test_fe.extract_features()
+   loader = DataLoader(dataset, batch_size=32)
+
+.. note::
+
+   Feature extractors output CSV files that should be consumed via ``StreamingCSVDataset`` 
+   and ``DataLoader`` for model training. Direct use of extracted features is discouraged.
+   State management is now handled internally by the feature extractors.