Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ROCES and NCES2 #511

Merged
merged 28 commits into from
Feb 4, 2025
Merged

Add ROCES and NCES2 #511

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
c74de16
add initial methods for NCES2 and ROCES
Jean-KOUAGOU Dec 6, 2024
4c9a9d2
add Dataset class for ROCES and further dev of NCES2 and ROCES
Jean-KOUAGOU Dec 9, 2024
7c81717
continued integration of NCES2 and ROCES
Jean-KOUAGOU Dec 10, 2024
5ac3ed8
continue implementation of NCES2 and ROCES
Jean-KOUAGOU Dec 11, 2024
2c882e1
work on trainer for all neural class expression synthesizers
Jean-KOUAGOU Dec 12, 2024
3bb2de1
update ncestrainer to accomodate nces2 and roces
Jean-KOUAGOU Dec 16, 2024
bfa492b
Merge remote-tracking branch 'origin/develop' into add_roces_nces2
Jean-KOUAGOU Dec 16, 2024
94a06e6
test nces trainer with new implementation, NCES works
Jean-KOUAGOU Dec 17, 2024
a10fae8
update NCES2 and ROCES scripts. Training NCES2 and ROCES works!
Jean-KOUAGOU Dec 18, 2024
4a2c847
Completed integration. Tests are left to make sure everything runs pr…
Jean-KOUAGOU Dec 19, 2024
d86ffe6
working on tests
Jean-KOUAGOU Dec 20, 2024
ede1978
update test scripts
Jean-KOUAGOU Jan 6, 2025
97a7174
test nces, nces2, roces working.
Jean-KOUAGOU Jan 7, 2025
5caefa4
update nces_trainer test script
Jean-KOUAGOU Jan 7, 2025
4b5fa54
update more scripts to accomodate integration of NCES2 and ROCES
Jean-KOUAGOU Jan 10, 2025
261e436
add default value in args
Jean-KOUAGOU Jan 13, 2025
af46315
update NCES2 and write config files
Jean-KOUAGOU Jan 17, 2025
74c4395
change learning rate scheduler to cosine in NCES, NCES2, ROCES
Jean-KOUAGOU Jan 17, 2025
3f596a0
add sampling set sizes for ROCES in config files
Jean-KOUAGOU Jan 20, 2025
1398ff5
remove outdated nces scripts and unify training script for NCES, NCES…
Jean-KOUAGOU Jan 21, 2025
5b1b47c
fix but in NCES dataset class
Jean-KOUAGOU Jan 22, 2025
990b854
fix but in NCES
Jean-KOUAGOU Jan 22, 2025
186ac9a
update inference scripts for NCES
Jean-KOUAGOU Jan 23, 2025
a46531f
review changes
alkidbaci Feb 3, 2025
5fafab1
Update README.md
Jean-KOUAGOU Feb 3, 2025
ddfb426
fix typo in train_nces.py
Jean-KOUAGOU Jan 24, 2025
478dc8c
add benchmarking results for NCES2, ROCES, and update concept_learnin…
Jean-KOUAGOU Feb 3, 2025
6424a1f
refactoring due to circular imports
alkidbaci Feb 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 43 additions & 25 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@ $$\forall p \in E^+\ \mathcal{K} \models H(p) \wedge \forall n \in E^-\ \mathcal
To tackle this supervised learning problem, ontolearn offers many symbolic, neuro-symbolic and deep learning based Learning algorithms:
- **Drill** → [Neuro-Symbolic Class Expression Learning](https://www.ijcai.org/proceedings/2023/0403.pdf)
- **EvoLearner** → [EvoLearner: Learning Description Logics with Evolutionary Algorithms](https://dl.acm.org/doi/abs/10.1145/3485447.3511925)
- **NCES2** → (soon) [Neural Class Expression Synthesis in ALCHIQ(D)](https://papers.dice-research.org/2023/ECML_NCES2/NCES2_public.pdf)
- **NCES2** → [Neural Class Expression Synthesis in ALCHIQ(D)](https://papers.dice-research.org/2023/ECML_NCES2/NCES2_public.pdf)
- **ROCES** → [Robust Class Expression Synthesis in Description Logics via Iterative Sampling](https://www.ijcai.org/proceedings/2024/0479.pdf)
- **NCES** → [Neural Class Expression Synthesis](https://link.springer.com/chapter/10.1007/978-3-031-33455-9_13)
- **NERO** → (soon) [Learning Permutation-Invariant Embeddings for Description Logic Concepts](https://link.springer.com/chapter/10.1007/978-3-031-30047-9_9)
- **CLIP** → [Learning Concept Lengths Accelerates Concept Learning in ALC](https://link.springer.com/chapter/10.1007/978-3-031-06981-9_14)
Expand Down Expand Up @@ -249,46 +250,47 @@ print(df[[col for col in df if col.startswith('Test-F1') or col.startswith('RT')

Below, we report the average test F1 score and the average runtimes of learners.

| LP | Test-F1-OCEL | RT-OCEL | Test-F1-CELOE | RT-CELOE | Test-F1-Evo | RT-Evo | Test-F1-DRILL | RT-DRILL | Test-F1-TDL | RT-TDL | Test-F1-NCES | RT-NCES | Test-F1-CLIP | RT-CLIP |
|:------------------:|-------------:|--------:|--------------:|---------:|------------:|-------:|--------------:|---------:|------------:|-------:|-------------:|--------:|-------------:|--------:|
| Aunt | 0.614 | 13.697 | 0.855 | 13.697 | 0.978 | 5.278 | 0.811 | 60.351 | 0.956 | 0.118 | 0.812 | 1.168 | 0.855 | 14.059 |
| Cousin | 0.712 | 10.846 | 0.789 | 10.846 | 0.993 | 3.311 | 0.701 | 60.485 | 0.820 | 0.176 | 0.677 | 1.050 | 0.779 | 9.050 |
| Grandgranddaughter | 1.000 | 0.013 | 1.000 | 0.013 | 1.000 | 0.426 | 0.980 | 17.486 | 1.000 | 0.050 | 1.000 | 0.843 | 1.000 | 0.639 |
| Grandgrandfather | 1.000 | 0.897 | 1.000 | 0.897 | 1.000 | 0.404 | 0.947 | 55.728 | 0.947 | 0.059 | 0.927 | 0.902 | 1.000 | 0.746 |91.66
| Grandgrandmother | 1.000 | 4.173 | 1.000 | 4.173 | 1.000 | 0.442 | 0.893 | 50.329 | 0.947 | 0.060 | 0.927 | 0.908 | 1.000 | 0.817 |
| Grandgrandson | 1.000 | 1.632 | 1.000 | 1.632 | 1.000 | 0.452 | 0.931 | 60.358 | 0.911 | 0.070 | 0.911 | 1.050 | 1.000 | 0.939 |
| Uncle | 0.876 | 16.244 | 0.891 | 16.244 | 0.964 | 4.516 | 0.876 | 60.416 | 0.933 | 0.098 | 0.891 | 1.256 | 0.928 | 17.682 |
| LP | Test-F1-OCEL | RT-OCEL | Test-F1-CELOE | RT-CELOE | Test-F1-Evo | RT-Evo | Test-F1-DRILL | RT-DRILL | Test-F1-TDL | RT-TDL | Test-F1-NCES | RT-NCES | Test-F1-NCES2 | RT-NCES2 | Test-F1-ROCES | RT-ROCES | Test-F1-CLIP | RT-CLIP |
|:------------------:|-------------:|--------:|--------------:|---------:|------------:|-------:|--------------:|---------:|------------:|-------:|---------------:|----------:|----------------:|-----------:|----------------:|-----------:|-------------:|--------:|
| Aunt | 0.614 | 13.697 | 0.855 | 13.697 | 0.978 | 5.278 | 0.811 | 60.351 | 0.956 | 0.118 | 0.805 | 0.632 | 0.812 | 1.136 | 0.812 | 1.119 | 0.855 | 14.059 |
| Cousin | 0.712 | 10.846 | 0.789 | 10.846 | 0.993 | 3.311 | 0.701 | 60.485 | 0.820 | 0.176 | 0.608 | 0.628 | 0.680 | 1.177 | 0.695 | 1.086 | 0.779 | 9.050 |
| Grandgranddaughter | 1.000 | 0.013 | 1.000 | 0.013 | 1.000 | 0.426 | 0.980 | 17.486 | 1.000 | 0.050 | 1.000 | 0.507 | 1.000 | 0.955 | 1.000 | 0.917 | 1.000 | 0.639 |
| Grandgrandfather | 1.000 | 0.897 | 1.000 | 0.897 | 1.000 | 0.404 | 0.947 | 55.728 | 0.947 | 0.059 | 0.927 | 0.505 | 0.947 | 0.944 | 0.927 | 0.924 | 1.000 | 0.746 |
| Grandgrandmother | 1.000 | 4.173 | 1.000 | 4.173 | 1.000 | 0.442 | 0.893 | 50.329 | 0.947 | 0.060 | 0.947 | 0.633 | 0.933 | 1.323 | 0.947 | 1.306 | 1.000 | 0.817 |
| Grandgrandson | 1.000 | 1.632 | 1.000 | 1.632 | 1.000 | 0.452 | 0.931 | 60.358 | 0.911 | 0.070 | 0.909 | 0.598 | 0.931 | 1.171 | 0.931 | 1.146 | 1.000 | 0.939 |
| Uncle | 0.876 | 16.244 | 0.891 | 16.244 | 0.964 | 4.516 | 0.876 | 60.416 | 0.933 | 0.098 | 0.854 | 0.538 | 0.891 | 0.948 | 0.891 | 0.905 | 0.928 | 17.682 |


| LP | Train-F1-OCEL | Train-F1-CELOE | Train-F1-Evo | Train-F1-DRILL | Train-F1-TDL | Train-F1-NCES | Train-F1-CLIP |
|:------------------:|--------------:|---------------:|-------------:|---------------:|-------------:|----------------:|----------------:|
| Aunt | 0.835 | 0.918 | 0.995 | 0.837 | 1.000 | 0.804 | 0.918 |
| Cousin | 0.746 | 0.796 | 1.000 | 0.732 | 1.000 | 0.681 | 0.798 |
| Grandgranddaughter | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
| Grandgrandfather | 1.000 | 1.000 | 1.000 | 0.968 | 1.000 | 0.973 | 1.000 |
| Grandgrandmother | 1.000 | 1.000 | 1.000 | 0.975 | 1.000 | 0.939 | 1.000 |
| Grandgrandson | 1.000 | 1.000 | 1.000 | 0.962 | 1.000 | 0.927 | 1.000 |
| Uncle | 0.904 | 0.907 | 0.996 | 0.908 | 1.000 | 0.884 | 0.940 |
| LP | Train-F1-OCEL | Train-F1-CELOE | Train-F1-Evo | Train-F1-DRILL | Train-F1-TDL | Train-F1-NCES | Train-F1-NCES2 | Train-F1-ROCES | Train-F1-CLIP |
|:------------------:|--------------:|---------------:|-------------:|---------------:|-------------:|----------------:|-----------------:|-----------------:|----------------:|
| Aunt | 0.835 | 0.918 | 0.995 | 0.837 | 1.000 | 0.759 | 0.804 | 0.804 | 0.918 |
| Cousin | 0.746 | 0.796 | 1.000 | 0.732 | 1.000 | 0.680 | 0.696 | 0.728 | 0.798 |
| Grandgranddaughter | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 |
| Grandgrandfather | 1.000 | 1.000 | 1.000 | 0.968 | 1.000 | 0.910 | 0.944 | 0.942 | 1.000 |
| Grandgrandmother | 1.000 | 1.000 | 1.000 | 0.975 | 1.000 | 0.923 | 0.941 | 0.944 | 1.000 |
| Grandgrandson | 1.000 | 1.000 | 1.000 | 0.962 | 1.000 | 0.911 | 0.923 | 0.923 | 1.000 |
| Uncle | 0.904 | 0.907 | 0.996 | 0.908 | 1.000 | 0.823 | 0.886 | 0.884 | 0.940 |


### 10-Fold Cross Validation Mutagenesis Benchmark Results
```shell
python examples/concept_learning_cv_evaluation.py --kb ./KGs/Mutagenesis/mutagenesis.owl --lps ./LPs/Mutagenesis/lps.json --path_of_nces_embeddings ./NCESData/mutagenesis/embeddings/ConEx_entity_embeddings.csv --path_of_clip_embeddings ./CLIPData/mutagenesis/embeddings/ConEx_entity_embeddings.csv --max_runtime 60 --report mutagenesis_results.csv
```

| LP | Train-F1-OCEL | Test-F1-OCEL | RT-OCEL | Train-F1-CELOE | Test-F1-CELOE | RT-CELOE | Train-F1-Evo | Test-F1-Evo | RT-Evo | Train-F1-DRILL | Test-F1-DRILL | RT-DRILL | Train-F1-TDL | Test-F1-TDL | RT-TDL | Train-F1-NCES | Test-F1-NCES | RT-NCES | Train-F1-CLIP | Test-F1-CLIP | RT-CLIP |
|:---------|--------------:|-------------:|--------:|---------------:|--------------:|---------:|-------------:|------------:|-------:|---------------:|--------------:|---------:|-------------:|------------:|-------:|--------------:|-------------:|--------:|--------------:|-------------:|--------:|
| NotKnown | 0.916 | 0.918 | 60.705 | 0.916 | 0.918 | 60.705 | 0.975 | 0.970 | 51.870 | 0.809 | 0.804 | 60.140 | 1.000 | 0.852 | 13.569 | 0.717 | 0.718 | 3.784 | 0.916 | 0.918 | 26.312|
| LP | Train-F1-OCEL | Test-F1-OCEL | RT-OCEL | Train-F1-CELOE | Test-F1-CELOE | RT-CELOE | Train-F1-Evo | Test-F1-Evo | RT-Evo | Train-F1-DRILL | Test-F1-DRILL | RT-DRILL | Train-F1-TDL | Test-F1-TDL | RT-TDL | Train-F1-NCES | Test-F1-NCES | RT-NCES | Train-F1-NCES2 | Test-F1-NCES2 | RT-NCES2 | Train-F1-ROCES | Test-F1-ROCES | RT-ROCES | Train-F1-CLIP | Test-F1-CLIP | RT-CLIP |
|:---------|--------------:|-------------:|--------:|---------------:|--------------:|---------:|-------------:|------------:|-------:|---------------:|--------------:|---------:|-------------:|------------:|-------:|----------------:|---------------:|----------:|-----------------:|----------------:|-----------:|-----------------:|----------------:|-----------:|--------------:|-------------:|--------:|
| NotKnown | 0.916 | 0.918 | 60.705 | 0.916 | 0.918 | 60.705 | 0.975 | 0.970 | 51.870 | 0.809 | 0.804 | 60.140 | 1.000 | 0.852 | 13.569 | 0.704 | 0.704 | 2.605 | 0.704 | 0.704 | 1.841 | 0.704 | 0.704 | 1.711 | 0.916 | 0.918 | 26.312|



### 10-Fold Cross Validation Carcinogenesis Benchmark Results
```shell
python examples/concept_learning_cv_evaluation.py --kb ./KGs/Carcinogenesis/carcinogenesis.owl --lps ./LPs/Carcinogenesis/lps.json --path_of_nces_embeddings ./NCESData/carcinogenesis/embeddings/ConEx_entity_embeddings.csv --path_of_clip_embeddings ./CLIPData/carcinogenesis/embeddings/ConEx_entity_embeddings.csv --max_runtime 60 --report carcinogenesis_results.csv
```
| LP | Train-F1-OCEL | Test-F1-OCEL | RT-OCEL | Train-F1-CELOE | Test-F1-CELOE | RT-CELOE | Train-F1-Evo | Test-F1-Evo | RT-Evo | Train-F1-DRILL | Test-F1-DRILL | RT-DRILL | Train-F1-TDL | Test-F1-TDL | RT-TDL | Train-F1-NCES | Test-F1-NCES | RT-NCES | Train-F1-CLIP | Test-F1-CLIP | RT-CLIP |
|:---------|--------------:|-------------:|--------:|---------------:|--------------:|---------:|-------------:|------------:|-------:|---------------:|--------------:|---------:|-------------:|------------:|-------:|--------------:|-------------:|--------:|--------------:|-------------:|--------:|
| NOTKNOWN | 0.737 | 0.711 | 62.048 | 0.740 | 0.701 | 62.048 | 0.822 | 0.628 | 64.508 | 0.740 | 0.707 | 60.120 | 1.000 | 0.616 | 5.196 | 0.705 | 0.704 | 4.157 | 0.740 | 0.701 | 48.475|

| LP | Train-F1-OCEL | Test-F1-OCEL | RT-OCEL | Train-F1-CELOE | Test-F1-CELOE | RT-CELOE | Train-F1-Evo | Test-F1-Evo | RT-Evo | Train-F1-DRILL | Test-F1-DRILL | RT-DRILL | Train-F1-TDL | Test-F1-TDL | RT-TDL | Train-F1-NCES | Test-F1-NCES | RT-NCES | Train-F1-NCES2 | Test-F1-NCES2 | RT-NCES2 | Train-F1-ROCES | Test-F1-ROCES | RT-ROCES | Train-F1-CLIP | Test-F1-CLIP | RT-CLIP |
|:---------|--------------:|-------------:|--------:|---------------:|--------------:|---------:|-------------:|------------:|-------:|---------------:|--------------:|---------:|-------------:|------------:|-------:|----------------:|---------------:|----------:|-----------------:|----------------:|-----------:|-----------------:|----------------:|-----------:|--------------:|-------------:|--------:|
| NOTKNOWN | 0.737 | 0.711 | 62.048 | 0.740 | 0.701 | 62.048 | 0.822 | 0.628 | 64.508 | 0.740 | 0.707 | 60.120 | 1.000 | 0.616 | 5.196 | 0.709 | 0.709 | 2.718 | 0.705 | 0.704 | 1.912 | 0.705 | 0.704 | 1.774 | 0.740 | 0.701 | 48.475|


### Benchmark Results on DBpedia. Results are based on the training examples only
Expand Down Expand Up @@ -335,6 +337,22 @@ pytest -p no:warnings -x # Running 76 tests takes ~ 17 mins
Currently, we are working on our manuscript describing our framework.
If you find our work useful in your research, please consider citing the respective paper:
```
# ROCES
@inproceedings{kouagou2024roces,
title = {ROCES: Robust Class Expression Synthesis in Description Logics via Iterative Sampling},
author = {Kouagou, N'Dah Jean and Heindorf, Stefan and Demir, Caglar and Ngonga Ngomo, Axel-Cyrille},
booktitle = {Proceedings of the Thirty-Third International Joint Conference on
Artificial Intelligence, {IJCAI-24}},
publisher = {International Joint Conferences on Artificial Intelligence Organization},
editor = {Kate Larson},
pages = {4335--4343},
year = {2024},
month = {8},
note = {Main Track},
doi = {10.24963/ijcai.2024/479},
url = {https://doi.org/10.24963/ijcai.2024/479},
}

# DRILL
@inproceedings{demir2023drill,
author = {Demir, Caglar and Ngomo, Axel-Cyrille Ngonga},
Expand Down
Loading
Loading