Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fixed bug in detection of pathological cases in devPAV #92

Merged
merged 115 commits into from
Nov 8, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
115 commits
Select commit Hold shift + click to select a range
0f2d543
dummy file to initialize TLM branch
vergep Sep 14, 2023
6ea6a44
nieuwe test
vergep Sep 14, 2023
bd09fdd
werkt dit?
vergep Sep 14, 2023
37c7416
test geschreven voor MSwithin
vergep Sep 14, 2023
9deb3ed
1. Creation of run.py, which contains main script used for debugging.
vergep Sep 15, 2023
5725935
removed some unused code in test_TLM.py
vergep Sep 18, 2023
a2ea89c
added code for calculation and test of means_z
vergep Sep 18, 2023
f956225
changed names of columns of covars df to more generic names
vergep Sep 18, 2023
6a29a8c
removed 'z' from calc_means_z
vergep Sep 18, 2023
1f826aa
code and test for kappa
vergep Sep 18, 2023
b17b482
added some comments
vergep Sep 18, 2023
9cd21bd
work in progress to write function for T0
vergep Sep 18, 2023
33cabd9
script and test for h^2 calculation
vergep Sep 19, 2023
7064b1b
added some comments
vergep Sep 19, 2023
8d3a11f
script and test for T0
vergep Sep 19, 2023
21e0498
Merge branch 'h' into T0
vergep Sep 19, 2023
4168c5f
input data en R-output data added to git
vergep Sep 19, 2023
aa0401e
Set up of calculations of U0 and UU for the predict part.
camillevandijk Sep 19, 2023
e8ebed5
calculation of U_h0 and U_hx
vergep Sep 19, 2023
6810345
added intermediate 'predict' results from R
vergep Sep 19, 2023
6052cdf
Added tests for U_h0 and U_hx.
camillevandijk Sep 20, 2023
a8dbec4
Add calculations and test for U_hn. Plus transformed T0 to numpy inst…
camillevandijk Sep 20, 2023
ef05093
Update comments
camillevandijk Sep 20, 2023
a2cb3be
create branch
Sep 21, 2023
77f9e4c
add basic properties
Sep 21, 2023
d650016
remove sklearn inheritance and replace predict_proba by transform
Sep 21, 2023
d981210
fix test
Sep 21, 2023
66f3918
changed some comment
vergep Sep 25, 2023
e3d9f94
added a model description
vergep Sep 25, 2023
7bdd07d
Merge branch 'predict_U' of https://github.com/NetherlandsForensicIns…
vergep Sep 25, 2023
b3a60f1
added calculation of mu_h
vergep Sep 25, 2023
2fafb14
added calculation of mu_h and test
vergep Sep 25, 2023
bc3c3f4
added calculation and test for natural log of numerator of LR in Bolc…
vergep Sep 25, 2023
8ae5206
added calculation and test for natural log of left denominator term o…
vergep Sep 25, 2023
7c88e96
added calculation and test for natural log of right denominator term …
vergep Sep 25, 2023
a838c8b
caculated and tested first log10_LR comparison, and removed redundant…
vergep Sep 25, 2023
44c76b9
made predict LLR function and test for all pairs
vergep Sep 25, 2023
1b5258c
updated some comments and run.py
vergep Sep 25, 2023
ce5b0ab
updated some comments
vergep Sep 25, 2023
95f70e2
changed some divisions to additions through log
vergep Sep 25, 2023
6bffdbe
changed some divisions to additions through log in TLM_calc_log10_LR.…
vergep Sep 25, 2023
7a8bece
minor change in comment
vergep Sep 25, 2023
6eee46a
revised some lines in TLM_calc_ln_den_term
vergep Sep 25, 2023
76ac4d5
added some more comments in TLM_calc_log_10_LR
vergep Sep 25, 2023
dd04940
added line break
vergep Sep 25, 2023
b74bafd
added predict_proba
vergep Sep 26, 2023
caeb251
Merge pull request #84 from NetherlandsForensicInstitute/feature/tlm_…
vergep Sep 26, 2023
e617afb
refined some comments
vergep Sep 26, 2023
5316a0f
refined some comments
vergep Sep 26, 2023
53c4a0f
added testdata in input folder
vergep Sep 26, 2023
992f88e
Resolved some comments, added comments, made it more generic and chan…
camillevandijk Sep 26, 2023
d3b5fd0
added testdata in Routput folder
vergep Sep 26, 2023
84c6102
Merge branch 'TLM' into MSwithin
camillevandijk Sep 26, 2023
14dbc2c
Incorporated the mean_covariance_within function and the test in the …
camillevandijk Sep 26, 2023
96fa48c
Merged TLM branch into means_z branch.
camillevandijk Sep 26, 2023
c3f1c27
Added the means function to the tlm class and updated the test accord…
camillevandijk Sep 26, 2023
6f6517d
Removed redundant files
camillevandijk Sep 26, 2023
1aca264
Merge pull request #80 from NetherlandsForensicInstitute/means_z
camillevandijk Sep 26, 2023
9bbae92
merged main TLM branch into this predict_log10_LR branch.
camillevandijk Sep 26, 2023
191230f
Included calculations for kernel bandwidth in the tlm class and test.
camillevandijk Sep 26, 2023
7335ff3
Added the H0 (between covariance) calculations to the tlm class and t…
camillevandijk Sep 26, 2023
aac38d6
added between_cov to two_level_model and test
vergep Sep 26, 2023
6a3240e
changed names of data folder to 'resources' and changed names of inpu…
vergep Sep 26, 2023
329cb96
changed names of TLM folder to 'two_level_model'
vergep Sep 26, 2023
2b37823
changed names of TLM folder to 'two_level_model'
vergep Sep 26, 2023
8daed0b
added class Test_TwoLevelModel_predict_functions and performed first …
vergep Sep 26, 2023
57ee3bd
Renamed the U variables and added some clearer explanation in the doc…
camillevandijk Sep 28, 2023
464a98e
added code and tests for updated_ref_mean (mu_h)
vergep Sep 28, 2023
7b1aacc
Made old code working again for debugging new code.
vergep Sep 28, 2023
7bc2f50
Added script for ln_den_terms to class and test
vergep Sep 29, 2023
7169922
added tests for fit&_predict_LLR, fit&transform and fit&predict_proba
vergep Sep 29, 2023
f64d508
added check that n_features is the same in train, trace and ref
vergep Sep 29, 2023
8fd4837
changed TwoLevelModel to TwoLevelModelNormalKDE and fixed minor bug i…
vergep Sep 29, 2023
f2df21d
clean tests
Oct 5, 2023
c423bfe
predict-lr takes multiple samples
Oct 5, 2023
8cd06bc
Merge pull request #91 from NetherlandsForensicInstitute/predict_log1…
vergep Oct 9, 2023
65bd6e4
Removed ValueError for difference in shapes of references and traces.
camillevandijk Oct 9, 2023
fcafeb7
fixed bug in detection of pathological cases in devPAV
vergep Oct 15, 2023
e42990e
Merge branch 'remove_devPAV_bug' into TLM
vergep Oct 15, 2023
4423990
Merge pull request #93 from NetherlandsForensicInstitute/remove_devPA…
vergep Oct 15, 2023
80bff2e
Merge branch 'TLM' of https://github.com/NetherlandsForensicInstitute…
vergep Oct 16, 2023
dca5bd7
added some more comments
vergep Oct 16, 2023
71926bf
fixed bug in predict_proba
vergep Oct 17, 2023
0c71c5f
some cosmetics
vergep Oct 17, 2023
203317f
bugfix: in predict_proba changed return array.reshape to np.transpose…
vergep Oct 19, 2023
a14ad9a
removed redundant 'dummmy' file and added pandas to setup.py
vergep Oct 19, 2023
210b879
changed datadirectory from data to recources
vergep Oct 19, 2023
4ae11b2
corrected path definitions
vergep Oct 19, 2023
1b659b6
static methods
wowtor Oct 25, 2023
250126e
corrected surface calculation when PAV spline is parallel to line y=x…
vergep Oct 31, 2023
92139cb
corrected surface calculation when PAV spline is parallel to line y=x…
vergep Oct 31, 2023
1ee6249
corrected surface calculation when PAV spline is parallel to line y=x…
vergep Oct 31, 2023
11aa45c
Merge remote-tracking branch 'origin/TLM' into TLM
vergep Oct 31, 2023
6c109a7
corrected surface calculation when PAV spline is parallel to line y=x…
vergep Oct 31, 2023
bfaeb8d
corrected surface calculation when PAV spline is parallel to line y=x…
vergep Oct 31, 2023
f46e21a
corrected surface calculation when PAV spline is parallel to line y=x…
vergep Oct 31, 2023
b0df56c
corrected tests and removed tests for devpav_estimated and replaced b…
vergep Nov 2, 2023
f8b14c5
some cosmetics and included error message when data under one of two …
vergep Nov 2, 2023
836b667
some cosmetics and included error message when data under one of two …
vergep Nov 2, 2023
dfe3e2c
removed obsolete return brackets
vergep Nov 2, 2023
48e90a4
removed obsolete brackets
vergep Nov 2, 2023
236c77c
made tests for _calcsurface working again
vergep Nov 2, 2023
bceee10
plot and test that shows that devpav_estimated gives a different resu…
vergep Nov 3, 2023
36bc837
removed devpav_estimated
vergep Nov 3, 2023
57ffefb
Merge branch 'TLM' into remove_devPAV_bug
vergep Nov 3, 2023
88012dc
some cosmetics
vergep Nov 3, 2023
d8c584b
Merge remote-tracking branch 'origin/remove_devPAV_bug' into remove_d…
vergep Nov 3, 2023
99122bb
changed is.nan test
vergep Nov 3, 2023
fe81d4d
changed test accuracy of test_fit_and_transform to 9 decimals instead…
vergep Nov 3, 2023
8b4bd07
deleted all files related to TLM
vergep Nov 7, 2023
ded127e
replace nose by pytest
wowtor Nov 8, 2023
226e95f
Merge branch 'tests_import' into remove_devPAV_bug
wowtor Nov 8, 2023
c871d40
removed nose dependency
wowtor Nov 8, 2023
ca047cd
Apply suggestions from code review
wowtor Nov 8, 2023
4a56c50
removed TLM code
wowtor Nov 8, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
calculation of U_h0 and U_hx
  • Loading branch information
vergep committed Sep 19, 2023
commit e8ebed5aced9f86dc97bbc0a8ba05b8de4d95be1
16 changes: 8 additions & 8 deletions lir/classifiers.py
Original file line number Diff line number Diff line change
Expand Up @@ -90,15 +90,15 @@ def TLM_calc_T0(X, y):


'''Functions for predict'''
def TLM_calc_U(X, y, MSwithin, means, h_sq, T0):
def TLM_calc_U(X_tr, X_ref, MSwithin, h_sq, T0):
"""
X np.array of measurements of paired objects, rows are objects, columns are variables
y np 1d-array of labels. labels {0, 1} meaning same source or different source
returns: U_h0 and U_hx ...
X_tr np.array of measurements of trace object, rows are repetitions, columns are variables
X_ref no.array of measurments of reference object, rows are repetitions, columns are variables
returns: U_h0 and U_hx, covariance matrices needed for LR calculation, one for trace, one for ref
"""
n_traces = 5
n_references = 30
U_h0 = h_sq * T0 + MSwithin/n_traces
U_hx = h_sq * T0 + MSwithin/n_references
n_trace = len(X_tr)
n_reference = len(X_ref)
U_h0 = h_sq * T0 + MSwithin/n_trace
U_hx = h_sq * T0 + MSwithin/n_reference

return U_h0, U_hx
4 changes: 3 additions & 1 deletion tests/run.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,5 +12,7 @@
means_z = TLM_calc_means(dataZ[:,1:], dataZ[:,0])
h_sq = TLM_calc_h_sq(dataZ[:,1:], dataZ[:,0])
T0 = TLM_calc_T0(dataZ[:,1:], dataZ[:,0])
U_h0 = TLM_calc_U(dataZ[:,1:], dataZ[:,0], MSwithin, means_z, h_sq, T0)
dataX = np.loadtxt(os.path.join(dirname, 'data/TLM/input/XDATA.csv'), delimiter=",", dtype="float", skiprows=1, usecols=range(1,11))
dataY = np.loadtxt(os.path.join(dirname, 'data/TLM/input/YDATA.csv'), delimiter=",", dtype="float", skiprows=1, usecols=range(1,12))
U_h0, U_hn = TLM_calc_U(dataY[[0, 1, 2], 1:], dataX, MSwithin, h_sq, T0)
print("t")