(WIP) Adding DIHARD 2018 recipe.

[HuangZiliAndy]

The script is not finished and David would review it.

[david-ryan-snyder]

Be sure to credit yourself here.

[david-ryan-snyder]

You can remove hte file local/prepare_for_eer.py, it's not used in this recipe.

[david-ryan-snyder]

This is a site specific file and should be deleted.

[johnjosephmorgan]

I know this file is explained in the documentation and in queue.pl, but it would have helped me to have had an example file under conf. Could this be made generic and part of the recipe?

[david-ryan-snyder]

This file was probably added by accident. Please check for other unused files that shouldn't be in the PR.

[david-ryan-snyder]

Also, please create a README.txt in egs/dihard_2018 in another in egs/dihard_2018/v2.

The first one should explain the DIHARD data and provide a high level overview of what recipes there are (right now, just v2). You may want to link to https://coml.lscp.ens.fr/dihard/index.html in this README.txt.

In the second README.txt, you can explain the recipe in more detail (e.g., clustering embeddings, PLDA scoring, AHC, how we determine the AHC stopping threshold, etc). You'll may want to refer to the paper http://www.danielpovey.com/files/2018_interspeech_dihard.pdf which this recipe is based on. There were two tracks in the DIHARD 2018 competition, one uses oracle SAD (track1) and the other required that SAD was performed form scratch (track2). You might mention that this recipe demonstrates a solution for track1 of that eval.

[david-ryan-snyder]

Also adding @leibny to this, for her info.

[danpovey]

Sorry, I don't want to have copies of this everywhere because then people will end up modifying it to suit their cluster.

…

On Tue, Nov 6, 2018 at 5:57 PM John J Morgan ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In egs/dihard_2018/v2/conf/queue.conf <#2822 (comment)>: > @@ -0,0 +1,10 @@ +# Default configuration +command qsub -v PATH -cwd -S /bin/bash -j y -l arch=*64* +option mem=* -l mem_free=$0,ram_free=$0 +option mem=0 # Do not add anything to qsub_opts +option num_threads=* -pe smp $0 +option num_threads=1 # Do not add anything to qsub_opts +option max_jobs_run=* -tc $0 +default gpu=0 +option gpu=0 -l 'hostname=b1[234568]*|c*,gpu=0' -q g.q +option gpu=* -l 'hostname=b1[234568]*|c*,gpu=$0' -q g.q I know this file is explained in the documentation and in queue.pl, but it would have helped me to have had an example file under conf. Could this be made generic and part of the recipe? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2822 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADJVu9qIN0hm-_Qig8xnQwVu7_y3PVoWks5ushPkgaJpZM4YQ1Ll> .

[danpovey]

... I mean, and checking in the changes, causing incompatibilities.

…

On Tue, Nov 6, 2018 at 5:59 PM Daniel Povey ***@***.***> wrote: Sorry, I don't want to have copies of this everywhere because then people will end up modifying it to suit their cluster. On Tue, Nov 6, 2018 at 5:57 PM John J Morgan ***@***.***> wrote: > ***@***.**** commented on this pull request. > ------------------------------ > > In egs/dihard_2018/v2/conf/queue.conf > <#2822 (comment)>: > > > @@ -0,0 +1,10 @@ > +# Default configuration > +command qsub -v PATH -cwd -S /bin/bash -j y -l arch=*64* > +option mem=* -l mem_free=$0,ram_free=$0 > +option mem=0 # Do not add anything to qsub_opts > +option num_threads=* -pe smp $0 > +option num_threads=1 # Do not add anything to qsub_opts > +option max_jobs_run=* -tc $0 > +default gpu=0 > +option gpu=0 -l 'hostname=b1[234568]*|c*,gpu=0' -q g.q > +option gpu=* -l 'hostname=b1[234568]*|c*,gpu=$0' -q g.q > > I know this file is explained in the documentation and in queue.pl, but > it would have helped me to have had an example file under conf. Could this > be made generic and part of the recipe? > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <#2822 (comment)>, or mute > the thread > <https://github.com/notifications/unsubscribe-auth/ADJVu9qIN0hm-_Qig8xnQwVu7_y3PVoWks5ushPkgaJpZM4YQ1Ll> > . >

[HuangZiliAndy]

... I mean, and checking in the changes, causing incompatibilities.
…
On Tue, Nov 6, 2018 at 5:59 PM Daniel Povey @.> wrote: Sorry, I don't want to have copies of this everywhere because then people will end up modifying it to suit their cluster. On Tue, Nov 6, 2018 at 5:57 PM John J Morgan @.> wrote: > @.**** commented on this pull request. > ------------------------------ > > In egs/dihard_2018/v2/conf/queue.conf > <#2822 (comment)>: > > > @@ -0,0 +1,10 @@ > +# Default configuration > +command qsub -v PATH -cwd -S /bin/bash -j y -l arch=64 > +option mem=* -l mem_free=$0,ram_free=$0 > +option mem=0 # Do not add anything to qsub_opts > +option num_threads=* -pe smp $0 > +option num_threads=1 # Do not add anything to qsub_opts > +option max_jobs_run=* -tc $0 > +default gpu=0 > +option gpu=0 -l 'hostname=b1[234568]|c,gpu=0' -q g.q > +option gpu=* -l 'hostname=b1[234568]|c,gpu=$0' -q g.q > > I know this file is explained in the documentation and in queue.pl, but > it would have helped me to have had an example file under conf. Could this > be made generic and part of the recipe? > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <#2822 (comment)>, or mute > the thread > https://github.com/notifications/unsubscribe-auth/ADJVu9qIN0hm-_Qig8xnQwVu7_y3PVoWks5ushPkgaJpZM4YQ1Ll > . >

Hi, Dan! David has told me about this problem and this is because this is a very very draft version. I will pay attention to that and make sure there is no personal settings in the final version.

[david-ryan-snyder]

Could you rewrite this comment so it makes sense for this file? (e.g., remove anything about MUSAN)

[david-ryan-snyder]

This comment should be rewritten to remove references to MUSAN.

[david-ryan-snyder]

Could you write a short comment for this? I suggest clarifying that this script is only ever called by make_dihard_2018_eval.sh.

[david-ryan-snyder]

Could you add a brief comment, and explain the relationship between this script and the *.sh file?

[david-ryan-snyder]

Stage should be set to 0

[david-ryan-snyder]

Could you fix this comment, so that it says "training data" instead of "VoxCeleb2?" Since the training data includes portions of both VoxCeleb1 and VoxCeleb2.

[david-ryan-snyder]

Comment appears to be incomplete.

[david-ryan-snyder]

If you want to include these results, can you clarify in the comment that this is different from how the DIHARD challenge was originally evaluated?

[david-ryan-snyder]

Could you clarify that this is how the RTTMs were scored during the DIHARD challenge? This would be helpful info for anyone who wants to compare results.

[HuangZiliAndy]

Hi! I think I have fixed all the problems you mentioned. And in the current version, I add a channel parameter to the diarization/make_rttm.py (to ensure the code won't change for other scripts, I set 0 as default). To make the results comparable with the paper, I just keep the strict version of DER. (no unscored collars and including overlapping speech) The result is also offered in the script.

[HuangZiliAndy]

Hi David and Paola! The i-vector baseline of DIHARD is included. I was referring to Voxceleb baseline so I didn't add data augmentation for PLDA. One thing I am not sure is the local/extract_ivectors.sh file. I don't think it is very elegant. Please tell me if you find any problems. Thanks!

[david-ryan-snyder]

Thanks Zili! There's just a few more things and I think it should be ready to go:

• Instead of creating multiple copies of each data prep script for each recipe, v1, v2, etc, it would be better if there is a single set of scripts, and you create symbolic links. E.g., v2/local/make_dihard_2018_dev.sh can be a symbolic link to v1/local/make_dihard_2018_dev.sh. It's a bit cleaner this way.

• What are the differences between the script diarization/extract_ivectors.sh and the one in local/extract_ivectors.sh? Try to use the existing one in diarization/ if you can. If there's some functionality it lacks, there's a good chance you can add it to the existing script, rather than creating a new one.

• Could you document the --channel option that is passed around a little better? It would help if you made it clear that this only affects the format of the RTTM file. You could even call the option you pass into cluster.sh --rttm-channel. The usage message could describe what it does more clearly, e.g., "the value passed into the RTTM channel field. Only affects the format of the RTTM file." Without a little more clarity, users might think it does something with the wav file channels.

[HuangZiliAndy]

Hi, David! I agree with your first and third comment and I have fixed them. For the second one, the situation is as follows. For callhome v1, we didn't apply-cmvn to the features, so that's not a problem. For callhome v2, we apply-cmvn to the features before segmentation and dump them. For dihard v1, we are following callhome v2, we also add-deltas before apply-cmvn.

diarization/extract_ivectors.sh is very similar to local/extract_ivectors.sh. The only difference is that diarization/extract_ivectors.sh directly copy the delta options in the ivector extractor while local/extract_ivectors.sh passes the delta options. Because we do some feature preprocessing before, the delta options may not stay the same. I can replace diarization/extract_ivectors.sh with local/extract_ivectors.sh, but that will influence the callhome recipe (You have to pass a delta-order argument now) .

Should I pass the delta options? I want to know what do you think.

[david-ryan-snyder]

You don't need v1/local/nnet3, since you already have this directory in v2/local/nnet3. You should delete it.

[david-ryan-snyder]

@HuangZiliAndy,

It's usually better to add the functionality you want to an existing script, rather than creating a new script, especially if the new one differs by only one line! At the same time, we want to make sure that changes to old scripts don't break things in the existing recipes. Fortunately, I think we can satisfy both of those requirements.

I suggest adding an option called apply_deltas to diarization/extract_ivectors.sh. This should be initialized to true. Please add a comment explaining that when this option is true, we use the options provided in $srcdir/delta_opts. You might want to say that in diarization recipes, we sometimes need to write features to disk that already have various post-processing applied. As a result, deltas may have already been applied to the input features.

Have an if statement that checks if apply_deltas is true. If it is, delta_opts is taken from $srcdir/delta_opts, otherwise it is assigned the value of --delta-order=0.

if $apply_deltas; then
delta_opts=$(cat $srcdir/delta_opts 2>/dev/null)
else
delta_opts="--delta-order=0"
fi

[HuangZiliAndy]

Hi, David! Thanks for your advice. I have fixed these problems.

[david-ryan-snyder]

@HuangZiliAndy, Looks pretty good. Good work! There's a few more minor cosmetic things.

One last thing: please take a look at the python scripts you added for preparing the DIHARD data, and try to make sure they don't go over 80 columns.

After this, it should be ready to go.

[david-ryan-snyder]

reates -> creates

[david-ryan-snyder]

Should say Copyright 2018 Zili Huang

[david-ryan-snyder]

Should say Copyright 2018 Zili Huang

[david-ryan-snyder]

reats -> creates

[HuangZiliAndy]

Thanks David! I have fixed them.

[david-ryan-snyder]

@HuangZiliAndy, good work!

@danpovey, @leibny, I think this is ready to merge.