Skip to content

dedupe.documents.attribute_name does not work #166

Open
@mathCrazyy

Description

@mathCrazyy

#96 has already been mentioned

and my version is tag 1.0.3, my command is :

dolma dedupe --documents "study/samples/v0/documents//*" --dedupe.documents.attribute_name 'bff_duplicate_documents' --dedupe.documents.key "metadata.id" --dedupe.skip_empty --bloom_filter.file /tmp/deduper_bloom_filter.bin --no-bloom_filter.read_only --bloom_filter.estimated_doc_count '6_000_000' --bloom_filter.desired_false_positive_rate '0.0001' --processes 188

but I also get the wrong info :
thread '' panicked at src/deduper.rs:156:26:
called Result::unwrap() on an Err value: Custom { kind: Other, error: "Failed to parse rule: --> 1:1\n |\n1 | metadata.id\n | ^---\n |\n = expected chain" }
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions