distinct() doesn't work on latest rust version (1.82.0)

Hi, I recently updated my rust version from version 1.75 to the stable 1.82.0.
When I tried to run distinct function on the current stable version of rust (1.82.0) we encountered an error on the following code (If we removed the distinct() the code doesn't get panicked:

fn main(){
    
        let _ = timely::execute_from_args(std::env::args(), move |worker| {
        
            let mut input_manages: InputSession::<usize, (i32, i32), isize> = InputSession::new();
        
            worker.dataflow(|scope| {
                let manages = input_manages.to_collection(scope);
                manages   // transitive contains (manager, person) for many hops.
                    .iterate(|transitive| {
                        let manages = manages.enter(&transitive.scope());
                        transitive
                            .map(|(mk, m1)| (m1, mk))
                            .join(&manages)
                            .map(|(m1, (mk, p))| (mk, p))
                            .concat(&manages)
                            .distinct()
                        });
                manages.inspect(|x| println!("{:?}", x));
            });
            
            
        input_manages.insert((0, 1));
        input_manages.insert((0, 2));
        input_manages.insert((1, 3));
        input_manages.insert((2, 3));
        input_manages.insert((3, 4));
                        
        });
}

The error was as follows:

thread 'timely:work-0' panicked at core/src/panicking.rs:221:5:
unsafe precondition(s) violated: slice::get_unchecked requires that the index is within the slice
stack backtrace:
   0: rust_begin_unwind
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/panicking.rs:662:5
   1: core::panicking::panic_nounwind_fmt::runtime
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/panicking.rs:112:18
   2: core::panicking::panic_nounwind_fmt
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/panicking.rs:122:5
   3: core::panicking::panic_nounwind
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/panicking.rs:221:5
   4: <usize as core::slice::index::SliceIndex<[T]>>::get_unchecked::precondition_check
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/ub_checks.rs:70:21
   5: <usize as core::slice::index::SliceIndex<[T]>>::get_unchecked
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/ub_checks.rs:77:17
   6: core::slice::<impl [T]>::get_unchecked
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/slice/mod.rs:665:20
   7: differential_dataflow::trace::implementations::merge_batcher::VecQueue<T>::peek
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/differential-dataflow-0.12.0/src/trace/implementations/merge_batcher.rs:132:18
   8: differential_dataflow::trace::implementations::merge_batcher::MergeSorter<D,T,R>::merge_by
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/differential-dataflow-0.12.0/src/trace/implementations/merge_batcher.rs:268:29
   9: differential_dataflow::trace::implementations::merge_batcher::MergeSorter<D,T,R>::push
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/differential-dataflow-0.12.0/src/trace/implementations/merge_batcher.rs:214:30
  10: <differential_dataflow::trace::implementations::merge_batcher::MergeBatcher<K,V,T,R,B> as differential_dataflow::trace::Batcher<K,V,T,R,B>>::push_batch
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/differential-dataflow-0.12.0/src/trace/implementations/merge_batcher.rs:37:9
  11: <differential_dataflow::trace::rc_blanket_impls::RcBatcher<K,V,T,R,B> as differential_dataflow::trace::Batcher<K,V,T,R,alloc::rc::Rc<B>>>::push_batch
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/differential-dataflow-0.12.0/src/trace/mod.rs:402:69
  12: <differential_dataflow::collection::Collection<G,(K,V),R> as differential_dataflow::operators::arrange::arrangement::Arrange<G,K,V,R>>::arrange_core::{{closure}}::{{closure}}::{{closure}}
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/differential-dataflow-0.12.0/src/operators/arrange/arrangement.rs:583:25
  13: timely::dataflow::operators::generic::handles::InputHandle<T,D,P>::for_each
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/timely-0.12.0/src/dataflow/operators/generic/handles.rs:92:13
  14: timely::dataflow::operators::generic::handles::FrontieredInputHandle<T,D,P>::for_each
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/timely-0.12.0/src/dataflow/operators/generic/handles.rs:136:9
  15: <differential_dataflow::collection::Collection<G,(K,V),R> as differential_dataflow::operators::arrange::arrangement::Arrange<G,K,V,R>>::arrange_core::{{closure}}::{{closure}}
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/differential-dataflow-0.12.0/src/operators/arrange/arrangement.rs:580:21
  16: <timely::dataflow::stream::Stream<G,D1> as timely::dataflow::operators::generic::operator::Operator<G,D1>>::unary_frontier::{{closure}}::{{closure}}
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/timely-0.12.0/src/dataflow/operators/generic/operator.rs:352:17
  17: timely::dataflow::operators::generic::builder_rc::OperatorBuilder<G>::build::{{closure}}::{{closure}}
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/timely-0.12.0/src/dataflow/operators/generic/builder_rc.rs:123:31
  18: timely::dataflow::operators::generic::builder_rc::OperatorBuilder<G>::build_reschedule::{{closure}}
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/timely-0.12.0/src/dataflow/operators/generic/builder_rc.rs:163:26
  19: <timely::dataflow::operators::generic::builder_raw::OperatorCore<T,L> as timely::scheduling::Schedule>::schedule
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/timely-0.12.0/src/dataflow/operators/generic/builder_raw.rs:203:9
  20: timely::progress::subgraph::PerOperatorState<T>::schedule
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/timely-0.12.0/src/progress/subgraph.rs:656:30
  21: timely::progress::subgraph::Subgraph<TOuter,TInner>::activate_child
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/timely-0.12.0/src/progress/subgraph.rs:338:26
  22: <timely::progress::subgraph::Subgraph<TOuter,TInner> as timely::scheduling::Schedule>::schedule
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/timely-0.12.0/src/progress/subgraph.rs:304:17
  23: timely::progress::subgraph::PerOperatorState<T>::schedule
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/timely-0.12.0/src/progress/subgraph.rs:656:30
  24: timely::progress::subgraph::Subgraph<TOuter,TInner>::activate_child
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/timely-0.12.0/src/progress/subgraph.rs:338:26
  25: <timely::progress::subgraph::Subgraph<TOuter,TInner> as timely::scheduling::Schedule>::schedule
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/timely-0.12.0/src/progress/subgraph.rs:304:17
  26: timely::worker::Wrapper::step::{{closure}}
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/timely-0.12.0/src/worker.rs:727:57
  27: core::option::Option<T>::map
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/option.rs:1105:29
  28: timely::worker::Wrapper::step
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/timely-0.12.0/src/worker.rs:727:26
  29: timely::worker::Worker<A>::step_or_park
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/timely-0.12.0/src/worker.rs:375:38
  30: timely::execute::execute::{{closure}}
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/timely-0.12.0/src/execute.rs:285:15
  31: timely_communication::initialize::initialize_from::{{closure}}
             at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/timely_communication-0.12.0/src/initialize.rs:285:33
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
thread caused non-unwinding panic. aborting.
Aborted

Heyo! You likely need to update to the most recent timely/differential (from the repo, rather than crates.io). At least, something happened in/around 1.76 where some unsafe code is now actively unsound. If you are using timely/differential 0.12 from crates.io, that's likely the issue! If you are using the repo already .. ah .. we'll have to look closer.

Yup, works great on the current master for me (I hope this is the right output):

     Running `target/debug/examples/test`
((0, 1), 0, 1)
((0, 2), 0, 1)
((1, 3), 0, 1)
((2, 3), 0, 1)
((3, 4), 0, 1)

Based on the example, I'm guessing you are working through the mdbook? The right thing for the moment is to do what the linked PR does, which is to swap out the Cargo.toml dependencies from the 0.12 crates.io versions, which are several years old at this point, and point at the repos themselves. It's not clear what changed with 1.76, but afaict all unsafe { is potentially unsound w/o a language spec, so we removed all unsafe {. Needs to get to a stable enough state to push at crates.io, though.

(( feel free to re-open if it turns out the above doesn't fix, but it's something we've seen before! ))

Thanks for the quick reply! Yes, we did use the mdbook from https://timelydataflow.github.io/differential-dataflow/

Changing the cargo.toml file as you suggested did fixed the issue :)

Thanks again!

Navigation Menu

Repository navigation

`distinct()` doesn't work on latest rust version (1.82.0) #528

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

distinct() doesn't work on latest rust version (1.82.0) #528

Description

Activity

frankmcsherry commented on Oct 26, 2024

frankmcsherry commented on Oct 26, 2024

frankmcsherry commented on Oct 26, 2024

frankmcsherry commented on Oct 26, 2024

IdoDesign commented on Oct 27, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`distinct()` doesn't work on latest rust version (1.82.0) #528