t5-small To TOSA

[AmosLewis]

https://huggingface.co/t5-small

This patch uses transformers==4.26.0. The 4.19.2 will lead to a make_fx bug with the multiple inputs.
t5_small.py
t5small_tosa_0420_transformers4.26.0.mlir
t5small_tosa_0420_transformers4.26.0_elide.mlir

t5_small_torchscript.mlir

Ops has been fixed

• select and fill: MERGED [MLIR] Fold aten select and fill_ pattern llvm/torch-mlir#2000
• AtenZerosOp: MERGED [TOSA] Add support for AtenZerosOp 0 layout llvm/torch-mlir#1983
• slice end < 0 MERGED [MLIR][TOSA] Add support for sliceOp end < 0 llvm/torch-mlir#2011
• todtype f32->i64 MERGED [TOSA] Add todtype f32 to i64 llvm/torch-mlir#2012
• torch.aten.abs MERGED [TOSA] Add aten.abs support llvm/torch-mlir#2032
• torch.aten.le.Tensor MERGED [TOSA] Add aten.le.tensor support llvm/torch-mlir#2033

Ops that good to work but not ready to be merged

• slice and copy: Ready For Using: [MLIR] fold slice and copy int64_max support llvm/torch-mlir#1970
• torch.aten._index_put_impl Ready For Using [TOSA] Add aten._index_put_impl support llvm/torch-mlir#2031
  But not ready to merge because tosa.scatter to linalg is not exist.

T5-large To TOSA: #987
T5 To Stablehlo#1336

[AmosLewis]

torch_mlir.compiler_utils.TorchMlirCompilerError: Lowering TorchScript IR -> Torch Backend IR failed with the following diagnostics:
error: unsupported by backend contract: tensor with unknown rank
note: see current operation: %161 = "torch.tensor_static_info_cast"(%160) : (!torch.vtensor<[1,4],si64>) -> !torch.vtensor<*,si64>
note: this is likely due to a missing transfer function in abstract_interp_lib_gen.py

For Torch-MLIR developers, the error can be reproduced with:
$ torch-mlir-opt -pass-pipeline='builtin.module(torchscript-module-to-torch-backend-pipeline{backend-legal-ops=torch.aten.flatten.using_ints,torch.aten.native_layer_norm,torch.aten.linear})' /tmp/_lambda.mlir
Add '-mlir-print-ir-after-all -mlir-disable-threading' to get the IR dump for debugging purpose.

Related discord discussion:

1. slice+copy_: https://discord.com/channels/636084430946959380/1074784278078038016
2. During the AdjustCallingConvention pass: https://discord.com/channels/636084430946959380/1080220956259270717

Detail narrow Ir debug print:
https://gist.github.com/AmosLewis/20895fa06e8b61ce83e9a2b9c2ee8ca6
https://gist.github.com/AmosLewis/f95225b25c33a0bf1662d1cd105d5009

Oh, I got where is the issue in my patch. There is a slice(arg0) + copy_ pattern in my case. Copy_ will copy a new value into slice(arg0), which will overwrite the original arg0. After this pattern, the select.int(arg0) use the mutated version of arg0 as input, which will lead to a bug. I think we definitely need the MaximizeValueSemantics pass to handle this pattern.

1st potential solution from @gpetters94. Fold slice and copy into index_put_ op. llvm/torch-mlir#1901

2nd potential solution from Groverkss: In case there are multiple slices of a tensor, and a single slice is overwritten, the analysis will become aware that if some slice is independent of this slice, it is not overwritten. llvm/torch-mlir#1909

Here is some doc about the value semantics llvm/torch-mlir@370e327

copy_ is done inplace, will change the original tensor.

torch.tensor_static_info_cast :

Adds/removes static information from a tensor type. it statically annotates (or erases) shape and dtype information from a tensor type. For converting between the value-semantic and non-value-semantic domains,
use torch.copy.tensor. The two ops are kept separate to prevent canonicalizations from accidentally dropping static information.

copy.tensor:

Makes a copy of a tensor. Changes to the original tensor will not be reflected in the copy. This op can be used to interconvert between value-semantic and non-value-semantic tensors. However, this op does not allow adding/removing static information about sizes/dtype. For that, use torch.tensor_static_info_cast.

[AmosLewis]

Fold slice and copy into index_put_ llvm/torch-mlir#1901
index_put_ can be turned into value semantics because it has as an input the entire tensor being overwritten as well as information on what slice of the tensor to overwrite. In the slice+copy case, the full tensor is in one place of the IR, while the slice information and overwrite part is somewhere else in the IR.

George think the following bug is because of the negative support for slice indices.
llvm/torch-mlir#1917

➜  /tmp torch-mlir-opt -pass-pipeline='builtin.module(torchscript-module-to-torch-backend-pipeline{backend-legal-ops=torch.aten.flatten.using_ints,torch.aten.native_layer_norm,torch.aten.linear})' /tmp/t5small_torchscript_0227_transformers4.26.0.mlir 

LLVM ERROR: SmallVector unable to grow. Requested capacity (4294967296) is larger than maximum value for size type (4294967295)
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.	Program arguments: /home/chi/src/ubuntu20/shark/torch-mlir/build/bin/torch-mlir-opt -pass-pipeline=builtin.module(torchscript-module-to-torch-backend-pipeline{backend-legal-ops=torch.aten.flatten.using_ints,torch.aten.native_layer_norm,torch.aten.linear}) /tmp/t5small_torchscript_0227_transformers4.26.0.mlir
 #0 0x000055ebe8cc8e2d llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/llvm/lib/Support/Unix/Signals.inc:567:11
 #1 0x000055ebe8cc92bb PrintStackTraceSignalHandler(void*) /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/llvm/lib/Support/Unix/Signals.inc:641:1
 #2 0x000055ebe8cc7666 llvm::sys::RunSignalHandlers() /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/llvm/lib/Support/Signals.cpp:104:5
 #3 0x000055ebe8cc99a5 SignalHandler(int) /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/llvm/lib/Support/Unix/Signals.inc:412:1
 #4 0x00007f397e242520 (/lib/x86_64-linux-gnu/libc.so.6+0x42520)
 #5 0x00007f397e296a7c __pthread_kill_implementation ./nptl/pthread_kill.c:44:76
 #6 0x00007f397e296a7c __pthread_kill_internal ./nptl/pthread_kill.c:78:10
 #7 0x00007f397e296a7c pthread_kill ./nptl/pthread_kill.c:89:10
 #8 0x00007f397e242476 gsignal ./signal/../sysdeps/posix/raise.c:27:6
 #9 0x00007f397e2287f3 abort ./stdlib/abort.c:81:7
#10 0x000055ebe8bc1f14 llvm::report_fatal_error(llvm::Twine const&, bool) /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/llvm/lib/Support/ErrorHandling.cpp:125:5
#11 0x000055ebe8c30fc3 /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/llvm/lib/Support/SmallVector.cpp:70:3
#12 0x000055ebe8c30e1a unsigned long getNewCapacity<unsigned int>(unsigned long, unsigned long, unsigned long) /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/llvm/lib/Support/SmallVector.cpp:102:7
#13 0x000055ebe8c311f5 llvm::SmallVectorBase<unsigned int>::grow_pod(void*, unsigned long, unsigned long) /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/llvm/lib/Support/SmallVector.cpp:140:24
#14 0x000055ebe3e5ac95 llvm::SmallVectorTemplateCommon<long, void>::grow_pod(unsigned long, unsigned long) /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/llvm/include/llvm/ADT/SmallVector.h:142:3
#15 0x000055ebe3e5ac52 llvm::SmallVectorTemplateBase<long, true>::grow(unsigned long) /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/llvm/include/llvm/ADT/SmallVector.h:529:71
#16 0x000055ebe3e5abec long const* llvm::SmallVectorTemplateCommon<long, void>::reserveForParamAndGetAddressImpl<llvm::SmallVectorTemplateBase<long, true>>(llvm::SmallVectorTemplateBase<long, true>*, long const&, unsigned long) /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/llvm/include/llvm/ADT/SmallVector.h:247:12
#17 0x000055ebe3e5ab75 llvm::SmallVectorTemplateBase<long, true>::reserveForParamAndGetAddress(long&, unsigned long) /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/llvm/include/llvm/ADT/SmallVector.h:540:5
#18 0x000055ebe3e469a6 llvm::SmallVectorTemplateBase<long, true>::push_back(long) /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/llvm/include/llvm/ADT/SmallVector.h:566:23
#19 0x000055ebe3e38123 mlir::torch::Torch::AtenSliceTensorOp::getCanonicalizationPatterns(mlir::RewritePatternSet&, mlir::MLIRContext*)::$_44::operator()(mlir::torch::Torch::AtenSliceTensorOp, mlir::PatternRewriter&) const /home/chi/src/ubuntu20/shark/torch-mlir/lib/Dialect/Torch/IR/TorchOps.cpp:2189:38
#20 0x000055ebe3e37d0d mlir::torch::Torch::AtenSliceTensorOp::getCanonicalizationPatterns(mlir::RewritePatternSet&, mlir::MLIRContext*)::$_44::__invoke(mlir::torch::Torch::AtenSliceTensorOp, mlir::PatternRewriter&) /home/chi/src/ubuntu20/shark/torch-mlir/lib/Dialect/Torch/IR/TorchOps.cpp:2160:17
#21 0x000055ebe3e7620e mlir::RewritePatternSet& mlir::RewritePatternSet::add<mlir::torch::Torch::AtenSliceTensorOp>(mlir::LogicalResult (*)(mlir::torch::Torch::AtenSliceTensorOp, mlir::PatternRewriter&), mlir::PatternBenefit, llvm::ArrayRef<llvm::StringRef>)::FnPattern::matchAndRewrite(mlir::torch::Torch::AtenSliceTensorOp, mlir::PatternRewriter&) const /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/llvm/../mlir/include/mlir/IR/PatternMatch.h:1664:16
#22 0x000055ebe3e7616b mlir::detail::OpOrInterfaceRewritePatternBase<mlir::torch::Torch::AtenSliceTensorOp>::matchAndRewrite(mlir::Operation*, mlir::PatternRewriter&) const /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/llvm/../mlir/include/mlir/IR/PatternMatch.h:330:12
#23 0x000055ebe88a7552 mlir::PatternApplicator::matchAndRewrite(mlir::Operation*, mlir::PatternRewriter&, llvm::function_ref<bool (mlir::Pattern const&)>, llvm::function_ref<void (mlir::Pattern const&)>, llvm::function_ref<mlir::LogicalResult (mlir::Pattern const&)>) /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/mlir/lib/Rewrite/PatternApplicator.cpp:200:25
#24 0x000055ebe884c211 (anonymous namespace)::GreedyPatternRewriteDriver::processWorklist() /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/mlir/lib/Transforms/Utils/GreedyPatternRewriteDriver.cpp:247:19
#25 0x000055ebe884a38d (anonymous namespace)::RegionPatternRewriteDriver::simplify() && /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/mlir/lib/Transforms/Utils/GreedyPatternRewriteDriver.cpp:461:15
#26 0x000055ebe884a0af mlir::applyPatternsAndFoldGreedily(mlir::Region&, mlir::FrozenRewritePatternSet const&, mlir::GreedyRewriteConfig) /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/mlir/lib/Transforms/Utils/GreedyPatternRewriteDriver.cpp:490:47
#27 0x000055ebe41663b3 mlir::applyPatternsAndFoldGreedily(mlir::Operation*, mlir::FrozenRewritePatternSet const&, mlir::GreedyRewriteConfig) /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/llvm/../mlir/include/mlir/Transforms/GreedyPatternRewriteDriver.h:112:15
#28 0x000055ebe87b43b5 (anonymous namespace)::Canonicalizer::runOnOperation() /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/mlir/lib/Transforms/Canonicalizer.cpp:61:9
#29 0x000055ebe88b210a mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int) /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/mlir/lib/Pass/Pass.cpp:471:21
#30 0x000055ebe88b26f4 mlir::detail::OpToOpPassAdaptor::runPipeline(mlir::OpPassManager&, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int, mlir::PassInstrumentor*, mlir::PassInstrumentation::PipelineParentInfo const*) /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/mlir/lib/Pass/Pass.cpp:534:16
#31 0x000055ebe88b68d1 mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int)::$_6::operator()(mlir::OpPassManager&, mlir::Operation*) const /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/mlir/lib/Pass/Pass.cpp:456:12
#32 0x000055ebe88b6645 mlir::LogicalResult llvm::function_ref<mlir::LogicalResult (mlir::OpPassManager&, mlir::Operation*)>::callback_fn<mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int)::$_6>(long, mlir::OpPassManager&, mlir::Operation*) /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:45:12
#33 0x000055ebe41ed609 llvm::function_ref<mlir::LogicalResult (mlir::OpPassManager&, mlir::Operation*)>::operator()(mlir::OpPassManager&, mlir::Operation*) const /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:68:12
#34 0x000055ebe41ea615 mlir::Pass::runPipeline(mlir::OpPassManager&, mlir::Operation*) /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/llvm/../mlir/include/mlir/Pass/Pass.h:196:12
#35 0x000055ebe87c71ad (anonymous namespace)::InlinerPass::optimizeCallable(mlir::CallGraphNode*, llvm::StringMap<mlir::OpPassManager, llvm::MallocAllocator>&) /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/mlir/lib/Transforms/Inliner.cpp:796:10
#36 0x000055ebe87c6f5c (anonymous namespace)::InlinerPass::optimizeSCCAsync(llvm::MutableArrayRef<mlir::CallGraphNode*>, mlir::MLIRContext*)::$_0::operator()(mlir::CallGraphNode*) const /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/mlir/lib/Transforms/Inliner.cpp:773:28
#37 0x000055ebe87c6cda mlir::LogicalResult mlir::failableParallelForEach<mlir::CallGraphNode**, (anonymous namespace)::InlinerPass::optimizeSCCAsync(llvm::MutableArrayRef<mlir::CallGraphNode*>, mlir::MLIRContext*)::$_0>(mlir::MLIRContext*, mlir::CallGraphNode**, mlir::CallGraphNode**, (anonymous namespace)::InlinerPass::optimizeSCCAsync(llvm::MutableArrayRef<mlir::CallGraphNode*>, mlir::MLIRContext*)::$_0&&) /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/mlir/include/mlir/IR/Threading.h:46:18
#38 0x000055ebe87c6c36 mlir::LogicalResult mlir::failableParallelForEach<llvm::MutableArrayRef<mlir::CallGraphNode*>&, (anonymous namespace)::InlinerPass::optimizeSCCAsync(llvm::MutableArrayRef<mlir::CallGraphNode*>, mlir::MLIRContext*)::$_0>(mlir::MLIRContext*, llvm::MutableArrayRef<mlir::CallGraphNode*>&, (anonymous namespace)::InlinerPass::optimizeSCCAsync(llvm::MutableArrayRef<mlir::CallGraphNode*>, mlir::MLIRContext*)::$_0&&) /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/mlir/include/mlir/IR/Threading.h:92:10
#39 0x000055ebe87c6bd8 (anonymous namespace)::InlinerPass::optimizeSCCAsync(llvm::MutableArrayRef<mlir::CallGraphNode*>, mlir::MLIRContext*) /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/mlir/lib/Transforms/Inliner.cpp:762:10
#40 0x000055ebe87c5d21 (anonymous namespace)::InlinerPass::optimizeSCC(mlir::CallGraph&, (anonymous namespace)::CGUseList&, (anonymous namespace)::CallGraphSCC&, mlir::MLIRContext*) /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/mlir/lib/Transforms/Inliner.cpp:729:14
#41 0x000055ebe87c5b1b (anonymous namespace)::InlinerPass::inlineSCC((anonymous namespace)::Inliner&, (anonymous namespace)::CGUseList&, (anonymous namespace)::CallGraphSCC&, mlir::MLIRContext*) /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/mlir/lib/Transforms/Inliner.cpp:695:16
#42 0x000055ebe87c5abf (anonymous namespace)::InlinerPass::runOnOperation()::$_5::operator()((anonymous namespace)::CallGraphSCC&) const /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/mlir/lib/Transforms/Inliner.cpp:678:12
#43 0x000055ebe87c5a7d mlir::LogicalResult llvm::function_ref<mlir::LogicalResult ((anonymous namespace)::CallGraphSCC&)>::callback_fn<(anonymous namespace)::InlinerPass::runOnOperation()::$_5>(long, (anonymous namespace)::CallGraphSCC&) /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:45:12
#44 0x000055ebe87c5a31 llvm::function_ref<mlir::LogicalResult ((anonymous namespace)::CallGraphSCC&)>::operator()((anonymous namespace)::CallGraphSCC&) const /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:68:12
#45 0x000055ebe87c34ea runTransformOnCGSCCs(mlir::CallGraph const&, llvm::function_ref<mlir::LogicalResult ((anonymous namespace)::CallGraphSCC&)>) /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/mlir/lib/Transforms/Inliner.cpp:303:16
#46 0x000055ebe87c2a66 (anonymous namespace)::InlinerPass::runOnOperation() /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/mlir/lib/Transforms/Inliner.cpp:677:26
#47 0x000055ebe88b210a mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int) /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/mlir/lib/Pass/Pass.cpp:471:21
#48 0x000055ebe88b26f4 mlir::detail::OpToOpPassAdaptor::runPipeline(mlir::OpPassManager&, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int, mlir::PassInstrumentor*, mlir::PassInstrumentation::PipelineParentInfo const*) /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/mlir/lib/Pass/Pass.cpp:534:16
#49 0x000055ebe88b4069 mlir::PassManager::runPasses(mlir::Operation*, mlir::AnalysisManager) /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/mlir/lib/Pass/Pass.cpp:843:10
#50 0x000055ebe88b3f95 mlir::PassManager::run(mlir::Operation*) /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/mlir/lib/Pass/Pass.cpp:823:60
#51 0x000055ebe39a97bf performActions(llvm::raw_ostream&, bool, bool, std::shared_ptr<llvm::SourceMgr> const&, mlir::MLIRContext*, llvm::function_ref<mlir::LogicalResult (mlir::PassManager&)>, bool, bool) /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/mlir/lib/Tools/mlir-opt/MlirOptMain.cpp:90:17
#52 0x000055ebe39a9457 processBuffer(llvm::raw_ostream&, std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer>>, bool, bool, bool, bool, bool, bool, llvm::function_ref<mlir::LogicalResult (mlir::PassManager&)>, mlir::DialectRegistry&, llvm::ThreadPool*) /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/mlir/lib/Tools/mlir-opt/MlirOptMain.cpp:138:12
#53 0x000055ebe39a9222 mlir::MlirOptMain(llvm::raw_ostream&, std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer>>, llvm::function_ref<mlir::LogicalResult (mlir::PassManager&)>, mlir::DialectRegistry&, bool, bool, bool, bool, bool, bool, bool)::$_0::operator()(std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer>>, llvm::raw_ostream&) const /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/mlir/lib/Tools/mlir-opt/MlirOptMain.cpp:180:12
#54 0x000055ebe39a9132 mlir::LogicalResult llvm::function_ref<mlir::LogicalResult (std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer>>, llvm::raw_ostream&)>::callback_fn<mlir::MlirOptMain(llvm::raw_ostream&, std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer>>, llvm::function_ref<mlir::LogicalResult (mlir::PassManager&)>, mlir::DialectRegistry&, bool, bool, bool, bool, bool, bool, bool)::$_0>(long, std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer>>, llvm::raw_ostream&) /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:45:12
#55 0x000055ebe8b6bfde llvm::function_ref<mlir::LogicalResult (std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer>>, llvm::raw_ostream&)>::operator()(std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer>>, llvm::raw_ostream&) const /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:68:12
#56 0x000055ebe8b6b5fd mlir::splitAndProcessBuffer(std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer>>, llvm::function_ref<mlir::LogicalResult (std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer>>, llvm::raw_ostream&)>, llvm::raw_ostream&, bool, bool) /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/mlir/lib/Support/ToolUtilities.cpp:28:12
#57 0x000055ebe39a818b mlir::MlirOptMain(llvm::raw_ostream&, std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer>>, llvm::function_ref<mlir::LogicalResult (mlir::PassManager&)>, mlir::DialectRegistry&, bool, bool, bool, bool, bool, bool, bool) /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/mlir/lib/Tools/mlir-opt/MlirOptMain.cpp:185:10
#58 0x000055ebe39a831f mlir::MlirOptMain(llvm::raw_ostream&, std::unique_ptr<llvm::MemoryBuffer, std::default_delete<llvm::MemoryBuffer>>, mlir::PassPipelineCLParser const&, mlir::DialectRegistry&, bool, bool, bool, bool, bool, bool, bool, bool) /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/mlir/lib/Tools/mlir-opt/MlirOptMain.cpp:208:10
#59 0x000055ebe39a8f59 mlir::MlirOptMain(int, char**, llvm::StringRef, mlir::DialectRegistry&, bool) /home/chi/src/ubuntu20/shark/torch-mlir/externals/llvm-project/mlir/lib/Tools/mlir-opt/MlirOptMain.cpp:305:14
#60 0x000055ebe395b6b7 main /home/chi/src/ubuntu20/shark/torch-mlir/tools/torch-mlir-opt/torch-mlir-opt.cpp:26:7
#61 0x00007f397e229d90 __libc_start_call_main ./csu/../sysdeps/nptl/libc_start_call_main.h:58:16
#62 0x00007f397e229e40 call_init ./csu/../csu/libc-start.c:128:20
#63 0x00007f397e229e40 __libc_start_main ./csu/../csu/libc-start.c:379:5
#64 0x000055ebe395b575 _start (/home/chi/src/ubuntu20/shark/torch-mlir/build/bin/torch-mlir-opt+0x2d57575)
[1]    42049 IOT instruction (core dumped)  /home/chi/src/ubuntu20/shark/torch-mlir/build/bin/torch-mlir-opt  

[AmosLewis]

Grover's patch for multiple slice: llvm/torch-mlir#1909
Basically, it creates many aliases for a SliceOp input tensor. When there is an overwritten over the input tensor, all the old aliases for the input tensor are removed. To deal with the aliases, Grover adds an AliasSliceInfo class.

Looks like it does not solve my case. Possibly need to support the viewlike op AtenSelectIntOp.
t5_small_torchscript_test.mlir

➜  t5small git:(maximizevaluesemantics) ✗ torch-mlir-opt -pass-pipeline='builtin.module(torchscript-module-to-torch-backend-pipeline{backend-legal-ops=torch.aten.flatten.using_ints,torch.aten.native_layer_norm,torch.aten.linear})' ./t5_small_torchscript_test.mlir  -mlir-print-ir-after-failure -mlir-disable-threading 
./test_torchscript.mlir:12:12: error: unsupported by backend contract: tensor with unknown rank
    %134 = torch.aten.new_zeros %arg2, %133, %int4, %int0, %cpu, %false : !torch.tensor, !torch.list<int>, !torch.int, !torch.int, !torch.Device, !torch.bool -> !torch.tensor
           ^
./test_torchscript.mlir:12:12: note: see current operation: %11 = "torch.tensor_static_info_cast"(%10) : (!torch.vtensor<[1,4],si64>) -> !torch.vtensor<*,si64>
./test_torchscript.mlir:12:12: note: this is likely due to a missing transfer function in abstract_interp_lib_gen.py
// -----// IR Dump After LowerToBackendContract Failed (torch-lower-to-backend-contract) //----- //
module attributes {torch.debug_module_name = "_lambda"} {
  func.func @forward(%arg0: !torch.vtensor<[1,15],si64>, %arg1: !torch.vtensor<[1,4],si64>) -> !torch.vtensor<*,si64> {
    %int1 = torch.constant.int 1
    %int3 = torch.constant.int 3
    %int0 = torch.constant.int 0
    %false = torch.constant.bool false
    %int4 = torch.constant.int 4
    %int-1 = torch.constant.int -1
    %none = torch.constant.none
    %int9223372036854775807 = torch.constant.int 9223372036854775807
    %cpu = torch.constant.device "cpu"
    %0 = torch.prim.ListConstruct %int1, %int4 : (!torch.int, !torch.int) -> !torch.list<int>
    %1 = torch.aten.zeros %0, %int4, %int0, %cpu, %false : !torch.list<int>, !torch.int, !torch.int, !torch.Device, !torch.bool -> !torch.vtensor<[1,4],si64>
    %2 = torch.tensor_static_info_cast %1 : !torch.vtensor<[1,4],si64> to !torch.vtensor<*,si64>
    %3 = torch.copy.to_tensor %2 : !torch.tensor<*,si64>
    %4 = torch.aten.slice.Tensor %arg1, %int1, %int0, %int-1, %int1 : !torch.vtensor<[1,4],si64>, !torch.int, !torch.int, !torch.int, !torch.int -> !torch.vtensor<[1,3],si64>
    %5 = torch.aten.clone %4, %none : !torch.vtensor<[1,3],si64>, !torch.none -> !torch.vtensor<[1,3],si64>
    %6 = torch.aten.slice.Tensor %3, %int1, %int1, %int9223372036854775807, %int1 : !torch.tensor<*,si64>, !torch.int, !torch.int, !torch.int, !torch.int -> !torch.tensor<[1,3],si64>
    %7 = torch.tensor_static_info_cast %6 : !torch.tensor<[1,3],si64> to !torch.tensor<*,si64>
    %8 = torch.prim.ListConstruct %int1, %int3 : (!torch.int, !torch.int) -> !torch.list<int>
    %9 = torch.aten.broadcast_to %5, %8 : !torch.vtensor<[1,3],si64>, !torch.list<int> -> !torch.vtensor<[1,3],si64>
    %10 = torch.tensor_static_info_cast %9 : !torch.vtensor<[1,3],si64> to !torch.vtensor<*,si64>
    torch.overwrite.tensor.contents %10 overwrites %7 : !torch.vtensor<*,si64>, !torch.tensor<*,si64>
    %11 = torch.aten.slice.Tensor %3, %int1, %int0, %int1, %int1 : !torch.tensor<*,si64>, !torch.int, !torch.int, !torch.int, !torch.int -> !torch.tensor<[1,1],si64>
    %12 = torch.aten.squeeze.dim %11, %int1 : !torch.tensor<[1,1],si64>, !torch.int -> !torch.tensor<[1],si64>
    %13 = torch.tensor_static_info_cast %12 : !torch.tensor<[1],si64> to !torch.tensor<*,si64>
    %14 = torch.copy.to_vtensor %13 : !torch.vtensor<*,si64>
    return %14 : !torch.vtensor<*,si64>
  }
}

0302 update: After talking with Ramiro, we think the Grover patch cannot handle t5 case. Need to use George's fold index and copy_ into the index_put_ solution.

0303 update: Grover's new implementation works well for t5_small_torchscript_test.mlir.
But after the torch.aten.select.int op in t5_small_torchscript_test.mlir. I also have the torch.aten.eq.Scalar and torch.aten.masked_fill_.Scalar in T5 model to use the %1 torch.aten.zeros. If I add these 2 ops, I still got the same bug. Here is the new t5_small_torchscript_test2.mlir. Then Grover found that this cannot be handle by his solution.

[AmosLewis]

Fix the slice and copy end = -1 llvm/torch-mlir#1901

 %135 = torch.aten.slice.Tensor %arg2, %int1, %int0, %int-1, %int1 : !torch.tensor, !torch.int, !torch.int, !torch.int, !torch.int -> !torch.tensor
    %136 = torch.aten.clone %135, %none_1 : !torch.tensor, !torch.none -> !torch.tensor

But end = INT_MAX / %int9223372036854775807 should be fixed next:
%137 = torch.aten.slice.Tensor %134, %int1, %int1, %int9223372036854775807, %int1 : !torch.tensor, !torch.int, !torch.int, !torch.int, !torch.int -> !torch.tensor
Then end should not INT_MAX. Looks like the INT_MAX is generated before ts_g = torch.jit.script(fx_g)

I try to print graph after makd_fx, it shows the 4 INT_MAX=9223372036854775807 in slice end :

print("fx_g.graph: ")
print(fx_g.graph)
# fx_g.graph:
# graph(): 9223372036854775807
#     %arg0_1 : [#users=1] = placeholder[target=arg0_1]
#     %arg1_1 : [#users=2] = placeholder[target=arg1_1]
#     %new_zeros : [#users=4] = call_function[target=torch.ops.aten.new_zeros.default](args = (%arg1_1, [1, 4]), kwargs = {dtype: torch.int64, layout: torch.strided, device: cpu, pin_memory: False})
#     %slice_1 : [#users=1] = call_function[target=torch.ops.aten.slice.Tensor](args = (%arg1_1, 1, 0, -1), kwargs = {})
#     %clone : [#users=1] = call_function[target=torch.ops.aten.clone.default](args = (%slice_1,), kwargs = {})
#     %slice_2 : [#users=1] = call_function[target=torch.ops.aten.slice.Tensor](args = (%new_zeros, 1, 1, 9223372036854775807), kwargs = {})
#     %copy_ : [#users=0] = call_function[target=torch.ops.aten.copy_.default](args = (%slice_2, %clone), kwargs = {})
#     %_tensor_constant0 : [#users=1] = get_attr[target=_tensor_constant0]
#     %lift_fresh_copy : [#users=1] = call_function[target=torch.ops.aten.lift_fresh_copy.default](args = (%_tensor_constant0,), kwargs = {})
#     %select : [#users=1] = call_function[target=torch.ops.aten.select.int](args = (%new_zeros, 1, 0), kwargs = {})
#     %fill_ : [#users=0] = call_function[target=torch.ops.aten.fill_.Tensor](args = (%select, %lift_fresh_copy), kwargs = {})
#     %eq : [#users=1] = call_function[target=torch.ops.aten.eq.Scalar](args = (%new_zeros, -100), kwargs = {})
#     %masked_fill_ : [#users=1] = call_function[target=torch.ops.aten.masked_fill_.Scalar](args = (%new_zeros, %eq, 0), kwargs = {})
#     %view : [#users=1] = call_function[target=torch.ops.aten.view.default](args = (%arg0_1, [-1, 15]), kwargs = {})
#     %_param_constant0 : [#users=1] = get_attr[target=_param_constant0]
#     %embedding : [#users=3] = call_function[target=torch.ops.aten.embedding.default](args = (%_param_constant0, %view), kwargs = {})
#     %ones : [#users=1] = call_function[target=torch.ops.aten.ones.default](args = ([1, 15],), kwargs = {device: cpu, pin_memory: False})
#     %slice_3 : [#users=1] = call_function[target=torch.ops.aten.slice.Tensor](args = (%ones, 0, 0, 9223372036854775807), kwargs = {})
#     %unsqueeze : [#users=1] = call_function[target=torch.ops.aten.unsqueeze.default](args = (%slice_3, 1), kwargs = {})
#     %unsqueeze_1 : [#users=1] = call_function[target=torch.ops.aten.unsqueeze.default](args = (%unsqueeze, 2), kwargs = {})
#     %slice_4 : [#users=1] = call_function[target=torch.ops.aten.slice.Tensor](args = (%unsqueeze_1, 3, 0, 9223372036854775807), kwargs = {})
#     %rsub : [#users=1] = call_function[target=torch.ops.aten.rsub.Scalar](args = (%slice_4, 1.0), kwargs = {})

The new_zeros(input of slice) is probably defined in T5Atention:
https://github.com/huggingface/transformers/blob/101a6cd276d454c6ab07aff3c54e598ff83d537c/src/transformers/models/t5/modeling_t5.py#L536
Then the slice ops should be right after this. Probably here: https://github.com/huggingface/transformers/blob/101a6cd276d454c6ab07aff3c54e598ff83d537c/src/transformers/models/t5/modeling_t5.py#L547 position_bias = position_bias[:, :, -hidden_states.size(1) :, :]

But after printing around this line, the position_bias = position_bias[:, :, -hidden_states.size(1) :, :] is not run because the past_key_value is None. Then I go back to print the self.compute_bias() -> self._relative_position_bucket(). There is index slice around the relative_position definition print("relative_position: ", relative_position.size()) # relative_position: torch.Size([15, 15]) and print("context_position: ", context_position.size()) # torch.Size([1, 15]). Based on the size info 15, I think it is probably the %slice_3. Then I guess the new_zeros should be defined inside self._relative_position_bucket().

In definition of slice.tensor
Function Schema: slice.Tensor(Tensor(a) self, int dim=0, int start=0, int end=9223372036854775807, int step=1) -> Tensor(a)
pytorch/TensorRT#111.

I thought the end=9223372036854775807 is probably because the end is not set, then it use default value.

But then check the definition of at::slice https://pytorch.org/cppdocs/api/function_namespaceat_1ae6bdafd2514da71bc5a656a649bcb575.html?highlight=slice#_CPPv4N2at5sliceERKN2at6TensorE7int64_tN3c108optionalI7int64_tEEN3c108optionalI7int64_tEE7int64_t
The end = c10::[nullopt] == 0. Its default is not 9223372036854775807.

So the set to 9223372036854775807 should be somewhere else.

[AmosLewis]

Finally find where the new_zeros and clone defined in t5 model. It locate at T5ForConditionalGeneration(T5PreTrainedModel) -> T5PreTrainedModel() -> _shift_right(self, decoder_input_ids): https://github.com/huggingface/transformers/blob/101a6cd276d454c6ab07aff3c54e598ff83d537c/src/transformers/models/t5/modeling_t5.py#L850 :

            shifted_input_ids = input_ids.new_zeros(input_ids.shape)
            shifted_input_ids[..., 1:] = input_ids[..., :-1].clone()
            shifted_input_ids[..., 0] = decoder_start_token_id

The small test to repeat this error: https://gist.github.com/AmosLewis/b53dfecbe4918618031cac01c8a88fb9
The filed torch_mlir issue and e2e test:
llvm/torch-mlir#1953

[AmosLewis]

The slice+copy int64_max is fixed by llvm/torch-mlir#1970
It only fixes this

            shifted_input_ids = input_ids.new_zeros(input_ids.shape)
            shifted_input_ids[..., 1:] = input_ids[..., :-1].clone()

But the
shifted_input_ids[..., 0] = decoder_start_token_id
is a slice+fill issue.