`MultiRegion` adaptation of the `NonhydrostaticModel` #2795

simone-silvestri · 2022-11-02T16:39:54Z

At the moment, the pressure solve is performed on one GPU/CPU, but this PR builds the infrastructure to allow multi-process global solves

tomchor · 2022-11-02T19:11:25Z

Instructions to adapt the code to MultiRegion from a Slack conversation with @simone-silvestri:

If you want to adapt your script to multiregion you have to:

define a multiregion grid with: grid = MultiRegionGrid(grid, partition = XPartition(n_gpus_you_want_to_use), devices = n_gpus_you_want_to_use) . The multiregion grid supersedes the immersed boundary grid, i.e. if you are using an immersed boundary grid then grid = MultiRegionGrid(ibg; kwargs...) , not the other way around.
if you are using any array for forcing or boundary condition, you have to adapt it to the multiregion paradigm as follows: using Oceananigans.MultiRegion: multi_region_object_from_array; my_adapted_array = multi_region_object_from_array(my_array, grid)

MultiRegion works only on single node multi GPU, so all the GPUs should be accessible from a single process in the node. You can check the number of GPUs available by logging in a node and typing nvidia-smi , if you want to split your grid on specific devices (let’s say GPU 0 and 3), you can pass devices = (0, 3) to the MultiRegionGrid constructor.

There is another thing that you have to take care of: the pressure solve is performed on one GPU only so both the storage and source term (a field of complex values of the size of the full grid) reside on 1 GPU only (usually the one corresponding to the last region). This means that if your grid is 100M points, 2.98 GB will have to be reserved for the solver’s auxiliary fields

julia> sizeof(complex(zeros(Int(100e6)))) / 1024 / 1024 / 1024 * 2
2.9802322387695312

So make sure you have that space available. (When I have time I ll try to find a solution to run truly parallel pressure solvers, for both nonhydrostatic and hydrostatic models)

In terms of outputs, we make use of reconstruct_global_field , a function used to reconstruct a global field from a MultiRegionField on the CPU. It is used by the output writers to spit out the full field. It is a slow procedure though, so I would advise against performing too many involved diagnostics.

tomchor · 2022-11-03T16:20:24Z

@simone-silvestri you mentioned that in this PR the pressure solver is done on a single GPU for the time being. Can #2538 be a stating point to optimize that as well? That PR is really close to ready I think

glwagner · 2022-11-03T21:22:00Z

#2538 uses PencilFFTs which last I checked does not support CuArray, and explicitly uses MPI (which MultiRegion does not), so you can't use the code directly. However, you can take inspiration from the algorithm there.

However, I suggest starting with a restricted decomposition where the domain is just distributed in x. (@simone-silvestri might start working on that...)

tomchor · 2022-11-04T03:08:24Z

@simone-silvestri this doesn't seem to be working when the domain is Bounded in the x direction. Is this expected?:

ERROR: LoadError: MethodError: no method matching PressureSolver(::CPU, ::MultiRegionGrid{Float64, Bounded, Bounded, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, RightConnected, Bounded, Bounded, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, RectilinearGrid{Float64, LeftConnected, Bounded, Bounded, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU})
Closest candidates are:
  PressureSolver(::Any, ::RectilinearGrid{<:Any, <:Any, <:Any, <:Any, <:Number, <:Number, <:Number}) at ~/.julia/packages/Oceananigans/F1fni/src/Models/NonhydrostaticModels/NonhydrostaticModels.jl:24
  PressureSolver(::Any, ::RectilinearGrid{<:Any, <:Any, <:Any, <:Any, <:Number, <:Number}) at ~/.julia/packages/Oceananigans/F1fni/src/Models/NonhydrostaticModels/NonhydrostaticModels.jl:25
  PressureSolver(::Any, ::ImmersedBoundaryGrid) at ~/.julia/packages/Oceananigans/F1fni/src/Models/NonhydrostaticModels/NonhydrostaticModels.jl:28
  ...
Stacktrace:
 [1] NonhydrostaticModel(; grid::MultiRegionGrid{Float64, Bounded, Bounded, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, RightConnected, Bounded, Bounded, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, RectilinearGrid{Float64, LeftConnected, Bounded, Bounded, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, clock::Clock{Float64}, advection::WENO{3, Float64, Nothing, Nothing, Nothing, Nothing, true, Nothing, WENO{2, Float64, Nothing, Nothing, Nothing, Nothing, true, Nothing, UpwindBiased{1, Float64, Nothing, Nothing, Nothing, Nothing, Centered{1, Float64, Nothing, Nothing, Nothing, Nothing}}, Centered{1, Float64, Nothing, Nothing, Nothing, Nothing}}, Centered{2, Float64, Nothing, Nothing, Nothing, Centered{1, Float64, Nothing, Nothing, Nothing, Nothing}}}, buoyancy::Nothing, coriolis::Nothing, stokes_drift::Nothing, forcing::NamedTuple{(), Tuple{}}, closure::Nothing, boundary_conditions::NamedTuple{(), Tuple{}}, tracers::Tuple{}, timestepper::Symbol, background_fields::NamedTuple{(), Tuple{}}, particles::Nothing, velocities::Nothing, pressures::Nothing, diffusivity_fields::Nothing, pressure_solver::Nothing, auxiliary_fields::NamedTuple{(), Tuple{}})
   @ Oceananigans.Models.NonhydrostaticModels ~/.julia/packages/Oceananigans/F1fni/src/Models/NonhydrostaticModels/nonhydrostatic_model.jl:167

tomchor · 2022-11-04T16:47:48Z

In addition to the bounded-x error above, interpolation methods also appear to be failing at times:

using Oceananigans

grid_base = RectilinearGrid(size=(4, 4, 4), extent = (1,1,1))
grid = MultiRegionGrid(grid_base, partition = XPartition(2), devices = 2)
@info grid

model = NonhydrostaticModel(grid = grid)
@info "" model

u, v, w = model.velocities

ω_x = Field((@at (Center, Face, Face) ∂y(w)-∂z(v)))

This throws me the following error:

ERROR: LoadError: MethodError: no method matching interpolate_index(::Tuple{Colon, Colon, Colon}, ::Colon, ::Type{Center}, ::Type{Center})
Closest candidates are:
  interpolate_index(::UnitRange, ::Colon, ::Any, ::Any) at ~/.julia/packages/Oceananigans/F1fni/src/AbstractOperations/at.jl:80
  interpolate_index(::Colon, ::Colon, ::Any...) at ~/.julia/packages/Oceananigans/F1fni/src/AbstractOperations/at.jl:77
  interpolate_index(::Colon, ::UnitRange, ::Any...) at ~/.julia/packages/Oceananigans/F1fni/src/AbstractOperations/at.jl:78
  ...
Stacktrace:
 [1] interpolate_indices(::Oceananigans.AbstractOperations.Derivative{Center, Face, Face, typeof(Oceananigans.Operators.∂yᶜᶠᶠ), Field{Center, Center, Face, Nothing, MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, MultiRegionObject{Tuple{Tuple{Colon, Colon, Colon}, Tuple{Colon, Colon, Colon}}, Tuple{CPU, CPU}}, MultiRegionObject{Tuple{OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}}, Tuple{CPU, CPU}}, Any, MultiRegionObject{Tuple{FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Tuple{CPU, CPU}}, Nothing, MultiRegionObject{Tuple{Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}, Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}}, Tuple{CPU, CPU}}}, typeof(Oceananigans.Operators.identity2), typeof(∂y), MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, Float64}, ::Vararg{Any}; loc_operation::Tuple{DataType, DataType, DataType})
   @ Oceananigans.AbstractOperations ~/.julia/packages/Oceananigans/F1fni/src/AbstractOperations/at.jl:70
 [2] construct_regionally(::typeof(Oceananigans.AbstractOperations.interpolate_indices), ::Oceananigans.AbstractOperations.Derivative{Center, Face, Face, typeof(Oceananigans.Operators.∂yᶜᶠᶠ), Field{Center, Center, Face, Nothing, MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, MultiRegionObject{Tuple{Tuple{Colon, Colon, Colon}, Tuple{Colon, Colon, Colon}}, Tuple{CPU, CPU}}, MultiRegionObject{Tuple{OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}}, Tuple{CPU, CPU}}, Any, MultiRegionObject{Tuple{FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Tuple{CPU, CPU}}, Nothing, MultiRegionObject{Tuple{Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}, Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}}, Tuple{CPU, CPU}}}, typeof(Oceananigans.Operators.identity2), typeof(∂y), MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, Float64}, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Tuple{DataType, DataType, DataType}, Tuple{Symbol}, NamedTuple{(:loc_operation,), Tuple{Tuple{DataType, DataType, DataType}}}})
   @ Oceananigans.Utils ~/.julia/packages/Oceananigans/F1fni/src/Utils/multi_region_transformation.jl:116
 [3] indices(β::Oceananigans.AbstractOperations.BinaryOperation{Center, Face, Face, typeof(-), Oceananigans.AbstractOperations.Derivative{Center, Face, Face, typeof(Oceananigans.Operators.∂yᶜᶠᶠ), Field{Center, Center, Face, Nothing, MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, MultiRegionObject{Tuple{Tuple{Colon, Colon, Colon}, Tuple{Colon, Colon, Colon}}, Tuple{CPU, CPU}}, MultiRegionObject{Tuple{OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}}, Tuple{CPU, CPU}}, Any, MultiRegionObject{Tuple{FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Tuple{CPU, CPU}}, Nothing, MultiRegionObject{Tuple{Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}, Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}}, Tuple{CPU, CPU}}}, typeof(Oceananigans.Operators.identity2), typeof(∂y), MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, Float64}, Oceananigans.AbstractOperations.Derivative{Center, Face, Face, typeof(Oceananigans.Operators.∂zᶜᶠᶠ), Field{Center, Face, Center, Nothing, MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, MultiRegionObject{Tuple{Tuple{Colon, Colon, Colon}, Tuple{Colon, Colon, Colon}}, Tuple{CPU, CPU}}, MultiRegionObject{Tuple{OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}}, Tuple{CPU, CPU}}, Any, MultiRegionObject{Tuple{FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Tuple{CPU, CPU}}, Nothing, MultiRegionObject{Tuple{Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}, Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}}, Tuple{CPU, CPU}}}, typeof(Oceananigans.Operators.identity3), typeof(∂z), MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, Float64}, typeof(Oceananigans.Operators.identity4), typeof(Oceananigans.Operators.identity5), MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, Float64})
   @ Oceananigans.AbstractOperations ~/.julia/packages/Oceananigans/F1fni/src/AbstractOperations/binary_operations.jl:32
 [4] Field(operand::Oceananigans.AbstractOperations.BinaryOperation{Center, Face, Face, typeof(-), Oceananigans.AbstractOperations.Derivative{Center, Face, Face, typeof(Oceananigans.Operators.∂yᶜᶠᶠ), Field{Center, Center, Face, Nothing, MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, MultiRegionObject{Tuple{Tuple{Colon, Colon, Colon}, Tuple{Colon, Colon, Colon}}, Tuple{CPU, CPU}}, MultiRegionObject{Tuple{OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}}, Tuple{CPU, CPU}}, Any, MultiRegionObject{Tuple{FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Tuple{CPU, CPU}}, Nothing, MultiRegionObject{Tuple{Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}, Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}}, Tuple{CPU, CPU}}}, typeof(Oceananigans.Operators.identity2), typeof(∂y), MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, Float64}, Oceananigans.AbstractOperations.Derivative{Center, Face, Face, typeof(Oceananigans.Operators.∂zᶜᶠᶠ), Field{Center, Face, Center, Nothing, MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, MultiRegionObject{Tuple{Tuple{Colon, Colon, Colon}, Tuple{Colon, Colon, Colon}}, Tuple{CPU, CPU}}, MultiRegionObject{Tuple{OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}}, Tuple{CPU, CPU}}, Any, MultiRegionObject{Tuple{FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Tuple{CPU, CPU}}, Nothing, MultiRegionObject{Tuple{Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}, Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}}, Tuple{CPU, CPU}}}, typeof(Oceananigans.Operators.identity3), typeof(∂z), MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, Float64}, typeof(Oceananigans.Operators.identity4), typeof(Oceananigans.Operators.identity5), MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, Float64})
   @ Oceananigans.AbstractOperations ~/.julia/packages/Oceananigans/F1fni/src/AbstractOperations/computed_field.jl:37

simone-silvestri · 2022-11-04T17:51:34Z

thanks for testing, both problems should be solved

tomchor · 2022-11-04T19:00:34Z

thanks for testing, both problems should be solved

Indeed they are! Thanks @simone-silvestri.

One thing I should note is that this fails when Nx isn't even (I guess becasue the partition is in x):

julia> grid_base = RectilinearGrid(topology=(Bounded, Periodic, Bounded), size=(5, 4, 4), extent = (1,1,1))
5×4×4 RectilinearGrid{Float64, Bounded, Periodic, Bounded} on CPU with 3×3×3 halo
├── Bounded  x ∈ [0.0, 1.0]  regularly spaced with Δx=0.2
├── Periodic y ∈ [0.0, 1.0)  regularly spaced with Δy=0.25
└── Bounded  z ∈ [-1.0, 0.0] regularly spaced with Δz=0.25

julia> grid = MultiRegionGrid(grid_base, partition = XPartition(2), devices = 2)
┌ Warning: MultiRegion functionalities are experimental: help the development by reporting bugs or non-implemented features!
└ @ Oceananigans.MultiRegion ~/.julia/packages/Oceananigans/E1180/src/MultiRegion/multi_region_grid.jl:64
ERROR: AssertionError: mod(Nx, p.div) == 0
Stacktrace:
 [1] partition_size(p::XPartition{Int64}, grid::RectilinearGrid{Float64, Bounded, Periodic, Bounded, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU})
   @ Oceananigans.MultiRegion ~/.julia/packages/Oceananigans/E1180/src/MultiRegion/x_partitions.jl:24
 [2] MultiRegionGrid(global_grid::RectilinearGrid{Float64, Bounded, Periodic, Bounded, Float64, Float64, Float64, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, CPU}; partition::XPartition{Int64}, devices::Int64)
   @ Oceananigans.MultiRegion ~/.julia/packages/Oceananigans/E1180/src/MultiRegion/multi_region_grid.jl:72

I'm assuming this is by design, no? Meaning: all the "sub"-grids have to have the same size and therefore be exactly divisible by the number of partitions?

This seems very reasonable, btw. I'm only asking because this wasn't the case before these last commits.

tomchor · 2022-11-04T19:10:17Z

Another issue! I'm getting this error when setting up a NetCDFWriter:

caused by: MethodError: no method matching construct_output(::Field{Center, Center, Face, Oceananigans.AbstractOperations.Derivative{Center, Center, Face, typeof(∂zᶜᶜᶠ), Field{Center, Center, Center, Nothing, MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, MultiRegionObject{Tuple{Tuple{Colon, Colon, Colon}, Tuple{Colon, Colon, Colon}}, Tuple{CPU, CPU}}, MultiRegionObject{Tuple{OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}}, Tuple{CPU, CPU}}, Any, MultiRegionObject{Tuple{FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Gradient, Float64}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Gradient, Float64}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Tuple{CPU, CPU}}, Nothing, MultiRegionObject{Tuple{Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}, Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}}, Tuple{CPU, CPU}}}, typeof(Oceananigans.Operators.identity1), typeof(∂z), MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, Float64}, MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, MultiRegionObject{Tuple{Tuple{Colon, Colon, Colon}, Tuple{Colon, Colon, Colon}}, Tuple{CPU, CPU}}, MultiRegionObject{Tuple{OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}}, Tuple{CPU, CPU}}, Any, MultiRegionObject{Tuple{FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Tuple{CPU, CPU}}, Nothing, MultiRegionObject{Tuple{Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}, Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}}, Tuple{CPU, CPU}}}, ::Tuple{Colon, Colon, Colon})
Closest candidates are:
  construct_output(::Field{LX, LY, LZ, O, <:MultiRegionGrid} where {LX, LY, LZ, O}, ::Any, ::Any, ::Any) at ~/.julia/packages/Oceananigans/E1180/src/MultiRegion/multi_region_output_writers.jl:17
  construct_output(::Union{Oceananigans.Fields.AbstractField, Reduction}, ::Any, ::Any, ::Any) at ~/.julia/packages/Oceananigans/E1180/src/OutputWriters/output_construction.jl:62
  construct_output(::Any, ::Any, ::Any, ::Any) at ~/.julia/packages/Oceananigans/E1180/src/OutputWriters/output_construction.jl:37
  ...
Stacktrace:
  [1] construct_output(mrf::Field{Center, Center, Face, Oceananigans.AbstractOperations.Derivative{Center, Center, Face, typeof(∂zᶜᶜᶠ), Field{Center, Center, Center, Nothing, MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, MultiRegionObject{Tuple{Tuple{Colon, Colon, Colon}, Tuple{Colon, Colon, Colon}}, Tuple{CPU, CPU}}, MultiRegionObject{Tuple{OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}}, Tuple{CPU, CPU}}, Any, MultiRegionObject{Tuple{FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Gradient, Float64}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Gradient, Float64}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Tuple{CPU, CPU}}, Nothing, MultiRegionObject{Tuple{Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}, Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}}, Tuple{CPU, CPU}}}, typeof(Oceananigans.Operators.identity1), typeof(∂z), MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, Float64}, MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, MultiRegionObject{Tuple{Tuple{Colon, Colon, Colon}, Tuple{Colon, Colon, Colon}}, Tuple{CPU, CPU}}, MultiRegionObject{Tuple{OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}}, Tuple{CPU, CPU}}, Any, MultiRegionObject{Tuple{FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Tuple{CPU, CPU}}, Nothing, MultiRegionObject{Tuple{Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}, Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}}, Tuple{CPU, CPU}}}, grid::MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, user_indices::Tuple{Colon, Colon, Colon}, with_halos::Bool)
    @ Oceananigans.MultiRegion ~/.julia/packages/Oceananigans/E1180/src/MultiRegion/multi_region_output_writers.jl:24
  [2] (::Oceananigans.OutputWriters.var"#41#42"{Tuple{Colon, Colon, Colon}, Bool, NonhydrostaticModel{Oceananigans.TimeSteppers.RungeKutta3TimeStepper{Float64, NamedTuple{(:u, :v, :w, :b), Tuple{Field{Face, Center, Center, Nothing, MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, MultiRegionObject{Tuple{Tuple{Colon, Colon, Colon}, Tuple{Colon, Colon, Colon}}, Tuple{CPU, CPU}}, MultiRegionObject{Tuple{OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}}, Tuple{CPU, CPU}}, Any, MultiRegionObject{Tuple{FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Tuple{CPU, CPU}}, Nothing, MultiRegionObject{Tuple{Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}, Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}}, Tuple{CPU, CPU}}}, Field{Center, Face, Center, Nothing, MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, MultiRegionObject{Tuple{Tuple{Colon, Colon, Colon}, Tuple{Colon, Colon, Colon}}, Tuple{CPU, CPU}}, MultiRegionObject{Tuple{OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}}, Tuple{CPU, CPU}}, Any, MultiRegionObject{Tuple{FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Tuple{CPU, CPU}}, Nothing, MultiRegionObject{Tuple{Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}, Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}}, Tuple{CPU, CPU}}}, Field{Center, Center, Face, Nothing, MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, MultiRegionObject{Tuple{Tuple{Colon, Colon, Colon}, Tuple{Colon, Colon, Colon}}, Tuple{CPU, CPU}}, MultiRegionObject{Tuple{OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}}, Tuple{CPU, CPU}}, Any, MultiRegionObject{Tuple{FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Tuple{CPU, CPU}}, Nothing, MultiRegionObject{Tuple{Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}, Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}}, Tuple{CPU, CPU}}}, Field{Center, Center, Center, Nothing, MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, MultiRegionObject{Tuple{Tuple{Colon, Colon, Colon}, Tuple{Colon, Colon, Colon}}, Tuple{CPU, CPU}}, MultiRegionObject{Tuple{OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}}, Tuple{CPU, CPU}}, Any, MultiRegionObject{Tuple{FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Tuple{CPU, CPU}}, Nothing, MultiRegionObject{Tuple{Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}, Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}}, Tuple{CPU, CPU}}}}}, Nothing}, SmagorinskyLilly{Oceananigans.TurbulenceClosures.ExplicitTimeDiscretization, Float64, NamedTuple{(:b,), Tuple{Float64}}}, CPU, MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, Float64, Buoyancy{BuoyancyTracer, Oceananigans.Grids.ZDirection}, FPlane{Float64}, Nothing, NamedTuple{(:u, :v, :w), Tuple{Field{Face, Center, Center, Nothing, MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, MultiRegionObject{Tuple{Tuple{Colon, Colon, Colon}, Tuple{Colon, Colon, Colon}}, Tuple{CPU, CPU}}, MultiRegionObject{Tuple{OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}}, Tuple{CPU, CPU}}, Any, MultiRegionObject{Tuple{FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, ImmersedBoundaryCondition{Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Face, Nothing, Center, Oceananigans.BoundaryConditions.LeftBoundary, typeof(τˣʸ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(Oceananigans.Operators.identity3), typeof(ℑxzᶠᵃᶜ)}}}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Face, Nothing, Center, Oceananigans.BoundaryConditions.RightBoundary, typeof(τˣʸ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(Oceananigans.Operators.identity4), typeof(ℑxzᶠᵃᶜ)}}}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Face, Center, Nothing, Oceananigans.BoundaryConditions.LeftBoundary, typeof(τˣᶻ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(Oceananigans.Operators.identity5), typeof(ℑxyᶠᶜᵃ)}}}, Nothing}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, ImmersedBoundaryCondition{Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Face, Nothing, Center, Oceananigans.BoundaryConditions.LeftBoundary, typeof(τˣʸ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(Oceananigans.Operators.identity3), typeof(ℑxzᶠᵃᶜ)}}}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Face, Nothing, Center, Oceananigans.BoundaryConditions.RightBoundary, typeof(τˣʸ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(Oceananigans.Operators.identity4), typeof(ℑxzᶠᵃᶜ)}}}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Face, Center, Nothing, Oceananigans.BoundaryConditions.LeftBoundary, typeof(τˣᶻ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(Oceananigans.Operators.identity5), typeof(ℑxyᶠᶜᵃ)}}}, Nothing}}}, Tuple{CPU, CPU}}, Nothing, MultiRegionObject{Tuple{Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}, Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}}, Tuple{CPU, CPU}}}, Field{Center, Face, Center, Nothing, MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, MultiRegionObject{Tuple{Tuple{Colon, Colon, Colon}, Tuple{Colon, Colon, Colon}}, Tuple{CPU, CPU}}, MultiRegionObject{Tuple{OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}}, Tuple{CPU, CPU}}, Any, MultiRegionObject{Tuple{FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, ImmersedBoundaryCondition{BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Nothing, Face, Center, Oceananigans.BoundaryConditions.LeftBoundary, typeof(τʸˣ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(Oceananigans.Operators.identity1), typeof(ℑyzᵃᶠᶜ)}}}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Nothing, Face, Center, Oceananigans.BoundaryConditions.RightBoundary, typeof(τʸˣ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(Oceananigans.Operators.identity2), typeof(ℑyzᵃᶠᶜ)}}}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Center, Face, Nothing, Oceananigans.BoundaryConditions.LeftBoundary, typeof(τʸᶻ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(ℑxyᶜᶠᵃ), typeof(Oceananigans.Operators.identity3)}}}, Nothing}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, ImmersedBoundaryCondition{BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Nothing, Face, Center, Oceananigans.BoundaryConditions.LeftBoundary, typeof(τʸˣ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(Oceananigans.Operators.identity1), typeof(ℑyzᵃᶠᶜ)}}}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Nothing, Face, Center, Oceananigans.BoundaryConditions.RightBoundary, typeof(τʸˣ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(Oceananigans.Operators.identity2), typeof(ℑyzᵃᶠᶜ)}}}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Center, Face, Nothing, Oceananigans.BoundaryConditions.LeftBoundary, typeof(τʸᶻ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(ℑxyᶜᶠᵃ), typeof(Oceananigans.Operators.identity3)}}}, Nothing}}}, Tuple{CPU, CPU}}, Nothing, MultiRegionObject{Tuple{Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}, Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}}, Tuple{CPU, CPU}}}, Field{Center, Center, Face, Nothing, MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, MultiRegionObject{Tuple{Tuple{Colon, Colon, Colon}, Tuple{Colon, Colon, Colon}}, Tuple{CPU, CPU}}, MultiRegionObject{Tuple{OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}}, Tuple{CPU, CPU}}, Any, MultiRegionObject{Tuple{FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open, Nothing}, ImmersedBoundaryCondition{BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Nothing, Center, Face, Oceananigans.BoundaryConditions.LeftBoundary, typeof(τᶻˣ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(ℑyzᵃᶜᶠ), typeof(Oceananigans.Operators.identity4)}}}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Nothing, Center, Face, Oceananigans.BoundaryConditions.RightBoundary, typeof(τᶻˣ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(ℑyzᵃᶜᶠ), typeof(Oceananigans.Operators.identity5)}}}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Center, Nothing, Face, Oceananigans.BoundaryConditions.LeftBoundary, typeof(τᶻʸ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(ℑxzᶜᵃᶠ), typeof(Oceananigans.Operators.identity1)}}}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Center, Nothing, Face, Oceananigans.BoundaryConditions.RightBoundary, typeof(τᶻʸ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(ℑxzᶜᵃᶠ), typeof(Oceananigans.Operators.identity2)}}}, Nothing, Nothing}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open, Nothing}, ImmersedBoundaryCondition{BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Nothing, Center, Face, Oceananigans.BoundaryConditions.LeftBoundary, typeof(τᶻˣ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(ℑyzᵃᶜᶠ), typeof(Oceananigans.Operators.identity4)}}}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Nothing, Center, Face, Oceananigans.BoundaryConditions.RightBoundary, typeof(τᶻˣ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(ℑyzᵃᶜᶠ), typeof(Oceananigans.Operators.identity5)}}}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Center, Nothing, Face, Oceananigans.BoundaryConditions.LeftBoundary, typeof(τᶻʸ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(ℑxzᶜᵃᶠ), typeof(Oceananigans.Operators.identity1)}}}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Center, Nothing, Face, Oceananigans.BoundaryConditions.RightBoundary, typeof(τᶻʸ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(ℑxzᶜᵃᶠ), typeof(Oceananigans.Operators.identity2)}}}, Nothing, Nothing}}}, Tuple{CPU, CPU}}, Nothing, MultiRegionObject{Tuple{Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}, Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}}, Tuple{CPU, CPU}}}}}, NamedTuple{(:b,), Tuple{Field{Center, Center, Center, Nothing, MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, MultiRegionObject{Tuple{Tuple{Colon, Colon, Colon}, Tuple{Colon, Colon, Colon}}, Tuple{CPU, CPU}}, MultiRegionObject{Tuple{OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}}, Tuple{CPU, CPU}}, Any, MultiRegionObject{Tuple{FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Gradient, Float64}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Gradient, Float64}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Tuple{CPU, CPU}}, Nothing, MultiRegionObject{Tuple{Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}, Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}}, Tuple{CPU, CPU}}}}}, NamedTuple{(:pHY′, :pNHS), Tuple{Field{Center, Center, Center, Nothing, MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, MultiRegionObject{Tuple{Tuple{Colon, Colon, Colon}, Tuple{Colon, Colon, Colon}}, Tuple{CPU, CPU}}, MultiRegionObject{Tuple{OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}}, Tuple{CPU, CPU}}, Any, MultiRegionObject{Tuple{FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Tuple{CPU, CPU}}, Nothing, MultiRegionObject{Tuple{Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}, Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}}, Tuple{CPU, CPU}}}, Field{Center, Center, Center, Nothing, MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, MultiRegionObject{Tuple{Tuple{Colon, Colon, Colon}, Tuple{Colon, Colon, Colon}}, Tuple{CPU, CPU}}, MultiRegionObject{Tuple{OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}}, Tuple{CPU, CPU}}, Any, MultiRegionObject{Tuple{FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Tuple{CPU, CPU}}, Nothing, MultiRegionObject{Tuple{Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}, Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}}, Tuple{CPU, CPU}}}}}, NamedTuple{(:u, :v, :w, :b), Tuple{Oceananigans.Forcings.ContinuousForcing{Face, Center, Center, NamedTuple{(:H, :L, :σ, :sp_length_y, :V∞, :f₀, :N²∞), NTuple{7, Float64}}, typeof(forc_u), Tuple{Symbol}, Tuple{Int64}, Tuple{typeof(Oceananigans.Operators.identity5)}}, Oceananigans.Forcings.ContinuousForcing{Center, Face, Center, NamedTuple{(:H, :L, :σ, :sp_length_y, :V∞, :f₀, :N²∞), NTuple{7, Float64}}, typeof(forc_v), Tuple{Symbol}, Tuple{Int64}, Tuple{typeof(Oceananigans.Operators.identity1)}}, Oceananigans.Forcings.ContinuousForcing{Center, Center, Face, NamedTuple{(:H, :L, :σ, :sp_length_y, :V∞, :f₀, :N²∞), NTuple{7, Float64}}, typeof(forc_w), Tuple{Symbol}, Tuple{Int64}, Tuple{typeof(Oceananigans.Operators.identity2)}}, Oceananigans.Forcings.ContinuousForcing{Center, Center, Center, NamedTuple{(:H, :L, :σ, :sp_length_y, :V∞, :f₀, :N²∞), NTuple{7, Float64}}, typeof(forc_b), Tuple{Symbol}, Tuple{Int64}, Tuple{typeof(Oceananigans.Operators.identity3)}}}}, WENO{3, Float64, Nothing, Nothing, NTuple{4, OffsetArrays.OffsetVector{Tuple{Float64, Float64, Float64}, Vector{Tuple{Float64, Float64, Float64}}}}, Nothing, true, Nothing, WENO{2, Float64, Nothing, Nothing, Tuple{OffsetArrays.OffsetVector{Tuple{Float64, Float64}, Vector{Tuple{Float64, Float64}}}, OffsetArrays.OffsetVector{Tuple{Float64, Float64}, Vector{Tuple{Float64, Float64}}}, OffsetArrays.OffsetVector{Tuple{Float64, Float64}, Vector{Tuple{Float64, Float64}}}}, Nothing, true, Nothing, UpwindBiased{1, Float64, Nothing, Nothing, Nothing, Nothing, Centered{1, Float64, Nothing, Nothing, Nothing, Nothing}}, Centered{1, Float64, Nothing, Nothing, Nothing, Nothing}}, Centered{2, Float64, Nothing, Nothing, Nothing, Centered{1, Float64, Nothing, Nothing, Nothing, Nothing}}}, Oceananigans.MultiRegion.MultiRegionPoissonSolver{MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, Oceananigans.Solvers.FourierTridiagonalPoissonSolver{RectilinearGrid{Float64, Periodic, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Oceananigans.Solvers.BatchedTridiagonalSolver{Vector{Float64}, Array{Float64, 3}, Vector{Float64}, Array{Float64, 3}, RectilinearGrid{Float64, Periodic, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Nothing}, Array{ComplexF64, 3}, Array{ComplexF64, 3}, Nothing, NamedTuple{(:forward, :backward), Tuple{Tuple{Oceananigans.Solvers.DiscreteTransform{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Oceananigans.Solvers.DiscreteTransform{FFTW.cFFTWPlan{ComplexF64, -1, true, 3, Vector{Int64}}, Oceananigans.Solvers.Forward, RectilinearGrid{Float64, Periodic, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Vector{Int64}, Vector{Periodic}, Int64, Nothing, Nothing}}, Tuple{Oceananigans.Solvers.DiscreteTransform{AbstractFFTs.ScaledPlan{ComplexF64, FFTW.cFFTWPlan{ComplexF64, 1, true, 3, Vector{Int64}}, Float64}, Oceananigans.Solvers.Backward, RectilinearGrid{Float64, Periodic, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Vector{Int64}, Vector{Periodic}, Int64, Nothing, Nothing}, Oceananigans.Solvers.DiscreteTransform{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}}}}, NamedTuple{(:νₑ, :κₑ), Tuple{Field{Center, Center, Center, Nothing, MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, MultiRegionObject{Tuple{Tuple{Colon, Colon, Colon}, Tuple{Colon, Colon, Colon}}, Tuple{CPU, CPU}}, MultiRegionObject{Tuple{OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}}, Tuple{CPU, CPU}}, Any, MultiRegionObject{Tuple{FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Tuple{CPU, CPU}}, Nothing, MultiRegionObject{Tuple{Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}, Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}}, Tuple{CPU, CPU}}}, NamedTuple{(:b,), Tuple{Oceananigans.AbstractOperations.BinaryOperation{Center, Center, Center, typeof(/), Field{Center, Center, Center, Nothing, MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, MultiRegionObject{Tuple{Tuple{Colon, Colon, Colon}, Tuple{Colon, Colon, Colon}}, Tuple{CPU, CPU}}, MultiRegionObject{Tuple{OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}}, Tuple{CPU, CPU}}, Any, MultiRegionObject{Tuple{FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Tuple{CPU, CPU}}, Nothing, MultiRegionObject{Tuple{Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}, Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}}, Tuple{CPU, CPU}}}, Float64, typeof(Oceananigans.Operators.identity3), typeof(Oceananigans.Operators.identity4), MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, Float64}}}}}, NamedTuple{(:velocities, :tracers), Tuple{NamedTuple{(:u, :v, :w), Tuple{Oceananigans.Fields.ZeroField{Int64, 3}, Oceananigans.Fields.ZeroField{Int64, 3}, Oceananigans.Fields.ZeroField{Int64, 3}}}, NamedTuple{(:b,), Tuple{Oceananigans.Fields.ZeroField{Int64, 3}}}}}, Nothing, NamedTuple{(), Tuple{}}}})(name::String)
    @ Oceananigans.OutputWriters ./none:0
  [3] iterate
    @ ./generator.jl:47 [inlined]
  [4] grow_to!(dest::Dict{Any, Any}, itr::Base.Generator{Base.KeySet{String, Dict{String, Oceananigans.Fields.AbstractField{LX, LY, LZ, MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, T, 3} where {LX, LY, LZ, T}}}, Oceananigans.OutputWriters.var"#41#42"{Tuple{Colon, Colon, Colon}, Bool, NonhydrostaticModel{Oceananigans.TimeSteppers.RungeKutta3TimeStepper{Float64, NamedTuple{(:u, :v, :w, :b), Tuple{Field{Face, Center, Center, Nothing, MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, MultiRegionObject{Tuple{Tuple{Colon, Colon, Colon}, Tuple{Colon, Colon, Colon}}, Tuple{CPU, CPU}}, MultiRegionObject{Tuple{OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}}, Tuple{CPU, CPU}}, Any, MultiRegionObject{Tuple{FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Tuple{CPU, CPU}}, Nothing, MultiRegionObject{Tuple{Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}, Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}}, Tuple{CPU, CPU}}}, Field{Center, Face, Center, Nothing, MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, MultiRegionObject{Tuple{Tuple{Colon, Colon, Colon}, Tuple{Colon, Colon, Colon}}, Tuple{CPU, CPU}}, MultiRegionObject{Tuple{OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}}, Tuple{CPU, CPU}}, Any, MultiRegionObject{Tuple{FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Tuple{CPU, CPU}}, Nothing, MultiRegionObject{Tuple{Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}, Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}}, Tuple{CPU, CPU}}}, Field{Center, Center, Face, Nothing, MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, MultiRegionObject{Tuple{Tuple{Colon, Colon, Colon}, Tuple{Colon, Colon, Colon}}, Tuple{CPU, CPU}}, MultiRegionObject{Tuple{OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}}, Tuple{CPU, CPU}}, Any, MultiRegionObject{Tuple{FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Tuple{CPU, CPU}}, Nothing, MultiRegionObject{Tuple{Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}, Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}}, Tuple{CPU, CPU}}}, Field{Center, Center, Center, Nothing, MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, MultiRegionObject{Tuple{Tuple{Colon, Colon, Colon}, Tuple{Colon, Colon, Colon}}, Tuple{CPU, CPU}}, MultiRegionObject{Tuple{OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}}, Tuple{CPU, CPU}}, Any, MultiRegionObject{Tuple{FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Tuple{CPU, CPU}}, Nothing, MultiRegionObject{Tuple{Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}, Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}}, Tuple{CPU, CPU}}}}}, Nothing}, SmagorinskyLilly{Oceananigans.TurbulenceClosures.ExplicitTimeDiscretization, Float64, NamedTuple{(:b,), Tuple{Float64}}}, CPU, MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, Float64, Buoyancy{BuoyancyTracer, Oceananigans.Grids.ZDirection}, FPlane{Float64}, Nothing, NamedTuple{(:u, :v, :w), Tuple{Field{Face, Center, Center, Nothing, MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, MultiRegionObject{Tuple{Tuple{Colon, Colon, Colon}, Tuple{Colon, Colon, Colon}}, Tuple{CPU, CPU}}, MultiRegionObject{Tuple{OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}}, Tuple{CPU, CPU}}, Any, MultiRegionObject{Tuple{FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, ImmersedBoundaryCondition{Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Face, Nothing, Center, Oceananigans.BoundaryConditions.LeftBoundary, typeof(τˣʸ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(Oceananigans.Operators.identity3), typeof(ℑxzᶠᵃᶜ)}}}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Face, Nothing, Center, Oceananigans.BoundaryConditions.RightBoundary, typeof(τˣʸ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(Oceananigans.Operators.identity4), typeof(ℑxzᶠᵃᶜ)}}}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Face, Center, Nothing, Oceananigans.BoundaryConditions.LeftBoundary, typeof(τˣᶻ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(Oceananigans.Operators.identity5), typeof(ℑxyᶠᶜᵃ)}}}, Nothing}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, ImmersedBoundaryCondition{Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Face, Nothing, Center, Oceananigans.BoundaryConditions.LeftBoundary, typeof(τˣʸ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(Oceananigans.Operators.identity3), typeof(ℑxzᶠᵃᶜ)}}}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Face, Nothing, Center, Oceananigans.BoundaryConditions.RightBoundary, typeof(τˣʸ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(Oceananigans.Operators.identity4), typeof(ℑxzᶠᵃᶜ)}}}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Face, Center, Nothing, Oceananigans.BoundaryConditions.LeftBoundary, typeof(τˣᶻ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(Oceananigans.Operators.identity5), typeof(ℑxyᶠᶜᵃ)}}}, Nothing}}}, Tuple{CPU, CPU}}, Nothing, MultiRegionObject{Tuple{Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}, Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}}, Tuple{CPU, CPU}}}, Field{Center, Face, Center, Nothing, MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, MultiRegionObject{Tuple{Tuple{Colon, Colon, Colon}, Tuple{Colon, Colon, Colon}}, Tuple{CPU, CPU}}, MultiRegionObject{Tuple{OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}}, Tuple{CPU, CPU}}, Any, MultiRegionObject{Tuple{FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, ImmersedBoundaryCondition{BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Nothing, Face, Center, Oceananigans.BoundaryConditions.LeftBoundary, typeof(τʸˣ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(Oceananigans.Operators.identity1), typeof(ℑyzᵃᶠᶜ)}}}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Nothing, Face, Center, Oceananigans.BoundaryConditions.RightBoundary, typeof(τʸˣ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(Oceananigans.Operators.identity2), typeof(ℑyzᵃᶠᶜ)}}}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Center, Face, Nothing, Oceananigans.BoundaryConditions.LeftBoundary, typeof(τʸᶻ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(ℑxyᶜᶠᵃ), typeof(Oceananigans.Operators.identity3)}}}, Nothing}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, ImmersedBoundaryCondition{BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Nothing, Face, Center, Oceananigans.BoundaryConditions.LeftBoundary, typeof(τʸˣ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(Oceananigans.Operators.identity1), typeof(ℑyzᵃᶠᶜ)}}}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Nothing, Face, Center, Oceananigans.BoundaryConditions.RightBoundary, typeof(τʸˣ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(Oceananigans.Operators.identity2), typeof(ℑyzᵃᶠᶜ)}}}, Nothing, Nothing, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Center, Face, Nothing, Oceananigans.BoundaryConditions.LeftBoundary, typeof(τʸᶻ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(ℑxyᶜᶠᵃ), typeof(Oceananigans.Operators.identity3)}}}, Nothing}}}, Tuple{CPU, CPU}}, Nothing, MultiRegionObject{Tuple{Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}, Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}}, Tuple{CPU, CPU}}}, Field{Center, Center, Face, Nothing, MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, MultiRegionObject{Tuple{Tuple{Colon, Colon, Colon}, Tuple{Colon, Colon, Colon}}, Tuple{CPU, CPU}}, MultiRegionObject{Tuple{OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}}, Tuple{CPU, CPU}}, Any, MultiRegionObject{Tuple{FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open, Nothing}, ImmersedBoundaryCondition{BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Nothing, Center, Face, Oceananigans.BoundaryConditions.LeftBoundary, typeof(τᶻˣ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(ℑyzᵃᶜᶠ), typeof(Oceananigans.Operators.identity4)}}}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Nothing, Center, Face, Oceananigans.BoundaryConditions.RightBoundary, typeof(τᶻˣ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(ℑyzᵃᶜᶠ), typeof(Oceananigans.Operators.identity5)}}}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Center, Nothing, Face, Oceananigans.BoundaryConditions.LeftBoundary, typeof(τᶻʸ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(ℑxzᶜᵃᶠ), typeof(Oceananigans.Operators.identity1)}}}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Center, Nothing, Face, Oceananigans.BoundaryConditions.RightBoundary, typeof(τᶻʸ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(ℑxzᶜᵃᶠ), typeof(Oceananigans.Operators.identity2)}}}, Nothing, Nothing}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Open, Nothing}, ImmersedBoundaryCondition{BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Nothing, Center, Face, Oceananigans.BoundaryConditions.LeftBoundary, typeof(τᶻˣ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(ℑyzᵃᶜᶠ), typeof(Oceananigans.Operators.identity4)}}}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Nothing, Center, Face, Oceananigans.BoundaryConditions.RightBoundary, typeof(τᶻˣ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(ℑyzᵃᶜᶠ), typeof(Oceananigans.Operators.identity5)}}}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Center, Nothing, Face, Oceananigans.BoundaryConditions.LeftBoundary, typeof(τᶻʸ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(ℑxzᶜᵃᶠ), typeof(Oceananigans.Operators.identity1)}}}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Oceananigans.BoundaryConditions.ContinuousBoundaryFunction{Center, Nothing, Face, Oceananigans.BoundaryConditions.RightBoundary, typeof(τᶻʸ_drag), Nothing, Tuple{Symbol, Symbol}, Tuple{Int64, Int64}, Tuple{typeof(ℑxzᶜᵃᶠ), typeof(Oceananigans.Operators.identity2)}}}, Nothing, Nothing}}}, Tuple{CPU, CPU}}, Nothing, MultiRegionObject{Tuple{Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}, Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}}, Tuple{CPU, CPU}}}}}, NamedTuple{(:b,), Tuple{Field{Center, Center, Center, Nothing, MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, MultiRegionObject{Tuple{Tuple{Colon, Colon, Colon}, Tuple{Colon, Colon, Colon}}, Tuple{CPU, CPU}}, MultiRegionObject{Tuple{OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}}, Tuple{CPU, CPU}}, Any, MultiRegionObject{Tuple{FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Gradient, Float64}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Gradient, Float64}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Tuple{CPU, CPU}}, Nothing, MultiRegionObject{Tuple{Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}, Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}}, Tuple{CPU, CPU}}}}}, NamedTuple{(:pHY′, :pNHS), Tuple{Field{Center, Center, Center, Nothing, MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, MultiRegionObject{Tuple{Tuple{Colon, Colon, Colon}, Tuple{Colon, Colon, Colon}}, Tuple{CPU, CPU}}, MultiRegionObject{Tuple{OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}}, Tuple{CPU, CPU}}, Any, MultiRegionObject{Tuple{FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Tuple{CPU, CPU}}, Nothing, MultiRegionObject{Tuple{Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}, Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}}, Tuple{CPU, CPU}}}, Field{Center, Center, Center, Nothing, MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, MultiRegionObject{Tuple{Tuple{Colon, Colon, Colon}, Tuple{Colon, Colon, Colon}}, Tuple{CPU, CPU}}, MultiRegionObject{Tuple{OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}}, Tuple{CPU, CPU}}, Any, MultiRegionObject{Tuple{FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Tuple{CPU, CPU}}, Nothing, MultiRegionObject{Tuple{Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}, Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}}, Tuple{CPU, CPU}}}}}, NamedTuple{(:u, :v, :w, :b), Tuple{Oceananigans.Forcings.ContinuousForcing{Face, Center, Center, NamedTuple{(:H, :L, :σ, :sp_length_y, :V∞, :f₀, :N²∞), NTuple{7, Float64}}, typeof(forc_u), Tuple{Symbol}, Tuple{Int64}, Tuple{typeof(Oceananigans.Operators.identity5)}}, Oceananigans.Forcings.ContinuousForcing{Center, Face, Center, NamedTuple{(:H, :L, :σ, :sp_length_y, :V∞, :f₀, :N²∞), NTuple{7, Float64}}, typeof(forc_v), Tuple{Symbol}, Tuple{Int64}, Tuple{typeof(Oceananigans.Operators.identity1)}}, Oceananigans.Forcings.ContinuousForcing{Center, Center, Face, NamedTuple{(:H, :L, :σ, :sp_length_y, :V∞, :f₀, :N²∞), NTuple{7, Float64}}, typeof(forc_w), Tuple{Symbol}, Tuple{Int64}, Tuple{typeof(Oceananigans.Operators.identity2)}}, Oceananigans.Forcings.ContinuousForcing{Center, Center, Center, NamedTuple{(:H, :L, :σ, :sp_length_y, :V∞, :f₀, :N²∞), NTuple{7, Float64}}, typeof(forc_b), Tuple{Symbol}, Tuple{Int64}, Tuple{typeof(Oceananigans.Operators.identity3)}}}}, WENO{3, Float64, Nothing, Nothing, NTuple{4, OffsetArrays.OffsetVector{Tuple{Float64, Float64, Float64}, Vector{Tuple{Float64, Float64, Float64}}}}, Nothing, true, Nothing, WENO{2, Float64, Nothing, Nothing, Tuple{OffsetArrays.OffsetVector{Tuple{Float64, Float64}, Vector{Tuple{Float64, Float64}}}, OffsetArrays.OffsetVector{Tuple{Float64, Float64}, Vector{Tuple{Float64, Float64}}}, OffsetArrays.OffsetVector{Tuple{Float64, Float64}, Vector{Tuple{Float64, Float64}}}}, Nothing, true, Nothing, UpwindBiased{1, Float64, Nothing, Nothing, Nothing, Nothing, Centered{1, Float64, Nothing, Nothing, Nothing, Nothing}}, Centered{1, Float64, Nothing, Nothing, Nothing, Nothing}}, Centered{2, Float64, Nothing, Nothing, Nothing, Centered{1, Float64, Nothing, Nothing, Nothing, Nothing}}}, Oceananigans.MultiRegion.MultiRegionPoissonSolver{MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, Oceananigans.Solvers.FourierTridiagonalPoissonSolver{RectilinearGrid{Float64, Periodic, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Oceananigans.Solvers.BatchedTridiagonalSolver{Vector{Float64}, Array{Float64, 3}, Vector{Float64}, Array{Float64, 3}, RectilinearGrid{Float64, Periodic, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Nothing}, Array{ComplexF64, 3}, Array{ComplexF64, 3}, Nothing, NamedTuple{(:forward, :backward), Tuple{Tuple{Oceananigans.Solvers.DiscreteTransform{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}, Oceananigans.Solvers.DiscreteTransform{FFTW.cFFTWPlan{ComplexF64, -1, true, 3, Vector{Int64}}, Oceananigans.Solvers.Forward, RectilinearGrid{Float64, Periodic, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Vector{Int64}, Vector{Periodic}, Int64, Nothing, Nothing}}, Tuple{Oceananigans.Solvers.DiscreteTransform{AbstractFFTs.ScaledPlan{ComplexF64, FFTW.cFFTWPlan{ComplexF64, 1, true, 3, Vector{Int64}}, Float64}, Oceananigans.Solvers.Backward, RectilinearGrid{Float64, Periodic, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, Vector{Int64}, Vector{Periodic}, Int64, Nothing, Nothing}, Oceananigans.Solvers.DiscreteTransform{Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing, Nothing}}}}}}, NamedTuple{(:νₑ, :κₑ), Tuple{Field{Center, Center, Center, Nothing, MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, MultiRegionObject{Tuple{Tuple{Colon, Colon, Colon}, Tuple{Colon, Colon, Colon}}, Tuple{CPU, CPU}}, MultiRegionObject{Tuple{OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}}, Tuple{CPU, CPU}}, Any, MultiRegionObject{Tuple{FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Tuple{CPU, CPU}}, Nothing, MultiRegionObject{Tuple{Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}, Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}}, Tuple{CPU, CPU}}}, NamedTuple{(:b,), Tuple{Oceananigans.AbstractOperations.BinaryOperation{Center, Center, Center, typeof(/), Field{Center, Center, Center, Nothing, MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, MultiRegionObject{Tuple{Tuple{Colon, Colon, Colon}, Tuple{Colon, Colon, Colon}}, Tuple{CPU, CPU}}, MultiRegionObject{Tuple{OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}, OffsetArrays.OffsetArray{Float64, 3, Array{Float64, 3}}}, Tuple{CPU, CPU}}, Any, MultiRegionObject{Tuple{FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}, FieldBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Communication, Oceananigans.MultiRegion.Connectivity}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Flux, Nothing}}}, Tuple{CPU, CPU}}, Nothing, MultiRegionObject{Tuple{Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}, Oceananigans.Fields.FieldBoundaryBuffers{NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, NamedTuple{(:send, :recv), Tuple{Array{Float64, 3}, Array{Float64, 3}}}, Nothing, Nothing}}, Tuple{CPU, CPU}}}, Float64, typeof(Oceananigans.Operators.identity3), typeof(Oceananigans.Operators.identity4), MultiRegionGrid{Float64, Periodic, Periodic, Bounded, XPartition{Int64}, MultiRegionObject{Tuple{RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}, RectilinearGrid{Float64, FullyConnected, Periodic, Bounded, Float64, Float64, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, OffsetArrays.OffsetVector{Float64, Vector{Float64}}, CPU}}, Tuple{CPU, CPU}}, Tuple{CPU, CPU}, CPU}, Float64}}}}}, NamedTuple{(:velocities, :tracers), Tuple{NamedTuple{(:u, :v, :w), Tuple{Oceananigans.Fields.ZeroField{Int64, 3}, Oceananigans.Fields.ZeroField{Int64, 3}, Oceananigans.Fields.ZeroField{Int64, 3}}}, NamedTuple{(:b,), Tuple{Oceananigans.Fields.ZeroField{Int64, 3}}}}}, Nothing, NamedTuple{(), Tuple{}}}}})

Let me know if you need a MWE! (I'm assuming you can figure it out from the error alone)

simone-silvestri · 2022-11-04T19:11:04Z

It is like that by design if you pass an integer to Xpartition. For non equal partitioning you can pass a vector with the required Nx. For example in your case XPartition([2, 3]). I would be weary to use this option though because there might be something missing

…ananigans.jl into ss/multi-region-nonhydrostatic

tomchor · 2022-11-10T22:41:05Z

@simone-silvestri You seem to be right. On the cluster I get:

julia> using Oceananigans.Architectures: DEVICE_STREAMS, DEVICE_FREE_STREAMS
[NVBLAS] NVBLAS_CONFIG_FILE environment variable is NOT set : relying on default config filename 'nvblas.conf'
[NVBLAS] Cannot open default config file 'nvblas.conf'
[NVBLAS] Config parsed
[NVBLAS] CPU Blas library need to be provided

julia> length(DEVICE_STREAMS)
1

julia> DEVICE_FREE_STREAMS
(CUDA.CuStream[],)

julia> CUDA.dev^C

julia> using CUDA

julia> CUDA.devices()
CUDA.DeviceIterator() for 2 devices:
0. Tesla V100-SXM2-32GB
1. Tesla V100-SXM2-32GB

julia> CUDA.ndevices()
2

If you need to reproduce this and aren't able to, we can set up a PP session via zoom and I can share my screen.

simone-silvestri · 2022-11-10T22:44:41Z

ok, then weirdly enough length(CUDA.devices()) has a different behaviour on different clusters. I think the problem should be solved; if not, a zoom session might be useful

navidcy · 2022-11-12T08:20:34Z

cc @mmr0

multiple gpus! (Within a node)

the feature you’ve been waiting for!

mmr0 · 2022-11-14T23:43:50Z

cc @mmr0

multiple gpus! (Within a node)

the feature you’ve been waiting for!

Wheee that's super exciting! Sounds like its time to science ;)

hdrake · 2022-12-28T16:17:06Z

If I want to test out multi-region nonhydrostatic simulations on a GPU cluster with immersed boundaries, is my best bet to start from validation/multi_region/multi_region_les.jl in this PR and just add something like

grid_with_bump = ImmersedBoundaryGrid(grid, GridFittedBottom(bump))
mrg_with_bump  = MultiRegionGrid(grid_with_bump, partition=XPartition(2), devices=(0, 1))

from the hydrostatic multi-region internal tide validation to include immersed boundaries?

Like @mmr0, this is pretty much what I've been waiting for to go all in on Oceananigans 💯

tomchor · 2022-12-29T12:32:10Z

If I want to test out multi-region nonhydrostatic simulations on a GPU cluster with immersed boundaries, is my best bet to start from validation/multi_region/multi_region_les.jl in this PR and just add something like
grid_with_bump = ImmersedBoundaryGrid(grid, GridFittedBottom(bump))
mrg_with_bump  = MultiRegionGrid(grid_with_bump, partition=XPartition(2), devices=(0, 1))
from the hydrostatic multi-region internal tide validation to include immersed boundaries?

Like @mmr0, this is pretty much what I've been waiting for to go all in on Oceananigans 100

Yes, that's how I was able to get this working for me. One note is that grid.Nx must be even atm.

Another note is that so far no OutputWriter works with this, so as far as I can tell the functionality here is very limited.

@glwagner @simone-silvestri I tried to merge main here but couldn't really figure out the proper way to do it. Could any of you do it? I wanna try to at least get NetCDFWriter working so that I can write results with this PR and analyze them!

glwagner · 2022-12-30T01:44:45Z

Another note is that so far no OutputWriter works with this, so as far as I can tell the functionality here is very limited.

Does the JLD2OutputWriter work?

tomchor · 2022-12-31T01:15:29Z

Another note is that so far no OutputWriter works with this, so as far as I can tell the functionality here is very limited.

Does the JLD2OutputWriter work?

I don't think so. At least I wasn't able to make it work

navidcy · 2023-04-19T09:56:06Z

Is this stale?

jagoosw · 2024-11-12T15:13:25Z

How much work would it be to resurrect this PR?

It's hard to tell but it looks like the main changes are the addition of the pressure solvers for the MRG (e.g. src/MultiRegion/multi_region_fft_based_solver.jl), should it be relatively easy for me to pull them out of this branch and get them working?

simone-silvestri · 2024-11-12T16:37:30Z

Heh, I am not sure. This PR is super stale, a lot of what is implemented here were corrections for MultiRegion that might already be in the source code. If you want to take a stab at it I would start from scratch, keeping the transpose implementation from here

glwagner · 2024-11-12T18:48:48Z

Heh, I am not sure. This PR is super stale, a lot of what is implemented here were corrections for MultiRegion that might already be in the source code. If you want to take a stab at it I would start from scratch, keeping the transpose implementation from here

Is there a point to it?

jagoosw · 2024-11-12T19:23:48Z

Okay sounds good, I thought some of it might have already made it in through different PRs.

If I have time I'll have a go at it.

glwagner · 2024-11-12T20:33:00Z

Okay sounds good, I thought some of it might have already made it in through different PRs.

If I have time I'll have a go at it.

What's the goal?

jagoosw · 2024-11-12T20:48:43Z

It would let us run nonhydrostatic models on multiple GPUs right? So we could run even bigger modes

glwagner · 2024-11-12T20:57:10Z

It would let us run nonhydrostatic models on multiple GPUs right? So we could run even bigger modes

You can already use NonhydrostaticModel with Distributed(GPU()). It is distributed with MPI.

glwagner · 2024-11-12T20:59:31Z

the two key PRs are

#3279

and

#3689

This was tested on up to 4096 GPUs.

Rather than multiregion I would focus on performance optimization of the MPI-distributed implementation.

jagoosw · 2024-11-12T21:02:38Z

Oh I didn't realise that! That seems like exactly what I want I'll try using that

glwagner · 2024-11-12T21:20:00Z

I think the scaling efficiency is rather poor. Maybe you can open a discussion to report what you find.

In addition to improving the scaling efficiency, I think work also needs to go into distributed output + FieldTimeSeries.

tomchor · 2024-11-12T21:38:06Z

I think the scaling efficiency is rather poor. Maybe you can open a discussion to report what you find.

I'm in favor of a discussion for that!

In addition to improving the scaling efficiency, I think work also needs to go into distributed output + FieldTimeSeries.

Quick question: is NetCDF output for distributed nonhydrostatic models working?

glwagner · 2024-11-13T22:50:53Z

I think the scaling efficiency is rather poor. Maybe you can open a discussion to report what you find.

I'm in favor of a discussion for that!

In addition to improving the scaling efficiency, I think work also needs to go into distributed output + FieldTimeSeries.

Quick question: is NetCDF output for distributed nonhydrostatic models working?

I don't see why it wouldn't work, but like JLD2 you have to output into separate files per rank.

PS I think its important to get FieldTimeSeries working for netcdf if we want to consider it as first class supported.

ali-ramadhan · 2024-11-13T23:50:45Z

PS I think its important to get FieldTimeSeries working for netcdf if we want to consider it as first class supported.

This is getting higher on my "want TODO" list! First #3143 then #2652...

glwagner · 2024-11-14T01:52:49Z

PS I think its important to get FieldTimeSeries working for netcdf if we want to consider it as first class supported.

This is getting higher on my "want TODO" list! First #3143 then #2652...

Technically, we already support some types of NetCDF at ClimaOcean since we have backends for JRA55 and ECCO! So it is probably not very hard to get this working for Oceananigans output.

simone-silvestri added 13 commits November 1, 2022 15:37

Multiregion for nonhydrostatic

3d08c1d

fix set!

3ab7a6a

everything working!

7e28dde

fixed abstractoperations

32b5dd0

removed old immersed boundaries

9b3dd19

removed old immersed boundary

3a56e8c

bugfix

c92b974

s

1a2d24d

removed vestigial immersed boundaries

95272ba

bugfix

2618af0

other bugfix

ca96e1b

back to inflate grid

8e21854

all tests pass

6f58afd

simone-silvestri changed the title ~~MultiRegion adaption of the NonhydrostaticModel~~ MultiRegion adaptation of the NonhydrostaticModel Nov 2, 2022

simone-silvestri added 2 commits November 2, 2022 09:54

add benchmarks

73a4444

last bugfix

c17e020

last bugfix

052e2c7

solve bounded grid

8d252ed

simone-silvestri added 2 commits November 4, 2022 10:48

solved abstract operations

5b38f2c

fixing abstract operations

3a0fc87

adding transpose

c0d2324

Merge branch 'ss/multi-region-nonhydrostatic' of github.com:CliMA/Oce…

082c78f

…ananigans.jl into ss/multi-region-nonhydrostatic

beautification

b2f2a2b

navidcy added feature 🌟 Something new and shiny GPU 👾 Where Oceananigans gets its powers from distributed 🕸️ Our plan for total cluster domination labels Nov 16, 2022

tomchor mentioned this pull request Feb 6, 2023

Removes binary operation from Smagorinsky diffusivity calculation #2904

Closed

navidcy mentioned this pull request Jul 1, 2023

[WIP] Extend MultiRegion to NonhydrostaticModel #2523

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`MultiRegion` adaptation of the `NonhydrostaticModel` #2795

`MultiRegion` adaptation of the `NonhydrostaticModel` #2795

simone-silvestri commented Nov 2, 2022

tomchor commented Nov 2, 2022

tomchor commented Nov 3, 2022

glwagner commented Nov 3, 2022 •

edited

Loading

tomchor commented Nov 4, 2022 •

edited

Loading

tomchor commented Nov 4, 2022

simone-silvestri commented Nov 4, 2022

tomchor commented Nov 4, 2022

tomchor commented Nov 4, 2022 •

edited

Loading

simone-silvestri commented Nov 4, 2022

tomchor commented Nov 10, 2022 •

edited

Loading

simone-silvestri commented Nov 10, 2022

navidcy commented Nov 12, 2022

mmr0 commented Nov 14, 2022

hdrake commented Dec 28, 2022 •

edited

Loading

tomchor commented Dec 29, 2022

glwagner commented Dec 30, 2022

tomchor commented Dec 31, 2022

navidcy commented Apr 19, 2023

jagoosw commented Nov 12, 2024

simone-silvestri commented Nov 12, 2024

glwagner commented Nov 12, 2024

jagoosw commented Nov 12, 2024

glwagner commented Nov 12, 2024

jagoosw commented Nov 12, 2024

glwagner commented Nov 12, 2024

glwagner commented Nov 12, 2024

jagoosw commented Nov 12, 2024

glwagner commented Nov 12, 2024

tomchor commented Nov 12, 2024

glwagner commented Nov 13, 2024

ali-ramadhan commented Nov 13, 2024

glwagner commented Nov 14, 2024

MultiRegion adaptation of the NonhydrostaticModel #2795

Are you sure you want to change the base?

MultiRegion adaptation of the NonhydrostaticModel #2795

Conversation

simone-silvestri commented Nov 2, 2022

tomchor commented Nov 2, 2022

tomchor commented Nov 3, 2022

glwagner commented Nov 3, 2022 • edited Loading

tomchor commented Nov 4, 2022 • edited Loading

tomchor commented Nov 4, 2022

simone-silvestri commented Nov 4, 2022

tomchor commented Nov 4, 2022

tomchor commented Nov 4, 2022 • edited Loading

simone-silvestri commented Nov 4, 2022

tomchor commented Nov 10, 2022 • edited Loading

simone-silvestri commented Nov 10, 2022

navidcy commented Nov 12, 2022

mmr0 commented Nov 14, 2022

hdrake commented Dec 28, 2022 • edited Loading

tomchor commented Dec 29, 2022

glwagner commented Dec 30, 2022

tomchor commented Dec 31, 2022

navidcy commented Apr 19, 2023

jagoosw commented Nov 12, 2024

simone-silvestri commented Nov 12, 2024

glwagner commented Nov 12, 2024

jagoosw commented Nov 12, 2024

glwagner commented Nov 12, 2024

jagoosw commented Nov 12, 2024

glwagner commented Nov 12, 2024

glwagner commented Nov 12, 2024

jagoosw commented Nov 12, 2024

glwagner commented Nov 12, 2024

tomchor commented Nov 12, 2024

glwagner commented Nov 13, 2024

ali-ramadhan commented Nov 13, 2024

glwagner commented Nov 14, 2024

`MultiRegion` adaptation of the `NonhydrostaticModel` #2795

`MultiRegion` adaptation of the `NonhydrostaticModel` #2795

glwagner commented Nov 3, 2022 •

edited

Loading

tomchor commented Nov 4, 2022 •

edited

Loading

tomchor commented Nov 4, 2022 •

edited

Loading

tomchor commented Nov 10, 2022 •

edited

Loading

hdrake commented Dec 28, 2022 •

edited

Loading