O1.2.1 Make ClimaAtmos GPU compatible #1980
Comments
|
Looks good to me, thanks Sriharsha! Maybe we should move As a side note, Insolation used to only work with Float64. It seems that problem has been solved, but we still use Float64 in ClimaAtmos here. We need to change it and check if it still works. |
|
I am flexible on that. The reason for placing this here is that |
|
Yes, when we use Float32 in ClimaAtmos it is converted to Float64 when calculating radiation. I forgot where this is done, perhaps inside RRTMGP? Anyway, if this task doesn't take long, I think it's ok to have it here. |
|
The next thing that is breaking in the moist GPU case (after #2180 is merged) is this line: @. ᶠρK_E[colidx] =
ᶠinterp(Y.c.ρ[colidx]) * eddy_diffusivity_coefficient(
C_E,
norm(interior_uₕ[colidx]),
interior_coordinates.z[colidx] - z_surface,
ᶠp[colidx],
)I imagine that the issue is one of 3 things:
The first is easiest to fix but seems unlikely. I think we fixed the second one. |
|
I think #2190 fixes the issue with @. ᶠρK_E[colidx] =
ᶠinterp(Y.c.ρ[colidx]) * eddy_diffusivity_coefficient(
C_E,
norm(interior_uₕ[colidx]),
interior_coordinates.z[colidx] - z_surface,
ᶠp[colidx],
)above. The next breaking change, seen in build https://buildkite.com/clima/climaatmos-ci/builds/13566#018afcbf-a78e-49da-b017-18d0c705c4fa is: ClimaAtmos.jl/src/prognostic_equations/vertical_diffusion_boundary_layer.jl Lines 112 to 117 in 3a80801 and the error is: ERROR: LoadError: GPU compilation of MethodInstance for copyto_stencil_kernel!(::Field{} (trunc disp), ::Broadcasted{CUDAColumnStencilStyle, PlaceholderSpace, typeof(rsub), Tuple{Field{} (trunc disp), StencilBroadcasted{CUDAColumnStencilStyle, DivergenceF2C{NamedTuple{(:top, :bottom), Tuple{SetValue{Covariant3Vector{Float32}}, SetValue{Field{} (trunc disp)}}}}, Tuple{Broadcasted{CUDAColumnStencilStyle, FacePlaceholderSpace, typeof(rsub), Tuple{Broadcasted{CUDAColumnStencilStyle, PlaceholderSpace, typeof(rmul), Tuple{Field{} (trunc disp), StencilBroadcasted{CUDAColumnStencilStyle, GradientC2F{NamedTuple{(), Tuple{}}}, Tuple{Field{} (trunc disp)}, PlaceholderSpace}}}}}}, PlaceholderSpace}}}, ::ExtrudedFiniteDifferenceSpace{CellCenter, SpectralElementSpace2D{Nothing, Quadratures.GLL{4}, SphericalGlobalGeometry{Float32}, IJFH{LocalGeometry{(1, 2), LatLongPoint{Float32}, Float32, SMatrix{2, 2, Float32, 4}}, 4, CuDeviceArray{Float32, 4, 1}}, IJFH{Float32, 4, CuDeviceArray{Float32, 4, 1}}, IFH{SurfaceGeometry{Float32, UVVector{Float32}}, 4, CuDeviceArray{Float32, 3, 1}}, NamedTuple{(), Tuple{}}}, Topologies.DeviceIntervalTopology{NamedTuple{(:bottom, :top), Tuple{Int64, Int64}}}, Flat, SphericalGlobalGeometry{Float32}, VIJFH{LocalGeometry{(1, 2, 3), LatLongZPoint{Float32}, Float32, SMatrix{3, 3, Float32, 9}}, 4, CuDeviceArray{Float32, 5, 1}}, VIJFH{LocalGeometry{(1, 2, 3), LatLongZPoint{Float32}, Float32, SMatrix{3, 3, Float32, 9}}, 4, CuDeviceArray{Float32, 5, 1}}}, ::NTuple{4, Int64}, ::Int64, ::Int64, ::Int64) failed
KernelError: passing and using non-bitstype argument
Argument 3 to your kernel function is of type Broadcasted{CUDAColumnStencilStyle, PlaceholderSpace, typeof(rsub), Tuple{Field{} (trunc disp), StencilBroadcasted{CUDAColumnStencilStyle, DivergenceF2C{NamedTuple{(:top, :bottom), Tuple{SetValue{Covariant3Vector{Float32}}, SetValue{Field{} (trunc disp)}}}}, Tuple{Broadcasted{CUDAColumnStencilStyle, FacePlaceholderSpace, typeof(rsub), Tuple{Broadcasted{CUDAColumnStencilStyle, PlaceholderSpace, typeof(rmul), Tuple{Field{} (trunc disp), StencilBroadcasted{CUDAColumnStencilStyle, GradientC2F{NamedTuple{(), Tuple{}}}, Tuple{Field{} (trunc disp)}, PlaceholderSpace}}}}}}, PlaceholderSpace}}}, which is not isbits:
.args is of type Tuple{Field{} (trunc disp), StencilBroadcasted{CUDAColumnStencilStyle, DivergenceF2C{NamedTuple{(:top, :bottom), Tuple{SetValue{Covariant3Vector{Float32}}, SetValue{Field{} (trunc disp)}}}}, Tuple{Broadcasted{CUDAColumnStencilStyle, FacePlaceholderSpace, typeof(rsub), Tuple{Broadcasted{CUDAColumnStencilStyle, PlaceholderSpace, typeof(rmul), Tuple{Field{} (trunc disp), StencilBroadcasted{CUDAColumnStencilStyle, GradientC2F{NamedTuple{(), Tuple{}}}, Tuple{Field{} (trunc disp)}, PlaceholderSpace}}}}}}, PlaceholderSpace}} which is not isbits.
.2 is of type StencilBroadcasted{CUDAColumnStencilStyle, DivergenceF2C{NamedTuple{(:top, :bottom), Tuple{SetValue{Covariant3Vector{Float32}}, SetValue{Field{} (trunc disp)}}}}, Tuple{Broadcasted{CUDAColumnStencilStyle, FacePlaceholderSpace, typeof(rsub), Tuple{Broadcasted{CUDAColumnStencilStyle, PlaceholderSpace, typeof(rmul), Tuple{Field{} (trunc disp), StencilBroadcasted{CUDAColumnStencilStyle, GradientC2F{NamedTuple{(), Tuple{}}}, Tuple{Field{} (trunc disp)}, PlaceholderSpace}}}}}}, PlaceholderSpace} which is not isbits.
.op is of type DivergenceF2C{NamedTuple{(:top, :bottom), Tuple{SetValue{Covariant3Vector{Float32}}, SetValue{Field{} (trunc disp)}}}} which is not isbits.
.bcs is of type NamedTuple{(:top, :bottom), Tuple{SetValue{Covariant3Vector{Float32}}, SetValue{Field{} (trunc disp)}}} which is not isbits.
.bottom is of type SetValue{Field{} (trunc disp)} which is not isbits.
.val is of type Field{} (trunc disp) which is not isbits.
.values is of type IJFH{Covariant3Vector{Float32}, 4, SubArray{Float32, 4, CuArray{Float32, 4, Mem.DeviceBuffer}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, UnitRange{Int64}, Base.Slice{Base.OneTo{Int64}}}, false}} which is not isbits.
.array is of type SubArray{Float32, 4, CuArray{Float32, 4, Mem.DeviceBuffer}, Tuple{Base.Slice{Base.OneTo{Int64}}, Base.Slice{Base.OneTo{Int64}}, UnitRange{Int64}, Base.Slice{Base.OneTo{Int64}}}, false} which is not isbits.
.parent is of type CuArray{Float32, 4, Mem.DeviceBuffer} which is not isbits.
.storage is of type Union{Nothing, ArrayStorage{Mem.DeviceBuffer}} which is not isbits.
.space is of type SpectralElementSpace2D{Topologies.Topology2D{ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, Meshes.EquiangularCubedSphere{Domains.SphereDomain{Float32}, Meshes.NormalizedBilinearMap}, Vector{CartesianIndex{3}}, Array{Int64, 3}, CuArray{Tuple{Int64, Int64, Int64, Int64, Bool}, 1, Mem.DeviceBuffer}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, CuArray{Tuple{Int64, Int64}, 1, Mem.DeviceBuffer}, CuArray{Int64, 1, Mem.DeviceBuffer}, CuArray{Tuple{Bool, Int64, Int64}, 1, Mem.DeviceBuffer}, CuArray{Int64, 1, Mem.DeviceBuffer}, NamedTuple{(), Tuple{}}, CuArray{Tuple{Int64, Int64}, 1, Mem.DeviceBuffer}}, Quadratures.GLL{4}, SphericalGlobalGeometry{Float32}, IJFH{LocalGeometry{(1, 2, 3), LatLongZPoint{Float32}, Float32, SMatrix{3, 3, Float32, 9}}, 4, SubArray{Float32, 4, CuArray{Float32, 5, Mem.DeviceBuffer}, Tuple{Int64, Vararg{Base.Slice{Base.OneTo{Int64}}, 4}}, true}}, IJFH{Float32, 4, CuArray{Float32, 4, Mem.DeviceBuffer}}, IFH{SurfaceGeometry{Float32, UVVector{Float32}}, 4, CuArray{Float32, 3, Mem.DeviceBuffer}}, NamedTuple{(), Tuple{}}} which is not isbits.
.topology is of type Topologies.Topology2D{ClimaComms.SingletonCommsContext{ClimaComms.CUDADevice}, Meshes.EquiangularCubedSphere{Domains.SphereDomain{Float32}, Meshes.NormalizedBilinearMap}, Vector{CartesianIndex{3}}, Array{Int64, 3}, CuArray{Tuple{Int64, Int64, Int64, Int64, Bool}, 1, Mem.DeviceBuffer}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, CuArray{Tuple{Int64, Int64}, 1, Mem.DeviceBuffer}, CuArray{Int64, 1, Mem.DeviceBuffer}, CuArray{Tuple{Bool, Int64, Int64}, 1, Mem.DeviceBuffer}, CuArray{Int64, 1, Mem.DeviceBuffer}, NamedTuple{(), Tuple{}}, CuArray{Tuple{Int64, Int64}, 1, Mem.DeviceBuffer}} which is not isbits.
.elemorder is of type Vector{CartesianIndex{3}} which is not isbits.
.orderindex is of type Array{Int64, 3} which is not isbits.
.elempid is of type Vector{Int64} which is not isbits.
.neighbor_pids is of type Vector{Int64} which is not isbits.
.send_elem_lidx is of type Vector{Int64} which is not isbits.
.send_elem_lengths is of type Vector{Int64} which is not isbits.
.recv_elem_gidx is of type Vector{Int64} which is not isbits.
.recv_elem_lengths is of type Vector{Int64} which is not isbits.
.interior_faces is of type CuArray{Tuple{Int64, Int64, Int64, Int64, Bool}, 1, Mem.DeviceBuffer} which is not isbits.
.storage is of type Union{Nothing, ArrayStorage{Mem.DeviceBuffer}} which is not isbits.
.ghost_faces is of type Vector{Tuple{Int64, Int64, Int64, Int64, Bool}} which is not isbits.
.local_vertices is of type CuArray{Tuple{Int64, Int64}, 1, Mem.DeviceBuffer} which is not isbits.
.storage is of type Union{Nothing, ArrayStorage{Mem.DeviceBuffer}} which is not isbits.
.local_vertex_offset is of type CuArray{Int64, 1, Mem.DeviceBuffer} which is not isbits.
.storage is of type Union{Nothing, ArrayStorage{Mem.DeviceBuffer}} which is not isbits.
.ghost_vertices is of type CuArray{Tuple{Bool, Int64, Int64}, 1, Mem.DeviceBuffer} which is not isbits.
.storage is of type Union{Nothing, ArrayStorage{Mem.DeviceBuffer}} which is not isbits.
.ghost_vertex_offset is of type CuArray{Int64, 1, Mem.DeviceBuffer} which is not isbits.
.storage is of type Union{Nothing, ArrayStorage{Mem.DeviceBuffer}} which is not isbits.
.local_neighbor_elem is of type Vector{Int64} which is not isbits.
.local_neighbor_elem_offset is of type Vector{Int64} which is not isbits.
.ghost_neighbor_elem is of type Vector{Int64} which is not isbits.
.ghost_neighbor_elem_offset is of type Vector{Int64} which is not isbits.
.internal_elems is of type Vector{Int64} which is not isbits.
.perimeter_elems is of type Vector{Int64} which is not isbits.
.ghost_vertex_gcidx is of type Vector{Int64} which is not isbits.
.ghost_face_gcidx is of type Vector{Int64} which is not isbits.
.comm_vertex_lengths is of type Vector{Int64} which is not isbits.
.comm_face_lengths is of type Vector{Int64} which is not isbits.
.ghost_vertex_neighbor_loc is of type Vector{Int64} which is not isbits.
.ghost_vertex_comm_idx_offset is of type Vector{Int64} which is not isbits.
.repr_ghost_vertex is of type CuArray{Tuple{Int64, Int64}, 1, Mem.DeviceBuffer} which is not isbits.
.storage is of type Union{Nothing, ArrayStorage{Mem.DeviceBuffer}} which is not isbits.
.ghost_face_neighbor_loc is of type Vector{Int64} which is not isbits.
.local_geometry is of type IJFH{LocalGeometry{(1, 2, 3), LatLongZPoint{Float32}, Float32, SMatrix{3, 3, Float32, 9}}, 4, SubArray{Float32, 4, CuArray{Float32, 5, Mem.DeviceBuffer}, Tuple{Int64, Vararg{Base.Slice{Base.OneTo{Int64}}, 4}}, true}} which is not isbits.
.array is of type SubArray{Float32, 4, CuArray{Float32, 5, Mem.DeviceBuffer}, Tuple{Int64, Vararg{Base.Slice{Base.OneTo{Int64}}, 4}}, true} which is not isbits.
.parent is of type CuArray{Float32, 5, Mem.DeviceBuffer} which is not isbits.
.ghost_geometry is of type IJFH{LocalGeometry{(1, 2, 3), LatLongZPoint{Float32}, Float32, SMatrix{3, 3, Float32, 9}}, 4, SubArray{Float32, 4, CuArray{Float32, 5, Mem.DeviceBuffer}, Tuple{Int64, Vararg{Base.Slice{Base.OneTo{Int64}}, 4}}, true}} which is not isbits.
.array is of type SubArray{Float32, 4, CuArray{Float32, 5, Mem.DeviceBuffer}, Tuple{Int64, Vararg{Base.Slice{Base.OneTo{Int64}}, 4}}, true} which is not isbits.
.parent is of type CuArray{Float32, 5, Mem.DeviceBuffer} which is not isbits.
.local_dss_weights is of type IJFH{Float32, 4, CuArray{Float32, 4, Mem.DeviceBuffer}} which is not isbits.
.array is of type CuArray{Float32, 4, Mem.DeviceBuffer} which is not isbits.
.storage is of type Union{Nothing, ArrayStorage{Mem.DeviceBuffer}} which is not isbits.
.ghost_dss_weights is of type IJFH{Float32, 4, CuArray{Float32, 4, Mem.DeviceBuffer}} which is not isbits.
.array is of type CuArray{Float32, 4, Mem.DeviceBuffer} which is not isbits.
.storage is of type Union{Nothing, ArrayStorage{Mem.DeviceBuffer}} which is not isbits.
.internal_surface_geometry is of type IFH{SurfaceGeometry{Float32, UVVector{Float32}}, 4, CuArray{Float32, 3, Mem.DeviceBuffer}} which is not isbits.
.array is of type CuArray{Float32, 3, Mem.DeviceBuffer} which is not isbits.
.storage is of type Union{Nothing, ArrayStorage{Mem.DeviceBuffer}} which is not isbits.
Stacktrace:
[1] check_invocation(job::GPUCompiler.CompilerJob)
... |
szy21
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The Climate Modeling Alliance
Software Design Issue 📜
Purpose
Make
ClimaAtmos.jlpackage GPU compatible.Cost/Benefits/Risks
People and Personnel
Components
Inputs
Results and Deliverables
SDI Revision Log
CC
@simonbyrne @cmbengue @tapios
RRTMGP.jlRRTMGPIcallbacks fail on P100 RRTMGP.jl#400SurfaceFluxes.jlCloudMicroPhysics.jlDiagnostic EDMF
Insolation
instantaneous_zenith_anglefunction Insolation.jl#60ClimaAtmos.jl
Miscellaneous
tomlin the yaml file assumes that the toml is in the working directory #2167AtmosCoveragePerfConfigare surprising #2093The text was updated successfully, but these errors were encountered: