Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update to v0.5.0 #78

Merged
merged 77 commits into from
Jun 8, 2022
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
77 commits
Select commit Hold shift + click to select a range
3db86b6
Update OCCA
noelchalmers Mar 12, 2022
fe568e5
Remove gslib
noelchalmers Mar 12, 2022
d41f047
Upgrade to newest ogs
noelchalmers Mar 12, 2022
72983ba
Add new comm_t object
noelchalmers Mar 12, 2022
4515afb
Add new memory objects
noelchalmers Mar 12, 2022
5898ed1
Add new timers
noelchalmers Mar 12, 2022
28fd90b
Move matrix routines into linAlg
noelchalmers Mar 12, 2022
71e61e5
Rename file
noelchalmers Mar 13, 2022
d7847f2
[Core] Update core lib
noelchalmers Mar 14, 2022
dd776b7
[LinAlg] Update linAlg lib
noelchalmers Mar 14, 2022
67a604c
[LinAlg] Rename some source files
noelchalmers Mar 14, 2022
4c45c96
[Core] Remove ambiguous scan overload
noelchalmers Mar 14, 2022
3297faa
[OCCA] Update to latest OCCA
noelchalmers Apr 26, 2022
228a47e
[Mesh] Move some mesh files
noelchalmers Apr 26, 2022
4c01601
[ParAdogs] Add parAdogs mesh partitioner
noelchalmers Apr 26, 2022
e655d09
[Mesh] Update mesh library
noelchalmers Apr 26, 2022
20c0cb3
[Core] Updates to core library
noelchalmers Apr 26, 2022
e8909c6
[LinAlg] Update linear algebra library
noelchalmers Apr 26, 2022
5c5f1e9
[TimeStepper] Update timeStepper library
noelchalmers Apr 26, 2022
7ffad49
[LinearSolver] Update linearSolver library
noelchalmers Apr 26, 2022
d7bc53a
[ParAlmond] Update parAlmond library
noelchalmers Apr 26, 2022
1941aa8
[Make] Makefile updates
noelchalmers Apr 26, 2022
543b116
[Advection] Update advection solver
noelchalmers Apr 26, 2022
037fdc4
[Acoustics] Update acoustics solver
noelchalmers Apr 26, 2022
cc5564c
[Gradient] Update gradient solver
noelchalmers Apr 26, 2022
08055cf
[Elliptic] Update elliptic solver
noelchalmers Apr 26, 2022
5d2c98b
[CNS] Update compressible navier stokes solver
noelchalmers Apr 26, 2022
8329841
[Test] Some test tweaks
noelchalmers Apr 26, 2022
715ea71
[LBS] Update DG Lattice Boltzmann solver
noelchalmers Apr 27, 2022
dfe568f
[LBS] Makefile tweak
noelchalmers Apr 27, 2022
231ba66
[BNS] Update Galerkin Boltzmann Navier-Stokes solver
noelchalmers Apr 27, 2022
76930d6
[FPE] Update Fokker-Planck solver
noelchalmers Apr 27, 2022
ebc1f2f
[INS] Update incompressible Navier-Stokes solver
noelchalmers Apr 28, 2022
326487c
[LinAlg] LinAlg fixes
noelchalmers Apr 28, 2022
cc2c959
[InitialGuess] FIxes in initial guess strategies
noelchalmers Apr 28, 2022
6e5234f
[Mesh] Initialize mapB to -1
noelchalmers Apr 28, 2022
1f9320b
[Mesh] mapB in mesh_t class
noelchalmers Apr 28, 2022
1f4147b
[Mesh] Bug fix in cubature setup
noelchalmers Apr 28, 2022
85b9a36
[Mesh] Bugifx in multirate setup
noelchalmers Apr 28, 2022
60a0b9f
[ParAlmond] Switch to shared_ptr to address mem leak
noelchalmers Apr 28, 2022
c072c59
[Test] Tweak one golden norm
noelchalmers Apr 28, 2022
3977517
[Test] Add ParAdogs tests
noelchalmers Apr 28, 2022
0415c68
[Solvers] Switch to strings for filenames and kernelname to avoid mem…
noelchalmers Apr 28, 2022
afdbb28
Version bump
noelchalmers Apr 28, 2022
e65f784
Merge branch 'master' of github.com:paranumal/libparanumal into staging
noelchalmers Apr 28, 2022
0c882da
Happy New Year
noelchalmers Apr 29, 2022
9139524
Need some manual barriers
noelchalmers Apr 29, 2022
2c21ada
[Solvers] Trigger ogs kernel builds during setups
noelchalmers Apr 29, 2022
6658636
[Git] Ignore rc files made by tests
noelchalmers May 1, 2022
1e1a4d3
[Make] Fix default openblas-serial lib path
noelchalmers May 1, 2022
3e15c25
[Git] Use openblas-serial in git workflow
noelchalmers May 1, 2022
febc58b
[Comm] Move some things out of the comm_t class, and into a namespace
noelchalmers May 1, 2022
8b4b759
[Mesh] Make the element type an enum
noelchalmers May 1, 2022
1502f87
[Make] Missing code coverage flags
noelchalmers May 1, 2022
d077084
[OGS] Add some more explicit instantiations to workaround issue in ol…
noelchalmers May 1, 2022
078a21c
[Timer] Can't use a the plaform's comm at the end of ogsSetup. Adding…
noelchalmers May 2, 2022
f9f3915
Update README.md
noelchalmers May 3, 2022
ef8555e
Add code diagram
noelchalmers May 5, 2022
341fd70
Merge branch 'staging' of github.com:paranumal/libparanumal into staging
noelchalmers May 5, 2022
86f0318
Update README.md
noelchalmers May 5, 2022
2be74fb
[LinearSolver] Bugfix for uninitialized memory
noelchalmers May 14, 2022
e0b3174
[Libs][Solvers] Small fixes for building in FP32 mode
noelchalmers May 14, 2022
7464423
[Core] Remove repeated compiler flags
noelchalmers May 14, 2022
8f36675
[Core] Workaround for occa dtypes being empty for user-types
noelchalmers May 14, 2022
8dbf988
Merge branch 'staging' of github.com:paranumal/libparanumal into staging
noelchalmers May 14, 2022
5152215
[ParAdogs] Reduce a termination tolerance when in FP32 mode
noelchalmers May 18, 2022
e4b42f0
[ParAdogs] Typo
noelchalmers May 18, 2022
c5b7b14
[OCCA] Switch to OCCA dev branch
noelchalmers May 20, 2022
10ec480
[Core] Add a check to not exceed the max thread count OpenMP reports
noelchalmers May 20, 2022
2f995eb
[OGS] Add some restricts to pointers for host operators
noelchalmers May 20, 2022
abaee13
[OCCA] Fix properties syntax
noelchalmers May 20, 2022
c5a56b8
[OGS] Add special code paths for scalar ogs ops
noelchalmers May 26, 2022
facc0f5
Add some notes on CPU binding to the README
noelchalmers May 26, 2022
672b275
[Acoustics] Hide more comm time with the surface kernel
noelchalmers May 26, 2022
ea5adeb
[OGS] Fix final row block entry being too large
noelchalmers May 31, 2022
f72e8de
Merge branch 'staging' of github.com:paranumal/libparanumal into staging
noelchalmers May 31, 2022
9068ad0
[OCCA] Update to OCCA v1.3
noelchalmers Jun 2, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[Comm] Move some things out of the comm_t class, and into a namespace
  • Loading branch information
noelchalmers committed May 1, 2022
commit febc58bcb9c5f30c71977e0037528e5b9f4d8361
102 changes: 54 additions & 48 deletions include/comm.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,33 @@ TYPE(float, MPI_FLOAT);
TYPE(double, MPI_DOUBLE);
#undef TYPE

class comm_t;

namespace Comm {

using request_t = MPI_Request;

/*Predefined ops*/
using op_t = MPI_Op;
static constexpr op_t Max = MPI_MAX;
static constexpr op_t Min = MPI_MIN;
static constexpr op_t Sum = MPI_SUM;
static constexpr op_t Prod = MPI_PROD;
static constexpr op_t And = MPI_LAND;
static constexpr op_t Or = MPI_LOR;
static constexpr op_t Xor = MPI_LXOR;

/*MPI_Init and MPI_Finalize*/
void Init(int &argc, char** &argv);
void Finalize();

/*handle to MPI_COMM_WORLD*/
comm_t World();

void GetProcessorName(char* name, int &namelen);

} //namespace Comm

/*Communicator class*/
class comm_t {

Expand All @@ -77,13 +104,6 @@ class comm_t {
comm_t(const comm_t &c) = default;
comm_t& operator = (const comm_t &c)=default;

/*Static MPI_Init and MPI_Finalize*/
static void Init(int &argc, char** &argv);
static void Finalize();

/*Static handle to MPI_COMM_WORLD*/
static comm_t world();

/*MPI_Comm_dup and MPI_Comm_delete*/
comm_t Dup() const;
comm_t Split(const int color, const int key) const;
Expand All @@ -96,18 +116,6 @@ class comm_t {
/*MPI_Comm getter*/
MPI_Comm comm() const;

using request_t = MPI_Request;

/*Predefined ops*/
using op_t = MPI_Op;
static constexpr op_t Max = MPI_MAX;
static constexpr op_t Min = MPI_MIN;
static constexpr op_t Sum = MPI_SUM;
static constexpr op_t Prod = MPI_PROD;
static constexpr op_t And = MPI_LAND;
static constexpr op_t Or = MPI_LOR;
static constexpr op_t Xor = MPI_LXOR;

/*libp::memory send*/
template <template<typename> class mem, typename T>
void Send(mem<T> m,
Expand Down Expand Up @@ -158,7 +166,7 @@ class comm_t {
const int dest,
const int count,
const int tag,
request_t &request) const {
Comm::request_t &request) const {
MPI_Datatype type = mpiType<T>::getMpiType();
MPI_Isend(m.ptr(), count, type, dest, tag, comm(), &request);
mpiType<T>::freeMpiType(type);
Expand All @@ -170,7 +178,7 @@ class comm_t {
const int source,
const int count,
const int tag,
request_t &request) const {
Comm::request_t &request) const {
MPI_Datatype type = mpiType<T>::getMpiType();
MPI_Irecv(m.ptr(), count, type, source, tag, comm(), &request);
mpiType<T>::freeMpiType(type);
Expand All @@ -181,7 +189,7 @@ class comm_t {
void Isend(T& val,
const int dest,
const int tag,
request_t &request) const {
Comm::request_t &request) const {
MPI_Datatype type = mpiType<T>::getMpiType();
MPI_Isend(&val, 1, type, dest, tag, comm(), &request);
mpiType<T>::freeMpiType(type);
Expand All @@ -192,7 +200,7 @@ class comm_t {
void Irecv(T& val,
const int source,
const int tag,
request_t &request) const {
Comm::request_t &request) const {
MPI_Datatype type = mpiType<T>::getMpiType();
MPI_Irecv(&val, 1, type, source, tag, comm(), &request);
mpiType<T>::freeMpiType(type);
Expand Down Expand Up @@ -223,7 +231,7 @@ class comm_t {
void Reduce(const mem<T> snd,
mem<T> rcv,
const int root,
const op_t op = Sum,
const Comm::op_t op = Comm::Sum,
const int count=-1) const {
MPI_Datatype type = mpiType<T>::getMpiType();
const int cnt = (count==-1) ? static_cast<int>(snd.length()) : count;
Expand All @@ -235,7 +243,7 @@ class comm_t {
template <template<typename> class mem, typename T>
void Reduce(mem<T> m,
const int root,
const op_t op = Sum,
const Comm::op_t op = Comm::Sum,
const int count=-1) const {
MPI_Datatype type = mpiType<T>::getMpiType();
const int cnt = (count==-1) ? static_cast<int>(m.length()) : count;
Expand All @@ -252,15 +260,15 @@ class comm_t {
void Reduce(const T& snd,
T& rcv,
const int root,
const op_t op = Sum) const {
const Comm::op_t op = Comm::Sum) const {
MPI_Datatype type = mpiType<T>::getMpiType();
MPI_Reduce(&snd, &rcv, 1, type, op, root, comm());
mpiType<T>::freeMpiType(type);
}
template <typename T>
void Reduce(T& val,
const int root,
const op_t op = Sum) const {
const Comm::op_t op = Comm::Sum) const {
T rcv=val;
Reduce(val, rcv, root, op);
if (rank()==root) val=rcv;
Expand All @@ -270,7 +278,7 @@ class comm_t {
template <template<typename> class mem, typename T>
void Allreduce(const mem<T> snd,
mem<T> rcv,
const op_t op = Sum,
const Comm::op_t op = Comm::Sum,
const int count=-1) const {
MPI_Datatype type = mpiType<T>::getMpiType();
const int cnt = (count==-1) ? static_cast<int>(snd.length()) : count;
Expand All @@ -281,7 +289,7 @@ class comm_t {
/*libp::memory in-place allreduce*/
template <template<typename> class mem, typename T>
void Allreduce(mem<T> m,
const op_t op = Sum,
const Comm::op_t op = Comm::Sum,
const int count=-1) const {
MPI_Datatype type = mpiType<T>::getMpiType();
const int cnt = (count==-1) ? static_cast<int>(m.length()) : count;
Expand All @@ -293,14 +301,14 @@ class comm_t {
template <typename T>
void Allreduce(const T& snd,
T& rcv,
const op_t op = Sum) const {
const Comm::op_t op = Comm::Sum) const {
MPI_Datatype type = mpiType<T>::getMpiType();
MPI_Allreduce(&snd, &rcv, 1, type, op, comm());
mpiType<T>::freeMpiType(type);
}
template <typename T>
void Allreduce(T& val,
const op_t op = Sum) const {
const Comm::op_t op = Comm::Sum) const {
T rcv=val;
Allreduce(val, rcv, op);
val = rcv;
Expand All @@ -310,9 +318,9 @@ class comm_t {
template <template<typename> class mem, typename T>
void Iallreduce(const mem<T> snd,
mem<T> rcv,
const op_t op,
const Comm::op_t op,
const int count,
request_t &request) const {
Comm::request_t &request) const {
MPI_Datatype type = mpiType<T>::getMpiType();
MPI_Iallreduce(snd.ptr(), rcv.ptr(), count, type, op, comm(), &request);
mpiType<T>::freeMpiType(type);
Expand All @@ -321,9 +329,9 @@ class comm_t {
/*libp::memory non-blocking in-place allreduce*/
template <template<typename> class mem, typename T>
void Iallreduce(mem<T> m,
const op_t op,
const Comm::op_t op,
const int count,
request_t &request) const {
Comm::request_t &request) const {
MPI_Datatype type = mpiType<T>::getMpiType();
MPI_Iallreduce(MPI_IN_PLACE, m.ptr(), count, type, op, comm(), &request);
mpiType<T>::freeMpiType(type);
Expand All @@ -333,17 +341,17 @@ class comm_t {
template <template<typename> class mem, typename T>
void Iallreduce(const T& snd,
T& rcv,
const op_t op,
request_t &request) const {
const Comm::op_t op,
Comm::request_t &request) const {
MPI_Datatype type = mpiType<T>::getMpiType();
MPI_Iallreduce(&snd, &rcv, 1, type, op, comm(), &request);
mpiType<T>::freeMpiType(type);
}
/*scalar non-blocking in-place allreduce*/
template <template<typename> class mem, typename T>
void Iallreduce(T& val,
const op_t op,
request_t &request) const {
const Comm::op_t op,
Comm::request_t &request) const {
MPI_Datatype type = mpiType<T>::getMpiType();
MPI_Iallreduce(MPI_IN_PLACE, &val, 1, type, op, comm(), &request);
mpiType<T>::freeMpiType(type);
Expand All @@ -353,7 +361,7 @@ class comm_t {
template <template<typename> class mem, typename T>
void Scan(const mem<T> snd,
mem<T> rcv,
const op_t op = Sum,
const Comm::op_t op = Comm::Sum,
const int count=-1) const {
MPI_Datatype type = mpiType<T>::getMpiType();
const int cnt = (count==-1) ? static_cast<int>(snd.length()) : count;
Expand All @@ -364,7 +372,7 @@ class comm_t {
/*libp::memory in-place scan*/
template <template<typename> class mem, typename T>
void Scan(mem<T> m,
const op_t op = Sum,
const Comm::op_t op = Comm::Sum,
const int count=-1) const {
MPI_Datatype type = mpiType<T>::getMpiType();
const int cnt = (count==-1) ? static_cast<int>(m.length()) : count;
Expand All @@ -376,7 +384,7 @@ class comm_t {
template <typename T>
void Scan(const T& snd,
T& rcv,
const op_t op = Sum) const {
const Comm::op_t op = Comm::Sum) const {
MPI_Datatype type = mpiType<T>::getMpiType();
MPI_Scan(&snd, &rcv, 1, type, op, comm());
mpiType<T>::freeMpiType(type);
Expand Down Expand Up @@ -537,21 +545,19 @@ class comm_t {
mem<T> rcv,
const memory<int> recvCounts,
const memory<int> recvOffsets,
request_t &request) const {
Comm::request_t &request) const {
MPI_Datatype type = mpiType<T>::getMpiType();
MPI_Ialltoallv(snd.ptr(), sendCounts.ptr(), sendOffsets.ptr(), type,
rcv.ptr(), recvCounts.ptr(), recvOffsets.ptr(), type,
comm(), &request);
mpiType<T>::freeMpiType(type);
}

void Wait(request_t &request) const;
void Waitall(const int count, memory<request_t> &requests) const;
void Wait(Comm::request_t &request) const;
void Waitall(const int count, memory<Comm::request_t> &requests) const;
void Barrier() const;

static void GetProcessorName(char* name, int &namelen) {
MPI_Get_processor_name(name,&namelen);
}
friend comm_t Comm::World();
};

} //namespace libp
Expand Down
4 changes: 2 additions & 2 deletions include/linearSolver.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -174,7 +174,7 @@ class nbpcg: public linearSolverBase_t {
kernel_t update1NBPCGKernel;
kernel_t update2NBPCGKernel;

comm_t::request_t request;
Comm::request_t request;

void Update1NBPCG(const dfloat beta);
void Update2NBPCG(const dfloat alpha, deviceMemory<dfloat>& o_r);
Expand All @@ -199,7 +199,7 @@ class nbfpcg: public linearSolverBase_t {
kernel_t update0NBFPCGKernel;
kernel_t update1NBFPCGKernel;

comm_t::request_t request;
Comm::request_t request;

void Update0NBFPCG(deviceMemory<dfloat>& o_r);
void Update1NBFPCG(const dfloat alpha, deviceMemory<dfloat>& o_x, deviceMemory<dfloat>& o_r);
Expand Down
6 changes: 3 additions & 3 deletions include/ogs/ogsExchange.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@ class ogsAllToAll_t: public ogsExchange_t {
memory<int> sendOffsets;
memory<int> recvOffsets;

comm_t::request_t request;
Comm::request_t request;

public:
ogsAllToAll_t(dlong Nshared,
Expand Down Expand Up @@ -192,7 +192,7 @@ class ogsPairwise_t: public ogsExchange_t {
memory<int> sendOffsetsT;
memory<int> recvOffsetsN;
memory<int> recvOffsetsT;
memory<comm_t::request_t> requests;
memory<Comm::request_t> requests;

public:
ogsPairwise_t(dlong Nshared,
Expand Down Expand Up @@ -268,7 +268,7 @@ class ogsCrystalRouter_t: public ogsExchange_t {
pinnedMemory<char> h_work[2];
deviceMemory<char> o_work[2];

memory<comm_t::request_t> request;
memory<Comm::request_t> request;

int Nlevels=0;
memory<crLevel> levelsN;
Expand Down
18 changes: 13 additions & 5 deletions libs/core/comm.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -28,12 +28,14 @@ SOFTWARE.

namespace libp {

namespace Comm {

/*Static MPI_Init and MPI_Finalize*/
void comm_t::Init(int &argc, char** &argv) { MPI_Init(&argc, &argv); }
void comm_t::Finalize() { MPI_Finalize(); }
void Init(int &argc, char** &argv) { MPI_Init(&argc, &argv); }
void Finalize() { MPI_Finalize(); }

/*Static handle to MPI_COMM_WORLD*/
comm_t comm_t::world() {
comm_t World() {
comm_t c;
c.comm_ptr = std::make_shared<MPI_Comm>();
*(c.comm_ptr) = MPI_COMM_WORLD;
Expand All @@ -42,6 +44,12 @@ comm_t comm_t::world() {
return c;
}

void GetProcessorName(char* name, int &namelen) {
MPI_Get_processor_name(name,&namelen);
}

} //namespace Comm

/*MPI_Comm_dup and free*/
comm_t comm_t::Dup() const {
comm_t c;
Expand Down Expand Up @@ -95,11 +103,11 @@ MPI_Comm comm_t::comm() const {
}
}

void comm_t::Wait(request_t &request) const {
void comm_t::Wait(Comm::request_t &request) const {
MPI_Wait(&request, MPI_STATUS_IGNORE);
}

void comm_t::Waitall(const int count, memory<request_t> &requests) const {
void comm_t::Waitall(const int count, memory<Comm::request_t> &requests) const {
MPI_Waitall(count, requests.ptr(), MPI_STATUSES_IGNORE);
}

Expand Down
2 changes: 1 addition & 1 deletion libs/core/platformDeviceConfig.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ void platform_t::DeviceConfig(){
memory<char> hostname = hostnames + rank()*MAX_PROCESSOR_NAME;

int namelen;
comm_t::GetProcessorName(hostname.ptr(), namelen);
Comm::GetProcessorName(hostname.ptr(), namelen);
comm.Allgather(hostnames, MAX_PROCESSOR_NAME);

int localRank = 0;
Expand Down
Loading