inciter::Transporter class

Transporter drives the time integration of transport equations.

Constructors, destructors, conversion operators

Transporter(CkMigrateMessage* m) explicit
Migrate constructor: returning from a checkpoint.

Public functions

auto Transporter() -> Transporter_SDAG_CODE explicit
Constructor.
void load(std::size_t meshid, std::size_t nelem)
Reduction target: the mesh has been read from file on all PEs.
void partitioned(std::size_t meshid)
Reduction target: a mesh has been partitioned.
void partition()
Reduction target: all Solver (PEs) have computed the number of chares they will recieve contributions from during linear solution.
void distributed(std::size_t meshid)
Reduction target: all compute nodes have distrbuted their mesh after partitioning.
void queriedRef(std::size_t meshid)
Reduction target: all Refiner chares have queried their boundary edges.
void respondedRef(std::size_t meshid)
Reduction target: all Refiner chares have setup their boundary edges.
void refinserted(std::size_t meshid, std::size_t error)
Reduction target: all compute nodes have created the mesh refiners.
void discinserted(std::size_t meshid)
Reduction target: all Discretization chares have been inserted.
void disccreated(std::size_t summeshid, std::size_t npoin)
Reduction target: all Discretization constructors have been called.
void workinserted(std::size_t meshid)
Reduction target: all worker (derived discretization) chares have been inserted.
void compatibility(std::size_t meshid)
Reduction target: all Refiner chares have received a round of edges, and have run their compatibility algorithm.
void matched(std::size_t summeshid, std::size_t nextra, std::size_t nref, std::size_t nderef, std::size_t sumrefmode)
Reduction target: all Refiner chares have matched/corrected the tagging of chare-boundary edges, all chares are ready to perform refinement.
void bndint(tk::real sx, tk::real sy, tk::real sz, tk::real cb, tk::real summeshid)
Compute surface integral across the whole problem and perform leak-test.
void refined(std::size_t summeshid, std::size_t nelem, std::size_t npoin)
Reduction target: all chares have refined their mesh.
void resized(std::size_t meshid)
Reduction target: all worker chares have resized their own data after AMR or ALE.
void startEsup(std::size_t meshid)
Reduction target: all worker chares have generated their own esup.
void queried(std::size_t meshid)
Reduction target: all Sorter chares have queried their boundary edges.
void responded(std::size_t meshid)
Reduction target: all Sorter chares have setup their boundary edges.
void pepartitioned()
Non-reduction target for receiving progress report on partitioning mesh.
void pedistributed()
Non-reduction target for receiving progress report on distributing mesh.
void chbnd()
Non-reduction target for receiving progress report on finding bnd nodes.
void chcomm()
Non-reduction target for receiving progress report on node ID comm map.
void chmask()
Non-reduction target for receiving progress report on node ID mask.
void chreordered()
Non-reduction target for receiving progress report on reordering mesh.
void chcreated()
Non-reduction target for receiving progress report on creating workers.
void chbndface()
Non-reduction target for receiving progress report on finding bnd faces.
void chcomfac()
Non-reduction target for receiving progress report on face communication.
void chghost()
Non-reduction target for receiving progress report on sending ghost data.
void chadj()
Non-reduction target for receiving progress report on face adjacency.
void doneInsertingGhosts(std::size_t meshid)
void comfinal(std::size_t initial, std::size_t summeshid)
Reduction target indicating that the communication maps have been setup.
void totalvol(tk::real v, tk::real initial, tk::real summeshid)
Reduction target summing total mesh volume.
void minstat(tk::real d0, tk::real d1, tk::real d2, tk::real rmeshid)
Reduction target yielding the minimum mesh statistics across all workers.
void maxstat(tk::real d0, tk::real d1, tk::real d2, tk::real rmeshid)
Reduction target yielding the maximum mesh statistics across all workers.
void sumstat(tk::real d0, tk::real d1, tk::real d2, tk::real d3, tk::real d4, tk::real d5, tk::real summeshid)
Reduction target yielding the sum of mesh statistics across all workers.
void pdfstat(CkReductionMsg* msg)
Reduction target yielding PDF of mesh statistics across all workers.
void boxvol(tk::real* meshdata, int n)
Reduction target computing total volume of IC box.
void solutionTransferred()
Reduction target broadcasting to Schemes after mesh transfer.
void minDtAcrossMeshes(tk::real* reducndata, int n)
Reduction target that computes minimum timestep across meshes.
void diagnostics(CkReductionMsg* msg)
Reduction target optionally collecting diagnostics, e.g., residuals, from all worker chares.
void resume()
Resume execution from checkpoint/restart files.
void checkpoint(std::size_t finished, std::size_t meshid)
Save checkpoint/restart files.
void finish(std::size_t meshid = 0)
Normal finish of time stepping.

Charm++ pack/unpack serializer member functions

enum class TimerTag { MESH_READ =0 }
Timer tags.
std::vector<std::string> m_input
List of mesh files to be used for potentially multiple solvers.
std::vector<int> m_nchare
Number of worker chares (one per mesh)
std::unordered_map<std::size_t, std::size_t> m_meshid
Sum of mesh ids (across all chares, key) for each meshid (value)
std::vector<std::size_t> m_ncit
Number of mesh ref corr iter (one per mesh)
std::size_t m_nload
Number of meshes loaded.
std::size_t m_ntrans
Number of meshes that have transferred solution.
std::size_t m_ndtmsh
Number of meshes that have computed their dt.
std::vector<tk::real> m_dtmsh
Minimum dt on each mesh (sized at each time step and then emptied)
std::size_t m_npart
Number of meshes partitioned.
std::size_t m_nstat
Number of mesh statistics computed.
std::size_t m_ndisc
Number of Discretization arrays created.
std::size_t m_nchk
Number of worker arrays checkpointed.
std::size_t m_ncom
Number of worker arrays have finished setting up their comm maps.
std::vector<std::size_t> m_nt0refit
Number of t0ref mesh ref iters (one per mesh)
std::vector<std::size_t> m_ndtrefit
Number of dtref mesh ref iters (one per mesh)
std::vector<std::size_t> m_noutrefit
Number of outref mesh ref iters (one per mesh)
std::vector<std::size_t> m_noutderefit
Number of outderef mesh ref iters (one per mesh)
std::vector<Scheme> m_scheme
Discretization scheme (one per mesh)
std::vector<CProxy_Partitioner> m_partitioner
Partitioner nodegroup proxies (one per mesh)
std::vector<CProxy_Refiner> m_refiner
Mesh refiner array proxies (one per mesh)
std::vector<tk::CProxy_MeshWriter> m_meshwriter
Mesh writer nodegroup proxies (one per mesh)
std::vector<CProxy_Sorter> m_sorter
Mesh sorter array proxy (one per mesh)
std::vector<std::size_t> m_nelem
Number of mesh points (per mesh)
std::vector<std::size_t> m_npoin
Nonzero if finished with timestepping (one per mesh)
std::vector<tk::real> m_meshvol
Total mesh volume (one per mesh)
std::vector<std::array<tk::real, 3>> m_minstat
Minimum mesh statistics (one per mesh)
std::vector<std::array<tk::real, 3>> m_maxstat
Maximum mesh statistics (one per mesh)
std::vector<std::array<tk::real, 3>> m_avgstat
Average mesh statistics (one per mesh)
std::map<TimerTag, tk::Timer> m_timer
Timers.
tk::Progress<7> m_progMesh
Progress object for preparing mesh.
tk::Progress<5> m_progWork
Progress object for preparing workers.
void pup(PUP::er& p) override
Pack/Unpack serialize member function.
void operator|(PUP::er& p, Transporter& t)
Pack/Unpack serialize operator|.

Function documentation

void inciter::Transporter::load(std::size_t meshid, std::size_t nelem)

Reduction target: the mesh has been read from file on all PEs.

Parameters
meshid in Mesh id (summed accross all compute nodes)
nelem in Number of mesh elements per mesh (summed across all compute nodes)

void inciter::Transporter::partitioned(std::size_t meshid)

Reduction target: a mesh has been partitioned.

Parameters
meshid in Mesh id

void inciter::Transporter::distributed(std::size_t meshid)

Reduction target: all compute nodes have distrbuted their mesh after partitioning.

Parameters
meshid in Mesh id

void inciter::Transporter::queriedRef(std::size_t meshid)

Reduction target: all Refiner chares have queried their boundary edges.

Parameters
meshid in Mesh id

void inciter::Transporter::respondedRef(std::size_t meshid)

Reduction target: all Refiner chares have setup their boundary edges.

Parameters
meshid in Mesh id

void inciter::Transporter::refinserted(std::size_t meshid, std::size_t error)

Reduction target: all compute nodes have created the mesh refiners.

Parameters
meshid in Mesh id (aggregated across all compute nodes with operator max)
error in Error code (aggregated across all compute nodes with operator max)

void inciter::Transporter::discinserted(std::size_t meshid)

Reduction target: all Discretization chares have been inserted.

Parameters
meshid in Mesh id

void inciter::Transporter::disccreated(std::size_t summeshid, std::size_t npoin)

Reduction target: all Discretization constructors have been called.

Parameters
summeshid in Mesh id (summed accross all chares)
npoin in Total number of mesh points (summed across all chares) Note that as opposed to npoin in refined(), this npoin is not multi-counted, and thus should be correct in parallel.

void inciter::Transporter::workinserted(std::size_t meshid)

Reduction target: all worker (derived discretization) chares have been inserted.

Parameters
meshid in Mesh id

void inciter::Transporter::compatibility(std::size_t meshid)

Reduction target: all Refiner chares have received a round of edges, and have run their compatibility algorithm.

Parameters
meshid in Mesh id (aggregated across all chares using operator max)

This is called iteratively, until convergence by Refiner. At this point all Refiner chares have received a round of edge data (tags whether an edge needs to be refined, etc.), and applied the compatibility algorithm independent of other Refiner chares. We keep going until the mesh is no longer modified by the compatibility algorithm, based on a new round of edge data communication started in Refiner::comExtra().

void inciter::Transporter::matched(std::size_t summeshid, std::size_t nextra, std::size_t nref, std::size_t nderef, std::size_t sumrefmode)

Reduction target: all Refiner chares have matched/corrected the tagging of chare-boundary edges, all chares are ready to perform refinement.

Parameters
summeshid in Mesh id (summed across all chares)
nextra in Sum (across all chares) of the number of edges on each chare that need correction along chare boundaries
nref in Sum of number of refined tetrahedra across all chares.
nderef in Sum of number of derefined tetrahedra across all chares.
sumrefmode in Sum of contributions from all chares, encoding refinement mode of operation.

void inciter::Transporter::bndint(tk::real sx, tk::real sy, tk::real sz, tk::real cb, tk::real summeshid)

Compute surface integral across the whole problem and perform leak-test.

Parameters
sx in X component of vector summed
sy in Y component of vector summed
sz in Z component of vector summed
cb in Invoke callback if positive
summeshid in Mesh id (summed accross all chares)

This function aggregates partial surface integrals across the boundary faces of the whole problem. After this global sum a non-zero vector result indicates a leak, e.g., a hole in the boundary, which indicates an error in the boundary face data structures used to compute the partial surface integrals.

void inciter::Transporter::refined(std::size_t summeshid, std::size_t nelem, std::size_t npoin)

Reduction target: all chares have refined their mesh.

Parameters
summeshid in Mesh id (summed accross all Refiner chares)
nelem in Total number of elements in mesh summed across the distributed mesh
npoin in Total number of mesh points summed across the distributed mesh. Note that in parallel this is larger than the number of points in the mesh, because the boundary nodes are multi-counted. But we only need an equal or larger than npoin for Sorter::setup, so this is okay.

void inciter::Transporter::resized(std::size_t meshid)

Reduction target: all worker chares have resized their own data after AMR or ALE.

Parameters
meshid in Mesh id

void inciter::Transporter::startEsup(std::size_t meshid)

Reduction target: all worker chares have generated their own esup.

Parameters
meshid in Mesh id

void inciter::Transporter::queried(std::size_t meshid)

Reduction target: all Sorter chares have queried their boundary edges.

Parameters
meshid in Mesh id

void inciter::Transporter::responded(std::size_t meshid)

Reduction target: all Sorter chares have setup their boundary edges.

Parameters
meshid in Mesh id

void inciter::Transporter::doneInsertingGhosts(std::size_t meshid)

Parameters
meshid in Mesh id

void inciter::Transporter::comfinal(std::size_t initial, std::size_t summeshid)

Reduction target indicating that the communication maps have been setup.

Parameters
initial in Sum of contributions from all chares. If larger than zero, we are during time stepping and if zero we are during setup.
summeshid in Mesh id (summed accross the distributed mesh)

void inciter::Transporter::totalvol(tk::real v, tk::real initial, tk::real summeshid)

Reduction target summing total mesh volume.

Parameters
in Mesh volume summed across the distributed mesh
initial in Sum of contributions from all chares. If larger than zero, we are during setup, if zero, during time stepping.
summeshid in Mesh id (summed accross the distributed mesh)

void inciter::Transporter::minstat(tk::real d0, tk::real d1, tk::real d2, tk::real rmeshid)

Reduction target yielding the minimum mesh statistics across all workers.

Parameters
d0 in Minimum mesh statistics collected over all chares
d1 in Minimum mesh statistics collected over all chares
d2 in Minimum mesh statistics collected over all chares
rmeshid in Mesh id as a real

void inciter::Transporter::maxstat(tk::real d0, tk::real d1, tk::real d2, tk::real rmeshid)

Reduction target yielding the maximum mesh statistics across all workers.

Parameters
d0 in Maximum mesh statistics collected over all chares
d1 in Maximum mesh statistics collected over all chares
d2 in Maximum mesh statistics collected over all chares
rmeshid in Mesh id as a real

void inciter::Transporter::sumstat(tk::real d0, tk::real d1, tk::real d2, tk::real d3, tk::real d4, tk::real d5, tk::real summeshid)

Reduction target yielding the sum of mesh statistics across all workers.

Parameters
d0 in Sum mesh statistics collected over all chares
d1 in Sum mesh statistics collected over all chares
d2 in Sum mesh statistics collected over all chares
d3 in Sum mesh statistics collected over all chares
d4 in Sum mesh statistics collected over all chares
d5 in Sum mesh statistics collected over all chares
summeshid in Mesh id (summed accross the distributed mesh)

void inciter::Transporter::pdfstat(CkReductionMsg* msg)

Reduction target yielding PDF of mesh statistics across all workers.

Parameters
msg in Serialized PDF

void inciter::Transporter::boxvol(tk::real* meshdata, int n)

Reduction target computing total volume of IC box.

Parameters
meshdata in Vector containing volumes of all IC mesh blocks, volume of IC box, and mesh id as a real summed across the distributed mesh
in Size of vector, automatically computed by Charm

void inciter::Transporter::minDtAcrossMeshes(tk::real* reducndata, int n)

Reduction target that computes minimum timestep across meshes.

Parameters
reducndata in Vector containing minimum values of dt and mesh-moved flags, collected across all meshes
in Size of vector, automatically computed by Charm

void inciter::Transporter::diagnostics(CkReductionMsg* msg)

Reduction target optionally collecting diagnostics, e.g., residuals, from all worker chares.

Parameters
msg in Serialized diagnostics vector aggregated across all PEs

void inciter::Transporter::resume()

Resume execution from checkpoint/restart files.

This is invoked by Charm++ after the checkpoint is done, as well as when the restart (returning from a checkpoint) is complete

void inciter::Transporter::checkpoint(std::size_t finished, std::size_t meshid)

Save checkpoint/restart files.

Parameters
finished in Nonzero if finished with time stepping
meshid in Mesh id

void inciter::Transporter::finish(std::size_t meshid = 0)

Normal finish of time stepping.

Parameters
meshid in Mesh id

void inciter::Transporter::pup(PUP::er& p) override

Pack/Unpack serialize member function.

Parameters
in/out Charm++'s PUP::er serializer object reference

void inciter::Transporter::operator|(PUP::er& p, Transporter& t)

Pack/Unpack serialize operator|.

Parameters
in/out Charm++'s PUP::er serializer object reference
in/out Transporter object reference

Variable documentation

std::vector<CProxy_Sorter> inciter::Transporter::m_sorter

Mesh sorter array proxy (one per mesh)

Number of mesh elements (per mesh)