inciter::Discretization class

Discretization Charm++ chare array holding common functinoality to all discretization schemes.

Contents

Public types

template<typename... tags>
struct SidesetNodes
Function object for querying the node ids that belong to side sets of the same type, called for each PDE type.

Public static functions

static void registerReducers()
Configure Charm++ reduction types.

Constructors, destructors, conversion operators

Discretization(CkMigrateMessage* msg) explicit
Migrate constructor.

Public functions

auto Discretization(std::size_t meshid, const std::vector<CProxy_Discretization>& disc, const CProxy_ALE& aleproxy, const tk::CProxy_ConjugateGradients& conjugategradientsproxy, const tk::CProxy_BiCG& implicitsolverproxy, const CProxy_Transporter& transporter, const tk::CProxy_MeshWriter& meshwriter, const tk::UnsMesh::CoordMap& coordmap, const tk::UnsMesh::Chunk& el, const tk::CommMaps& msum, const std::map<int, std::vector<std::size_t>>& bface, const std::vector<std::size_t>& triinpoel, const std::unordered_map<std::size_t, std::set<std::size_t>>& elemblockid, int nc) -> Discretization_SDAG_CODE explicit
Constructor.
void meshvelStart(const tk::UnsMesh::Coords vel, const std::vector<tk::real>& soundspeed, const std::unordered_map<int, std::unordered_map<std::size_t, std::array<tk::real, 4>>>& bnorm, tk::real adt, CkCallback done) const
Start computing new mesh veloctity for ALE mesh motion.
auto meshvel() const -> const tk::Fields&
Query the mesh velocity.
void meshvelBnd(const std::map<int, std::vector<std::size_t>>& bface, const std::map<int, std::vector<std::size_t>>& bnode, const std::vector<std::size_t>& triinpoel) const
Query ALE mesh velocity boundary condition node lists and node lists at which ALE moves boundaries.
void meshvelConv()
Assess and record mesh velocity linear solver convergence.
void transferInit()
Our mesh has been registered with the mesh-to-mesh transfer library (if coupled to other solver)
void comfinal()
Finish setting up communication maps and solution transfer callbacks.
void transfer(tk::Fields& u, std::size_t dirn, CkCallback cb)
Start solution transfer (if coupled)
void to_complete()
Solution transfer from background to overset mesh completed (from ExaM2M)
void from_complete()
Solution transfer from overset to background mesh completed (from ExaM2M)
void transfer_complete_from_dest()
Solution transfer completed (from dest Discretization)
void transfer_complete()
Solution transfer completed (one-way)
void resizePostAMR(const tk::UnsMesh::Chunk& chunk, const tk::UnsMesh::Coords& coord, const std::unordered_map<std::size_t, std::size_t>& amrNodeMap, const tk::NodeCommMap& nodeCommMap, const std::set<std::size_t>& removedNodes, const std::unordered_map<std::size_t, std::set<std::size_t>>& elemblockid)
Resize mesh data structures after mesh refinement.
void startvol()
Get ready for (re-)computing/communicating nodal volumes.
void vol()
Sum mesh volumes to nodes, start communicating them on chare-boundaries.
void setRefiner(const CProxy_Refiner& ref)
Set Refiner Charm++ proxy.
void comvol(const std::vector<std::size_t>& gid, const std::vector<tk::real>& nodevol)
Collect nodal volumes across chare boundaries.
void totalvol()
Sum mesh volumes and contribute own mesh volume to total volume.
void stat(tk::real mesh_volume)
Compute mesh cell statistics.
void boxvol(const std::vector<std::unordered_set<std::size_t>>& nodes, const std::unordered_map<std::size_t, std::set<std::size_t>>& nodeblk, std::size_t nuserblk)
Compute total box IC volume.
void setdt(tk::real newdt)
Set time step size.
void next()
Prepare for next step.
void status()
Otput one-liner status report.
auto histfilename(const std::string& id, std::streamsize precision) -> std::string
Construct history output filename.
void histheader(std::vector<std::string>&& names)
Output headers for time history files (one for each point)
void history(std::vector<std::vector<tk::real>>&& data)
Output time history for a time step.
void write(const std::vector<std::size_t>& inpoel, const tk::UnsMesh::Coords& coord, const std::map<int, std::vector<std::size_t>>& bface, const std::map<int, std::vector<std::size_t>>& bnode, const std::vector<std::size_t>& triinpoel, const std::vector<std::string>& elemfieldnames, const std::vector<std::string>& nodefieldnames, const std::vector<std::string>& elemsurfnames, const std::vector<std::string>& nodesurfnames, const std::vector<std::vector<tk::real>>& elemfields, const std::vector<std::vector<tk::real>>& nodefields, const std::vector<std::vector<tk::real>>& elemsurfs, const std::vector<std::vector<tk::real>>& nodesurfs, CkCallback c)
Output mesh and fields data (solution dump) to file(s)
void grindZero()
Zero grind-timer.
auto restarted(int nrestart) -> bool
Detect if just returned from a checkpoint and if so, zero timers.
void remap(const std::unordered_map<std::size_t, std::size_t>& map)
Remap mesh data due to new local ids.
template<typename... tags>
auto bcnodes(const std::map<int, std::vector<std::size_t>>& bface, const std::vector<std::size_t>& triinpoel) const -> std::unordered_map<int, std::unordered_set<std::size_t>>
Query nodes that belong to side sets of the same type for all PDE types.
auto bndel() const -> std::vector<std::size_t>
Find elements along our mesh chunk boundary.
auto fielditer() const -> bool
Decide if field output iteration count interval is hit.
auto fieldtime() const -> bool
Decide if field output physics time interval is hit.
auto fieldrange() const -> bool
Decide if physics time falls into a field output time range.
auto histiter() const -> bool
Decide if history output iteration count interval is hit.
auto histtime() const -> bool
Decide if history output physics time interval is hit.
auto histrange() const -> bool
Decide if physics time falls into a history output time range.
auto finished() const -> bool
Decide if this is the last time step.

Accessors

auto Coord() const -> const tk::UnsMesh::Coords&
auto Coord() -> tk::UnsMesh::Coords&
Coordinates accessor as reference.
auto Coordn() const -> const tk::UnsMesh::Coords&
Coordinates at time n accessor as const-ref.
auto Gid() const -> const std::vector<std::size_t>&
Global ids accessors as const-ref.
auto Lid() const -> const std::unordered_map<std::size_t, std::size_t>&
Local ids accessors as const-ref.
auto Inpoel() const -> const std::vector<std::size_t>&
Tetrahedron element connectivity (with local ids) accessors as const-ref.
auto Chunk() const -> const tk::UnsMesh::Chunk&
Mesh chunk accessor as const-ref.
auto ElemBlockId() const -> const std::unordered_map<std::size_t, std::set<std::size_t>>&
Mesh block id accessor as const-ref.
auto meshvol() const -> tk::real
Total mesh volume accessor.
auto V() const -> const std::vector<tk::real>&
Nodal mesh volume accessors const-ref.
auto Vol() const -> const std::vector<tk::real>&
Nodal mesh volumes at current time step accessors as const-ref.
auto Voln() const -> const std::vector<tk::real>&
Nodal mesh volumes at previous time step accessors as const-ref.
auto Voln() -> std::vector<tk::real>&
Nodal mesh volumes at previous time step accessors as ref.
auto Vol0() const -> const std::vector<tk::real>&
Element mesh volumes at t=t0 accessors as const-ref.
auto MeshVel() const -> const tk::Fields&
Element mesh velocity accessor as const-ref.
auto MeshVel() -> tk::Fields&
Element mesh velocity accessor as ref.
void Initial(std::size_t i)
auto Initial() const -> bool
void UpdateCoordn()
Update coordinates at time n.
auto Hist() const -> const std::vector<HistData>&
History points data accessor as const-ref.
auto Boxvol() -> tk::real&
Box volume accessor.
auto MeshBlkVol() -> std::vector<tk::real>&
Block volume accessor.
auto MeshId() const -> std::size_t
Mesh ID accessor.
auto Dt() const -> tk::real
Time step size accessor.
auto Dtn() const -> tk::real
Time step size at previous time step accessor.
auto T() const -> tk::real
Physical time accessor.
auto It() const -> uint64_t
Iteration count accessor.
auto Itr() -> uint64_t&
Non-const-ref refinement iteration count accessor.
auto Itf() -> uint64_t&
Non-const-ref field-output iteration count accessor.
auto Nrestart() -> int&
Non-const-ref number of restarts accessor.
auto Timer() const -> const tk::Timer&
Timer accessor as const-ref.
auto Timer() -> tk::Timer&
Timer accessor as non-const-ref.
auto refined() const -> int
Accessor to flag indicating if the mesh was refined as a value.
auto refined() -> int&
Accessor to flag indicating if the mesh was refined as non-const-ref.
auto ImplicitSolver() -> tk::CProxy_BiCG&
Accessor to bound implicit solver non-const-ref.
auto Tr() const -> const CProxy_Transporter&
Transporter proxy accessor as const-ref.
auto Tr() -> CProxy_Transporter&
Transporter proxy accessor as non-const-ref.
auto Ref() const -> Refiner*
Access bound Refiner class pointer.
auto coupled(std::size_t meshid) const -> CProxy_Discretization
Access Discretization proxy for a mesh.
auto Transfers() const -> const std::vector<Transfer>&
Const-ref accessor to solver/mesh transfer configuration.
auto Bid() const -> const std::unordered_map<std::size_t, std::size_t>&
Boundary node ids accessor as const-ref.
auto NodeCommMap() const -> const tk::NodeCommMap&
Node communication map accessor as const-ref.
auto EdgeCommMap() const -> const tk::EdgeCommMap&
Edge communication map accessor as const-ref.

Charm++ pack/unpack serializer member functions

void pup(PUP::er& p) override
Pack/Unpack serialize member function.
void operator|(PUP::er& p, Discretization& i)
Pack/Unpack serialize operator|.

Function documentation

static void inciter::Discretization::registerReducers()

Configure Charm++ reduction types.

Since this is a [initnode] routine, see the .ci file, the Charm++ runtime system executes the routine exactly once on every logical node early on in the Charm++ init sequence. Must be static as it is called without an object. See also: Section "Initializations at Program Startup" at in the Charm++ manual http://charm.cs.illinois.edu/manuals/html/charm++/manual.html.

Discretization_SDAG_CODE inciter::Discretization::Discretization(std::size_t meshid, const std::vector<CProxy_Discretization>& disc, const CProxy_ALE& aleproxy, const tk::CProxy_ConjugateGradients& conjugategradientsproxy, const tk::CProxy_BiCG& implicitsolverproxy, const CProxy_Transporter& transporter, const tk::CProxy_MeshWriter& meshwriter, const tk::UnsMesh::CoordMap& coordmap, const tk::UnsMesh::Chunk& el, const tk::CommMaps& msum, const std::map<int, std::vector<std::size_t>>& bface, const std::vector<std::size_t>& triinpoel, const std::unordered_map<std::size_t, std::set<std::size_t>>& elemblockid, int nc) explicit

Constructor.

Parameters
meshid in Mesh ID
disc in All Discretization proxies (one per mesh)
aleproxy in Distributed ALE proxy
conjugategradientsproxy in Distributed Conjugrate Gradients linear solver proxy
implicitsolverproxy
transporter in Host (Transporter) proxy
meshwriter in Mesh writer proxy
coordmap in Coordinates of mesh nodes and their global IDs
el in Elements of the mesh chunk we operate on
msum in Communication maps associated to chare IDs bordering the mesh chunk we operate on
bface in Face lists mapped to side set ids
triinpoel in Interconnectivity of points and boundary-faces
elemblockid in Local tet ids associated with mesh block ids
nc in Total number of Discretization chares

void inciter::Discretization::meshvelStart(const tk::UnsMesh::Coords vel, const std::vector<tk::real>& soundspeed, const std::unordered_map<int, std::unordered_map<std::size_t, std::array<tk::real, 4>>>& bnorm, tk::real adt, CkCallback done) const

Start computing new mesh veloctity for ALE mesh motion.

Parameters
vel in Fluid velocity at mesh nodes
soundspeed in Speed of sound at mesh nodes
bnorm in Face normals in boundary points associated to side sets
adt in alpha*dt of the RK time step
done in Function to continue with when mesh velocity has been computed

const tk::Fields& inciter::Discretization::meshvel() const

Query the mesh velocity.

Returns Mesh velocity

Query the mesh velocity

void inciter::Discretization::transfer(tk::Fields& u, std::size_t dirn, CkCallback cb)

Start solution transfer (if coupled)

Parameters
u in/out Solution to transfer from/to
dirn in Direction of solution transfer. 0: from background to overset, 1: from overset to background
cb in Callback to call when back and forth transfers complete.

This function initiates the solution transfer (direction dependent on 'dirn') between meshes. It invokes a reduction to Transporter when the transfer in one direction is complete (dirn == 0), or calls back the 'cb' function in Scheme when transfers both directions are complete. The function relies on 'dirn' to make this decision.

void inciter::Discretization::to_complete()

Solution transfer from background to overset mesh completed (from ExaM2M)

This is called by ExaM2M on the destination mesh when the transfer completes. Since this is called only on the destination, we find and notify the corresponding source of the completion.

Solution transfer from background to overset mesh completed (from ExaM2M)

void inciter::Discretization::from_complete()

Solution transfer from overset to background mesh completed (from ExaM2M)

This is called by ExaM2M on the destination mesh when the transfer completes. Since this is called only on the destination, we find and notify the corresponding source of the completion.

Solution transfer from overset to background mesh completed (from ExaM2M)

void inciter::Discretization::transfer_complete_from_dest()

Solution transfer completed (from dest Discretization)

Solution transfer completed (from dest Discretization)

Called on the source only by the destination when a back and forth transfer step completes.

void inciter::Discretization::transfer_complete()

Solution transfer completed (one-way)

Solution transfer completed (one-way)

void inciter::Discretization::resizePostAMR(const tk::UnsMesh::Chunk& chunk, const tk::UnsMesh::Coords& coord, const std::unordered_map<std::size_t, std::size_t>& amrNodeMap, const tk::NodeCommMap& nodeCommMap, const std::set<std::size_t>& removedNodes, const std::unordered_map<std::size_t, std::set<std::size_t>>& elemblockid)

Resize mesh data structures after mesh refinement.

Parameters
chunk in New mesh chunk (connectivity and global<->local id maps)
coord in New mesh node coordinates
amrNodeMap in Node id map after amr (local ids)
nodeCommMap in New node communication map
removedNodes in Newly removed mesh node local ids
elemblockid in New local tet ids associated with mesh block ids

void inciter::Discretization::setRefiner(const CProxy_Refiner& ref)

Set Refiner Charm++ proxy.

Parameters
ref in Incoming refiner proxy to store

void inciter::Discretization::comvol(const std::vector<std::size_t>& gid, const std::vector<tk::real>& nodevol)

Collect nodal volumes across chare boundaries.

Parameters
gid in Global mesh node IDs at which we receive volume contributions
nodevol in Partial sums of nodal volume contributions to chare-boundary nodes

This function receives contributions to m_vol, which stores the nodal volumes. While m_vol stores own contributions, m_volc collects the neighbor chare contributions during communication. This way work on m_vol and m_volc is overlapped. The contributions are applied in totalvol().

void inciter::Discretization::stat(tk::real mesh_volume)

Compute mesh cell statistics.

Parameters
mesh_volume in Total mesh volume

void inciter::Discretization::boxvol(const std::vector<std::unordered_set<std::size_t>>& nodes, const std::unordered_map<std::size_t, std::set<std::size_t>>& nodeblk, std::size_t nuserblk)

Compute total box IC volume.

Parameters
nodes in Node list contributing to box IC volume (for each IC box)
nodeblk in Node list associated to mesh blocks contributing to block volumes (for each IC box)
nuserblk in Number of user IC mesh blocks

void inciter::Discretization::setdt(tk::real newdt)

Set time step size.

Parameters
newdt in Size of the new time step

std::string inciter::Discretization::histfilename(const std::string& id, std::streamsize precision)

Construct history output filename.

Parameters
id in History point id
precision in Floating point precision to use for output
Returns History file name

void inciter::Discretization::histheader(std::vector<std::string>&& names)

Output headers for time history files (one for each point)

Parameters
names in History output variable names

void inciter::Discretization::history(std::vector<std::vector<tk::real>>&& data)

Output time history for a time step.

Parameters
data in Time history data for all variables and equations integrated

void inciter::Discretization::write(const std::vector<std::size_t>& inpoel, const tk::UnsMesh::Coords& coord, const std::map<int, std::vector<std::size_t>>& bface, const std::map<int, std::vector<std::size_t>>& bnode, const std::vector<std::size_t>& triinpoel, const std::vector<std::string>& elemfieldnames, const std::vector<std::string>& nodefieldnames, const std::vector<std::string>& elemsurfnames, const std::vector<std::string>& nodesurfnames, const std::vector<std::vector<tk::real>>& elemfields, const std::vector<std::vector<tk::real>>& nodefields, const std::vector<std::vector<tk::real>>& elemsurfs, const std::vector<std::vector<tk::real>>& nodesurfs, CkCallback c)

Output mesh and fields data (solution dump) to file(s)

Parameters
inpoel in Mesh connectivity for the mesh chunk to be written
coord in Node coordinates of the mesh chunk to be written
bface in Map of boundary-face lists mapped to corresponding side set ids for this mesh chunk
bnode in Map of boundary-node lists mapped to corresponding side set ids for this mesh chunk
triinpoel in Interconnectivity of points and boundary-face in this mesh chunk
elemfieldnames in Names of element fields to be output to file
nodefieldnames in Names of node fields to be output to file
elemsurfnames in Names of elemental surface fields to be output to file
nodesurfnames in Names of node surface fields to be output to file
elemfields in Field data in mesh elements to output to file
nodefields in Field data in mesh nodes to output to file
elemsurfs in Surface field data in mesh elements to output to file
nodesurfs in Surface field data in mesh nodes to output to file
c in Function to continue with after the write

Since m_meshwriter is a Charm++ chare group, it never migrates and an instance is guaranteed on every PE. We index the first PE on every logical compute node. In Charm++'s non-SMP mode, a node is the same as a PE, so the index is the same as CkMyPe(). In SMP mode the index is the first PE on every logical node. In non-SMP mode this yields one or more output files per PE with zero or non-zero virtualization, respectively. If there are multiple chares on a PE, the writes are serialized per PE, since only a single entry method call can be executed at any given time. In SMP mode, still the same number of files are output (one per chare), but the output is serialized through the first PE of each compute node. In SMP mode, channeling multiple files via a single PE on each node is required by NetCDF and HDF5, as well as ExodusII, since none of these libraries are thread-safe.

bool inciter::Discretization::restarted(int nrestart)

Detect if just returned from a checkpoint and if so, zero timers.

Parameters
nrestart in Number of times restarted
Returns True if restart detected

void inciter::Discretization::remap(const std::unordered_map<std::size_t, std::size_t>& map)

Remap mesh data due to new local ids.

Parameters
map in Mapping of old->new local ids

template<typename... tags>
std::unordered_map<int, std::unordered_set<std::size_t>> inciter::Discretization::bcnodes(const std::map<int, std::vector<std::size_t>>& bface, const std::vector<std::size_t>& triinpoel) const

Query nodes that belong to side sets of the same type for all PDE types.

Template parameters
tags Tags addressing the location of a vector of vectors of side set ids in the input deck
Parameters
bface in Boundary-faces mapped to side set ids
triinpoel in Boundary-face connectivity
Returns Node ids that belong side sets of the same type (value), associated to sides set id (key)

std::vector<std::size_t> inciter::Discretization::bndel() const

Find elements along our mesh chunk boundary.

Returns List of local element ids that have at least a single node contributing to a chare boundary

bool inciter::Discretization::fielditer() const

Decide if field output iteration count interval is hit.

Returns True if field output iteration count interval is hit

bool inciter::Discretization::fieldtime() const

Decide if field output physics time interval is hit.

Returns True if field output physics time interval is hit

bool inciter::Discretization::fieldrange() const

Decide if physics time falls into a field output time range.

Returns True if physics time falls into a field output time range

bool inciter::Discretization::histiter() const

Decide if history output iteration count interval is hit.

Returns True if history output iteration count interval is hit

bool inciter::Discretization::histtime() const

Decide if history output physics time interval is hit.

Returns True if history output physics time interval is hit

bool inciter::Discretization::histrange() const

Decide if physics time falls into a history output time range.

Returns True if physics time falls into a history output time range

bool inciter::Discretization::finished() const

Decide if this is the last time step.

Returns True if this is the last time step

const tk::UnsMesh::Coords& inciter::Discretization::Coord() const

Coordinates accessor as const-ref

void inciter::Discretization::Initial(std::size_t i)

Parameters
i in Value to put in 'initial'

Set 'initial' flag

bool inciter::Discretization::Initial() const

Returns True during setup, false durign time stepping

Query 'initial' flag

void inciter::Discretization::pup(PUP::er& p) override

Pack/Unpack serialize member function.

Parameters
p in/out Charm++'s PUP::er serializer object reference

void inciter::Discretization::operator|(PUP::er& p, Discretization& i)

Pack/Unpack serialize operator|.

Parameters
p in/out Charm++'s PUP::er serializer object reference
i in/out Discretization object reference