Lethe Open-Source Library version 1.2.0
Developer Guide
Loading...
Searching...
No Matches
load_balancing.h
Go to the documentation of this file.
1// SPDX-FileCopyrightText: Copyright (c) 2022-2026 The Lethe Authors
2// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception OR LGPL-2.1-or-later
3
4#ifndef lethe_load_balancing_h
5#define lethe_load_balancing_h
6
9
11
12#include <deal.II/distributed/tria.h>
13
14using namespace dealii;
15
16/**
17 * @brief Manages the load balancing (repartitioning), of the domain according
18 * to the computational load of the cells and the particles.
19 * It is used by the DEM and the coupling CFD-DEM solvers.
20 *
21 * @tparam dim An integer that denotes the number of spatial dimensions.
22 * @tparam PropertiesIndex Index of the properties used within the ParticleHandler.
23 */
24template <int dim, typename PropertiesIndex>
26{
27public:
29
30public:
31 /**
32 * @brief Sets the parameters for the load balancing method given by the model
33 * parameters. It also initializes the iteration load balancing check method.
34 *
35 * @param[in] model_parameters The model parameter of DEM including the load
36 * balancing parameters.
37 */
38 inline void
41 {
42 // Load balancing method and the setup of the check iteration function
45
46 // Parameters related to the total cell weight
51
52 // Parameters related to the frequency of the load balance execution
55 load_balance_step = model_parameters.load_balance_step;
57 load_threshold = model_parameters.load_balance_threshold;
58 }
59
60 /**
61 * @brief Copies the references to the simulation control, triangulation,
62 * particle handler and adaptive sparse contacts (enabled or disabled).
63 * This is necessary to access the data structures and functions of these
64 * during the load balancing check
65 *
66 * @param[in] simulation_control The simulation control object (accesses to
67 * the time step)
68 * @param[in] triangulation The triangulation object (for dynamics connexion
69 * of signals with ASC)
70 * @param[in] particle_handler The particle handler object (accesses to the
71 * particles in cells)
72 * @param[in] adaptive_sparse_contacts The adaptive sparse contacts object
73 * (accesses to the mobility status of the cells)
74 */
75 inline void
77 std::shared_ptr<SimulationControl> &simulation_control,
78 parallel::distributed::Triangulation<dim> &triangulation,
79 Particles::ParticleHandler<dim> &particle_handler,
81 {
82 this->simulation_control = simulation_control;
83 this->triangulation = &triangulation;
84 this->particle_handler = &particle_handler;
85 this->adaptive_sparse_contacts = &adaptive_sparse_contacts;
86 }
87
88 /**
89 * @brief Checks if the current iteration is a load balance iteration.
90 *
91 * @return bool indicating if this is a load balance iteration.
92 */
93 inline void
98
99 /**
100 * @brief Sets the load balancing iteration check function according
101 * to the chosen load balancing method. During the setup of the parameters,
102 * the validation of the method is already checked, so the default value is
103 * when the method is `none`.
104 *
105 * @return Return a function that returns a bool indicating if the current
106 * time step is a load balance iteration.
107 */
108 inline std::function<void()>
110 {
111 using namespace Parameters::Lagrangian;
112
113 switch (load_balance_method)
114 {
116 return [&] { check_load_balance_once(); };
118 return [&] { check_load_balance_frequent(); };
120 return [&] { check_load_balance_dynamic(); };
121 case ModelParameters<
122 dim>::LoadBalanceMethod::dynamic_with_sparse_contacts:
124 default: // Default is no load balance (none)
125 return [&]() { return; };
126 }
127 }
128
129 /**
130 * @brief Determines whether the present iteration is the load balance step
131 * when load balance method is `once`.
132 *
133 * @return bool indicating if the current time step is a load balance
134 * iteration according to the load balancing `step`.
135 */
136 void
138
139 /**
140 * @brief Determines whether the present iteration is the load balance step
141 * when load balance method is `frequent`.
142 *
143 * @return bool indicating if the current time step is a load balance
144 * iteration according to the load balancing `frequency`.
145 */
146 void
148
149 /**
150 * @brief Determines whether the present iteration is the load balance step
151 * when load balance method is `dynamic`.
152 *
153 * @return bool indicating if the current time step is a load balance
154 * iteration according to the load balancing `dynamic check frequency`and
155 * `threshold`.
156 */
157 void
159
160 /**
161 * @brief Determines whether the present is the load balance step when
162 * load balance method is `dynamic_with_sparse_contacts`. It used the same
163 * method as `dynamic` but with the addition of factors according to the cell
164 * mobility status of the sparse contacts mechanism.
165 *
166 * @return bool indicating if the current time step is a load balance
167 * iteration according to the load balancing `dynamic check frequency`,
168 * `threshold` and load factors.
169 */
170 void
172
173 /**
174 * @brief Connects the weight signals of the cells to the triangulation.
175 *
176 * In order to consider the particles when repartitioning the triangulation
177 * the algorithm needs to know three things:
178 * 1. How much weight to assign to each cell (how many particles are in
179 * there)
180 * 2. How to pack the particles before shipping data around
181 * 3. How to unpack the particles after repartitioning
182 * Attach the correct functions to the signals inside
183 * parallel::distributed::Triangulation, which will be called every time the
184 * load balancing or refinement functions are called.
185 */
186 inline void
188 {
189 triangulation->signals.weight.connect(
190 [&](const typename Triangulation<dim>::cell_iterator &cell,
191 const CellStatus) -> unsigned int {
192 return static_cast<int>(cell_weight_function->value(cell->center()));
193 });
194
195 triangulation->signals.weight.connect(
196 [&](const typename parallel::distributed::Triangulation<
197 dim>::cell_iterator &cell,
198 const CellStatus status) -> unsigned int {
199 return this->calculate_total_cell_weight(cell, status);
200 });
201 }
202
203 /**
204 * @brief Connects the weight signals of the cells to the triangulation with
205 * the mobility status. Cells are reconnected when load balancing is checked
206 * with the new mobility status.
207 */
208 inline void
210 {
211 // Clear and connect a new cell weight function
212 triangulation->signals.weight.disconnect_all_slots();
213
214 // Connect, or reconnect, the default cell weight function
215 triangulation->signals.weight.connect(
216 [&](const typename Triangulation<dim>::cell_iterator &,
217 const CellStatus) -> unsigned int { return cell_weight; });
218
219 triangulation->signals.weight.connect(
220 [&](const typename parallel::distributed::Triangulation<
221 dim>::cell_iterator &cell,
222 const CellStatus status) -> unsigned int {
224 status);
225 });
226 }
227
228private:
229 /**
230 * @brief Indicates to the triangulation how much computational work is
231 * expected to happen on this cell, and consequently how the domain needs to
232 * be partitioned.
233 *
234 * Every MPI rank receives a roughly equal amount of work (potentially not an
235 * equal number of cells). While the function is called from the outside,
236 * it is connected to the corresponding signal from inside this class,
237 * therefore it can be private. This function is the key component that allows
238 * dynamically balance the computational load. The function attributes a
239 * weight to every cell that represents the computational work on this cell.
240 * Here the majority of work is expected to happen on the particles, therefore
241 * the return value of this function is calculated based on the number of
242 * particles in the current cell. The function is connected to the
243 * cell_weight() signal inside the triangulation, and will be called once per
244 * cell, whenever the triangulation repartition the domain between ranks.
245 *
246 * @param[in] cell The cell for which the load is calculated.
247 *
248 * @param[in] status The status of the cell related to the coarsening level.
249 *
250 * TODO: Remove preprocessor directives after deal.ii v.9.6 release
251 *
252 * @return The total weight of the cell.
253 */
254 unsigned int
256 const typename parallel::distributed::Triangulation<dim>::cell_iterator
257 &cell,
258 const CellStatus status) const;
259
260 /**
261 * @brief Indicates to the triangulation how much computational work is
262 * expected to happen on this cell, and consequently how the domain needs to
263 * be partitioned. Only used with the Adaptive Sparse Contacts mechanism.
264 *
265 * Similar to the calculate_total_cell_weight(), this function is used when
266 * the cell weight is adapted to the mobility status. For instance, if the
267 * cell is inactive, its computational load will be significantly lower than
268 * if it is a mobile cell since there is no force calculation and no velocity
269 * integration for the particles that lie within it. The weight of the cells
270 * must thus be adapted to the status of the cell.
271 *
272 * total cell weight = cell weight +
273 * load balancing factor * n particles * particle weight
274 *
275 *
276 * @param[in] cell The cell for which the load is calculated
277 * @param[in] status The status of the cell related to the coarsening level
278 *
279 * @return The total weight of the cell
280 */
281 unsigned int
283 const typename parallel::distributed::Triangulation<dim>::cell_iterator
284 &cell,
285 const CellStatus status) const;
286
287 /**
288 * @brief The load balancing method chosen by the user.
289 */
292
293 /**
294 * @brief The load balancing iteration check function according to the load
295 * balancing method.
296 */
297 std::function<void()> iteration_check_function;
298
299 /**
300 * @brief Default hard-coded load weight of a cell.
301 */
302 const unsigned int cell_weight = 1000;
303
304 std::shared_ptr<Function<dim>> cell_weight_function;
305
306 /**
307 * @brief Load weight of a particle, the default parameters is 10000.
308 */
309 unsigned int particle_weight;
310
311 /**
312 * @brief Load weight factor of particle weight in a cell with an inactive
313 * mobility status (only with ASC).
314 */
316
317 /**
318 * @brief Load weight factor of particle weight in a cell with an active
319 * mobility status (only with ASC).
320 */
322
323 /**
324 * @brief Iteration number for load balancing execution, only when method
325 * is `once`.
326 */
327 unsigned int load_balance_step;
328
329 /**
330 * @brief Frequency of load balancing execution, only when method is
331 * `frequent`.
332 */
334
335 /**
336 * @brief Frequency of check for the load balancing execution, only when
337 * method is `dynamic`.
338 */
340
341 /**
342 * @brief Threshold (\beta) for the check load balancing execution, only when
343 * method is `dynamic`.
344 *
345 * Load balancing will be done when the computational load amongst core is too
346 * uneven. If L_{max} - L_{min} > \beta \bar{L}, where L_{max}, L_{min}, and
347 * \bar{L} are the maximum, minimum, and the average load, the load balancing
348 * will be executed.
349 */
351
352 /**
353 * @brief MPI communicator.
354 */
356
357 /**
358 * @brief Number of MPI processes.
359 */
360 const unsigned int n_mpi_processes;
361
362 /**
363 * @brief Rank of the current MPI process.
364 */
365 const unsigned int this_mpi_process;
366
367 /**
368 * @brief Pointer to the simulation control object.
369 */
370 std::shared_ptr<SimulationControl> simulation_control;
371
372 /**
373 * @brief Pointer to the triangulation object.
374 */
375 parallel::distributed::Triangulation<dim> *triangulation;
376
377 /**
378 * @brief Pointer to the particle handler object.
379 */
380 Particles::ParticleHandler<dim> *particle_handler;
381
382 /**
383 * @brief Pointer to the adaptive sparse contacts object.
384 */
386};
387
388#endif
Mechanism that disables contacts computation through mobility status of cells based on granular tempe...
Parameters::Lagrangian::ModelParameters< dim >::LoadBalanceMethod load_balance_method
The load balancing method chosen by the user.
Particles::ParticleHandler< dim > * particle_handler
Pointer to the particle handler object.
AdaptiveSparseContacts< dim, PropertiesIndex > * adaptive_sparse_contacts
Pointer to the adaptive sparse contacts object.
void connect_mobility_status_weight_signals()
Connects the weight signals of the cells to the triangulation with the mobility status....
const unsigned int this_mpi_process
Rank of the current MPI process.
std::shared_ptr< SimulationControl > simulation_control
Pointer to the simulation control object.
void check_load_balance_iteration() const
Checks if the current iteration is a load balance iteration.
double active_status_factor
Load weight factor of particle weight in a cell with an active mobility status (only with ASC).
const unsigned int n_mpi_processes
Number of MPI processes.
unsigned int load_balance_step
Iteration number for load balancing execution, only when method is once.
parallel::distributed::Triangulation< dim > * triangulation
Pointer to the triangulation object.
double inactive_status_factor
Load weight factor of particle weight in a cell with an inactive mobility status (only with ASC).
const unsigned int cell_weight
Default hard-coded load weight of a cell.
std::shared_ptr< Function< dim > > cell_weight_function
unsigned int load_balance_frequency
Frequency of load balancing execution, only when method is frequent.
void check_load_balance_once() const
Determines whether the present iteration is the load balance step when load balance method is once.
unsigned int dynamic_check_frequency
Frequency of check for the load balancing execution, only when method is dynamic.
unsigned int particle_weight
Load weight of a particle, the default parameters is 10000.
std::function< void()> iteration_check_function
The load balancing iteration check function according to the load balancing method.
void check_load_balance_with_sparse_contacts()
Determines whether the present is the load balance step when load balance method is dynamic_with_spar...
unsigned int calculate_total_cell_weight(const typename parallel::distributed::Triangulation< dim >::cell_iterator &cell, const CellStatus status) const
Indicates to the triangulation how much computational work is expected to happen on this cell,...
void connect_weight_signals()
Connects the weight signals of the cells to the triangulation.
double load_threshold
Threshold (\beta) for the check load balancing execution, only when method is dynamic.
void copy_references(std::shared_ptr< SimulationControl > &simulation_control, parallel::distributed::Triangulation< dim > &triangulation, Particles::ParticleHandler< dim > &particle_handler, AdaptiveSparseContacts< dim, PropertiesIndex > &adaptive_sparse_contacts)
Copies the references to the simulation control, triangulation, particle handler and adaptive sparse ...
unsigned int calculate_total_cell_weight_with_mobility_status(const typename parallel::distributed::Triangulation< dim >::cell_iterator &cell, const CellStatus status) const
Indicates to the triangulation how much computational work is expected to happen on this cell,...
std::function< void()> set_iteration_check_function()
Sets the load balancing iteration check function according to the chosen load balancing method....
void check_load_balance_dynamic()
Determines whether the present iteration is the load balance step when load balance method is dynamic...
MPI_Comm mpi_communicator
MPI communicator.
void set_parameters(const Parameters::Lagrangian::ModelParameters< dim > &model_parameters)
Sets the parameters for the load balancing method given by the model parameters. It also initializes ...
void check_load_balance_frequent() const
Determines whether the present iteration is the load balance step when load balance method is frequen...
Numerical model parameters for DEM simulations.
unsigned int dynamic_load_balance_check_frequency
Check frequency for dynamic load balancing.
std::shared_ptr< Function< dim > > cell_weight_function
LoadBalanceMethod
Load balancing strategy for parallel DEM simulations.
@ frequent
Perform load-balancing at a given frequency.
unsigned int load_balance_particle_weight
Particle weight used for load balancing.
enum Parameters::Lagrangian::ModelParameters::LoadBalanceMethod load_balance_method
unsigned int load_balance_frequency
Load balance frequency (in time steps).
double load_balance_threshold
Load balance threshold (for dynamic load balancing).
unsigned int load_balance_step
Load balance step (for single-step load balancing).
Parameter structures for Lagrangian (DEM) particle simulations.
Simulation control classes for managing time-stepping and iteration control in Lethe.