Custom NVector for GPU by bendudson · Pull Request #3390 · boutproject/BOUT-dev

bendudson · 2026-06-16T01:15:19Z

Intended to allow SUNDIALS to operate on BOUT++ state data without copying into vectors. The hope is to avoid copying between CPU and GPU.

CVODE, IDA and Arkode can now use this ManyVector backend by setting

solver:nvector=manyvector

The default is still to use the MPI parallel N_Vector packaged with Sundials.

github-actions

clang-tidy made some suggestions

There were too many comments to post at once. Showing the first 25 out of 42. Check the log or trigger a new build to see more.

github-actions · 2026-06-16T01:19:22Z

  }
 }

+void Field2D::swapData(Field2D& other) { std::swap(data, other.data); }


warning: no header providing "std::swap" is directly included [misc-include-cleaner]

void Field2D::swapData(Field2D& other) { std::swap(data, other.data); } ^

github-actions · 2026-06-16T01:19:23Z

+  N_Vector nvector_from_state(const sundials::Context& ctx) {
+    std::vector<N_Vector> subvectors;
+    subvectors.reserve(f2d.size() + f3d.size());
+    const auto inserter = std::back_inserter(subvectors);


warning: no header providing "std::back_inserter" is directly included [misc-include-cleaner]

src/solver/impls/arkode/arkode.hxx:33:

+ #include <iterator>

github-actions · 2026-06-16T01:19:23Z

+    const auto var_str_to_nvector = [&ctx](auto &var_str) {
+      return BoutNVector::create(ctx, *var_str.var, var_str.evolve_bndry);
+    };
+    std::transform(f2d.cbegin(), f2d.cend(), inserter, var_str_to_nvector);


warning: no header providing "std::transform" is directly included [misc-include-cleaner]

src/solver/impls/arkode/arkode.hxx:33:

+ #include <algorithm>

github-actions · 2026-06-16T01:19:23Z

+      return BoutNVector::create(ctx, *var_str.var, var_str.evolve_bndry);
+    };
+    std::transform(f2d.cbegin(), f2d.cend(), inserter, var_str_to_nvector);
+    std::transform(f3d.cbegin(), f3d.cend(), inserter, var_str_to_nvector);


warning: no header providing "std::transform" is directly included [misc-include-cleaner]

std::transform(f3d.cbegin(), f3d.cend(), inserter, var_str_to_nvector); ^

github-actions · 2026-06-16T01:19:23Z

+    return BoutNVector::create(ctx, subvectors);
+  }
+
+  void swap_state(const N_Vector u) {


warning: 'u' declared with a const-qualified typedef; results in the type being '_generic_N_Vector *const' instead of 'const _generic_N_Vector *' [misc-misplaced-const]

void swap_state(const N_Vector u) { ^

Additional context

/usr/include/sundials/sundials_nvector.h:91: typedef declared here

typedef _SUNDIALS_STRUCT_ _generic_N_Vector* N_Vector; ^

github-actions · 2026-06-16T01:19:24Z

+  template <typename T>
+  struct Content {
+    T& field;
+    const bool own;


warning: member 'own' of type 'const bool' is const qualified [cppcoreguidelines-avoid-const-or-ref-data-members]

const bool own; ^

github-actions · 2026-06-16T01:19:24Z

+  struct Content {
+    T& field;
+    const bool own;
+    const bool evolve_bndry;


warning: member 'evolve_bndry' of type 'const bool' is const qualified [cppcoreguidelines-avoid-const-or-ref-data-members]

const bool evolve_bndry; ^

github-actions · 2026-06-16T01:19:24Z

+    T& field;
+    const bool own;
+    const bool evolve_bndry;
+    const sunindextype length;


warning: member 'length' of type 'const sunindextype' (aka 'const int') is const qualified [cppcoreguidelines-avoid-const-or-ref-data-members]

const sunindextype length; ^

github-actions · 2026-06-16T01:19:24Z

+          length(all_reduce(getRegion().size())) {}
+
+    auto getRegion() const {
+      return field.getRegion(evolve_bndry ? RGN_ALL : RGN_NOBNDRY);


warning: no header providing "RGN_ALL" is directly included [misc-include-cleaner]

src/solver/nvector.hxx:25:

- #include <bout/field.hxx> + #include "bout/bout_types.hxx" + #include <bout/field.hxx>

github-actions · 2026-06-16T01:19:24Z

+          length(all_reduce(getRegion().size())) {}
+
+    auto getRegion() const {
+      return field.getRegion(evolve_bndry ? RGN_ALL : RGN_NOBNDRY);


warning: no header providing "RGN_NOBNDRY" is directly included [misc-include-cleaner]

return field.getRegion(evolve_bndry ? RGN_ALL : RGN_NOBNDRY); ^

During configuration catch when `SUNDIALS::nvecmanyvector` target is missing. If so, fall back to Sundials' nvector.

Defaults to using the Sundials parallel MPI implementation. If Sundials many-vector is available then the custom N_Vector is available.

Extends the Sundials ManyVector to Sundials and IDA interfaces. To use, set ``` solver:nvector=manyvector ``` The default remains the MPI parallel N_Vector.

Backend can select either the Sundials MPI parallel N_Vector, or a ManyVector backed by BOUT++ fields.

github-actions

clang-tidy made some suggestions

There were too many comments to post at once. Showing the first 25 out of 34. Check the log or trigger a new build to see more.

github-actions · 2026-06-17T22:18:47Z

                       .withDefault(false)),
+      nvector_type((*options)["nvector"]
+                       .doc("N_Vector backend to use: sundials or manyvector")
+                       .withDefault(NVectorType::Sundials)),


warning: no header providing "NVectorType" is directly included [misc-include-cleaner]

src/solver/impls/arkode/arkode.cxx:25:

- #include "bout/build_defines.hxx" + #include "/github/workspace/src/solver/sundials_nvector_interface.hxx" + #include "bout/build_defines.hxx"

github-actions · 2026-06-17T22:18:47Z

      use_jacobian((*options)["use_jacobian"].withDefault(false)),
+      nvector_type((*options)["nvector"]
+                       .doc("N_Vector backend to use: sundials or manyvector")
+                       .withDefault(NVectorType::Sundials)),


warning: no header providing "NVectorType" is directly included [misc-include-cleaner]

src/solver/impls/cvode/cvode.cxx:29:

+ #include "/github/workspace/src/solver/sundials_nvector_interface.hxx"

github-actions · 2026-06-17T22:18:47Z

                        .withDefault(true)),
+      nvector_type((*options)["nvector"]
+                       .doc("N_Vector backend to use: sundials or manyvector")
+                       .withDefault(NVectorType::Sundials)),


warning: no header providing "NVectorType" is directly included [misc-include-cleaner]

src/solver/impls/ida/ida.cxx:32:

+ #include "/github/workspace/src/solver/sundials_nvector_interface.hxx"

github-actions · 2026-06-17T22:18:47Z

+#include <bout/field.hxx>
+#include <bout/field2d.hxx>
+#include <bout/field3d.hxx>
+#include <functional>


warning: included header field3d.hxx is not used directly [misc-include-cleaner]

Suggested change

#include <functional>

#include <functional>

github-actions · 2026-06-17T22:18:47Z

+
+class BoutNVector {
+private:
+  static double all_reduce(const double local, const MPI_Op op = MPI_SUM) {


warning: no header providing "MPI_Op" is directly included [misc-include-cleaner]

src/solver/nvector.hxx:31:

- #include <nvector/nvector_manyvector.h> + #include <mpi.h> + #include <nvector/nvector_manyvector.h>

github-actions · 2026-06-17T22:18:48Z

+
+  template <typename T, typename = bout::utils::EnableIfField<T>>
+  static T& get(const N_Vector v, std::size_t subvector) {
+    return BoutNVector::get<T>(N_VGetSubvector_ManyVector(v, subvector));


warning: narrowing conversion from 'std::size_t' (aka 'unsigned long') to signed type 'sunindextype' (aka 'int') is implementation-defined [bugprone-narrowing-conversions]

return BoutNVector::get<T>(N_VGetSubvector_ManyVector(v, subvector)); ^

github-actions · 2026-06-17T22:18:48Z

+#include "bout/solver.hxx"
+#include "bout/sundials_backports.hxx"
+
+#if BOUT_HAS_SUNDIALS_MANYVECTOR


warning: no header providing "BOUT_HAS_SUNDIALS_MANYVECTOR" is directly included [misc-include-cleaner]

src/solver/sundials_nvector_interface.hxx:30:

- #include "bout/solver.hxx" + #include "bout/build_defines.hxx" + #include "bout/solver.hxx"

github-actions · 2026-06-17T22:18:48Z

+#endif
+    }
+
+    auto* vec = callWithSUNContext(N_VNew_Parallel, ctx, BoutComm::get(), local_N, neq);


warning: no header providing "BoutComm" is directly included [misc-include-cleaner]

src/solver/sundials_nvector_interface.hxx:30:

- #include "bout/solver.hxx" + #include "bout/field2d.hxx" + #include "bout/solver.hxx"

github-actions · 2026-06-17T22:18:48Z

+  }
+
+  /// Clone a vector of the same backend/layout, with a descriptive error on failure.
+  N_Vector clone_vector_like(N_Vector source, const char* description) const {


warning: method 'clone_vector_like' can be made static [readability-convert-member-functions-to-static]

Suggested change

N_Vector clone_vector_like(N_Vector source, const char* description) const {

static N_Vector clone_vector_like(N_Vector source, const char* description) {

github-actions · 2026-06-17T22:18:48Z

+
+    if (use_manyvector()) {
+#if BOUT_HAS_SUNDIALS_MANYVECTOR
+      std::size_t i = 0;


warning: no header providing "std::size_t" is directly included [misc-include-cleaner]

src/solver/sundials_nvector_interface.hxx:32:

+ #include <cstddef>

Includes guards to handle when ManyVector is unavailable.

github-actions

clang-tidy made some suggestions

github-actions · 2026-06-17T22:49:03Z

+#if BOUT_HAS_SUNDIALS_MANYVECTOR
+      std::size_t i = 0;
+      for (std::size_t j = 0; j < f2d_values.size(); ++j, ++i) {
+        fill_field(BoutNVector::get<Field2D>(vec, i), f2d_values[j],


warning: no header providing "Field2D" is directly included [misc-include-cleaner]

fill_field(BoutNVector::get<Field2D>(vec, i), f2d_values[j], ^

github-actions · 2026-06-17T22:49:03Z

+                   solver.f2d[j].evolve_bndry);
+      }
+      for (std::size_t j = 0; j < f3d_values.size(); ++j, ++i) {
+        fill_field(BoutNVector::get<Field3D>(vec, i), f3d_values[j],


warning: no header providing "Field3D" is directly included [misc-include-cleaner]

src/solver/sundials_nvector_interface.hxx:30:

- #include "bout/solver.hxx" + #include "bout/field3d.hxx" + #include "bout/solver.hxx"

github-actions · 2026-06-17T22:49:03Z

+
+    BoutReal* option_data = N_VGetArrayPointer(vec);
+    int p = 0;
+    for (const auto& i2d : bout::globals::mesh->getRegion2D("RGN_BNDRY")) {


warning: no header providing "bout::globals::mesh" is directly included [misc-include-cleaner]

src/solver/sundials_nvector_interface.hxx:30:

- #include "bout/solver.hxx" + #include "bout/globals.hxx" + #include "bout/solver.hxx"

github-actions · 2026-06-17T22:49:04Z

+  }
+
+private:
+  Solver& solver;


warning: member 'solver' of type 'Solver &' is a reference [cppcoreguidelines-avoid-const-or-ref-data-members]

Solver& solver; ^

github-actions · 2026-06-17T22:49:04Z

+
+private:
+  Solver& solver;
+  const sundials::Context& ctx;


warning: member 'ctx' of type 'const sundials::Context &' is a reference [cppcoreguidelines-avoid-const-or-ref-data-members]

const sundials::Context& ctx; ^

github-actions · 2026-06-17T22:49:04Z

+  const sundials::Context& ctx;
+  NVectorType nvector_type;
+
+  void fill_vector_values_op(Ind2D UNUSED(i2d), BoutReal* option_data, int& p,


warning: no header providing "UNUSED" is directly included [misc-include-cleaner]

src/solver/sundials_nvector_interface.hxx:32:

+ #include "bout/unused.hxx"

github-actions · 2026-06-17T22:49:04Z

+  }
+
+  /// Swap the active solver state fields with the subvectors contained in ``u``.
+  void swap_state(const N_Vector u) const {


warning: 'u' declared with a const-qualified typedef; results in the type being '_generic_N_Vector *const' instead of 'const _generic_N_Vector *' [misc-misplaced-const]

void swap_state(const N_Vector u) const { ^

Additional context

/usr/include/sundials/sundials_nvector.h:91: typedef declared here

typedef _SUNDIALS_STRUCT_ _generic_N_Vector* N_Vector; ^

github-actions · 2026-06-17T22:49:04Z

+  }
+
+  /// Swap the active solver derivative fields with the subvectors in ``du``.
+  void swap_deriv(const N_Vector du) const {


warning: 'du' declared with a const-qualified typedef; results in the type being '_generic_N_Vector *const' instead of 'const _generic_N_Vector *' [misc-misplaced-const]

void swap_deriv(const N_Vector du) const { ^

Additional context

/usr/include/sundials/sundials_nvector.h:91: typedef declared here

typedef _SUNDIALS_STRUCT_ _generic_N_Vector* N_Vector; ^

github-actions · 2026-06-17T22:49:04Z

+
+  template <typename FieldType>
+  static void fill_field(FieldType& field, BoutReal value, bool evolve_bndry) {
+    BOUT_FOR(i, field.getRegion(evolve_bndry ? RGN_ALL : RGN_NOBNDRY)) {


warning: no header providing "BOUT_FOR" is directly included [misc-include-cleaner]

src/solver/sundials_nvector_interface.hxx:30:

- #include "bout/solver.hxx" + #include "bout/region.hxx" + #include "bout/solver.hxx"

Conditionally include `nvector/nvector_manyvector.h`

Missing includes and some `const` annotations.

github-actions

clang-tidy made some suggestions

github-actions · 2026-06-17T23:38:28Z

+
+class BoutNVector {
+private:
+  static double all_reduce(const double local, const MPI_Op op = MPI_SUM) {


warning: no header providing "MPI_Op" is directly included [misc-include-cleaner]

src/solver/nvector.hxx:32:

- #if BOUT_HAS_SUNDIALS_MANYVECTOR + #include <mpi.h> + #if BOUT_HAS_SUNDIALS_MANYVECTOR

github-actions · 2026-06-17T23:38:28Z

+class BoutNVector {
+private:
+  static double all_reduce(const double local, const MPI_Op op = MPI_SUM) {
+    double global;


warning: variable 'global' is not initialized [cppcoreguidelines-init-variables]

src/solver/nvector.hxx:33:

- #include <nvector/nvector_manyvector.h> + #include <math.h> + #include <nvector/nvector_manyvector.h>

Suggested change

double global;

double global = NAN;

github-actions · 2026-06-17T23:38:28Z

+      if (x == nullptr) {
+        return;
+      }
+      delete &get_content<T>(x);


warning: deleting a pointer through a type that is not marked 'gsl::owner<>'; consider using a smart pointer instead [cppcoreguidelines-owning-memory]

delete &get_content<T>(x); ^

Additional context

src/solver/nvector.hxx:125: variable declared here

v->ops->nvdestroy = [](N_Vector x) { ^

github-actions · 2026-06-17T23:38:29Z

+    v->ops->nvmaxnorm = [](N_Vector x) { return max(abs(get_field<T>(x)), true); };
+
+    v->ops->nvwrmsnorm = [](N_Vector x, N_Vector y) {
+      return N_VWL2Norm(x, y) / std::sqrt(static_cast<BoutReal>(N_VGetLength(x)));


warning: no header providing "std::sqrt" is directly included [misc-include-cleaner]

src/solver/nvector.hxx:31:

- #include <functional> + #include <cmath> + #include <functional>

github-actions · 2026-06-17T23:38:29Z

+  }
+
+  template <typename T, typename = bout::utils::EnableIfField<T>>
+  static void swap(const N_Vector v, T& field, std::size_t subvector) {


warning: no header providing "std::size_t" is directly included [misc-include-cleaner]

src/solver/nvector.hxx:31:

- #include <functional> + #include <cstddef> + #include <functional>

github-actions · 2026-06-17T23:38:29Z

+
+    if (use_manyvector()) {
+#if BOUT_HAS_SUNDIALS_MANYVECTOR
+      std::size_t i = 0;


warning: no header providing "std::size_t" is directly included [misc-include-cleaner]

src/solver/sundials_nvector_interface.hxx:38:

+ #include <cstddef>

Steven-Roberts and others added 26 commits May 6, 2024 17:27

Start N_Vector implementation

efd6237

Add clone operation

aaac372

Allow N_Vector template type to be Vector2D/Vector3D

8a06159

Apply formatter

a49253a

Add remaining basic operations

184b7e9

Add missing sqrt

bfdf9ec

Merge branch 'next' into cusotm-nvector

a92de11

Start adding vector to arkode integrator

6ead1ae

Merge branch 'next' into cusotm-nvector

cfbc21f

Add functions to swap state

c200e2f

Finish swap state after evolve

3518b56

Merge branch 'next' into cusotm-nvector

d2fbc28

Get prototype running

937a3c4

Merge branch 'next' into cusotm-nvector

58dc11d

Merge branch 'next' into cusotm-nvector

c62c551

Switch to sundials::Context

3fc1914

Merge branch 'next' into cusotm-nvector

ccb47fd

Use move assignment rather than pointer swapping

d7db014

Fix access to deriv field

a0eeb26

Get things working with implicit and IMEX methods

88dfb29

Fix iterator const

fa7f651

Merge branch 'next' into cusotm-nvector

4f01b4c

Merge branch 'next' into cusotm-nvector

a9d7766

Add swapData function

c52fb53

Merge branch 'next' into cusotm-nvector

506e208

Merge branch 'next' into custom-nvector

42e239f

github-actions Bot reviewed Jun 16, 2026

View reviewed changes

bendudson added 3 commits June 17, 2026 13:06

NVector: Fix reduce_field callable

b51d38a

Sundials nvecmanyvector may be missing

45f6473

During configuration catch when `SUNDIALS::nvecmanyvector` target is missing. If so, fall back to Sundials' nvector.

ARKode: nvector option to select N_Vector

b649511

Defaults to using the Sundials parallel MPI implementation. If Sundials many-vector is available then the custom N_Vector is available.

bendudson added 3 commits June 17, 2026 14:58

Sundials N_Vector backend for CVODE and IDA

d894b63

Extends the Sundials ManyVector to Sundials and IDA interfaces. To use, set ``` solver:nvector=manyvector ``` The default remains the MPI parallel N_Vector.

Document Sundials N_Vector options

ceeeb80

Backend can select either the Sundials MPI parallel N_Vector, or a ManyVector backed by BOUT++ fields.

Formatting changes

d997e6c

bendudson changed the title ~~WIP: Custom NVector for GPU~~ Custom NVector for GPU Jun 17, 2026

github-actions Bot reviewed Jun 17, 2026

View reviewed changes

Unit tests for custom N_Vector

c0abd20

Includes guards to handle when ManyVector is unavailable.

github-actions Bot reviewed Jun 17, 2026

View reviewed changes

bendudson added 2 commits June 17, 2026 16:18

NVector: Patch for MultiVector

dfa59c7

Conditionally include `nvector/nvector_manyvector.h`

Clang tidy

307a8e0

Missing includes and some `const` annotations.

github-actions Bot reviewed Jun 17, 2026

View reviewed changes

Unit tests: Disable testing if missing ManyVector

1968ac0

bendudson merged commit 611ef8d into next Jun 18, 2026
22 of 23 checks passed

bendudson deleted the custom-nvector branch June 18, 2026 01:39

	N_Vector clone_vector_like(N_Vector source, const char* description) const {
	static N_Vector clone_vector_like(N_Vector source, const char* description) {

Conversation

bendudson commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jun 16, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jun 16, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jun 16, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jun 16, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jun 16, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jun 16, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jun 16, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jun 16, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jun 16, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jun 16, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jun 17, 2026

bendudson commented Jun 16, 2026 •

edited

Loading