digitalmars.D - Using dpp on the openmpi headers
- Peter Jacobs (67/67) Mar 13 2020 Given the recent thread that mentioned dpp for generating an
- rikki cattermole (2/15) Mar 13 2020 https://dlang.org/phobos/core_runtime.html#.Runtime.cArgs
- Peter Jacobs (2/3) Mar 13 2020 Oh, that's neat. There's so much that I don't know...
- Andre Pany (7/13) Mar 13 2020 DPP cannot translate macros. Another problem is inlined
- evilrat (8/12) Mar 13 2020 With my crappy binding generator it might be possible to generate
- bachmeier (3/17) Mar 13 2020 Some of that can be done by dstep:
- Peter Jacobs (18/25) Mar 15 2020 Tried the most recent dstep but was unsuccessful...
- Jacob Carlborg (6/26) Mar 16 2020 Please report an issue here [1]. Seems to be similar to this issue [2]
- Atila Neves (9/22) Mar 13 2020 Yes. Technically I could add support for them by adding
- bachmeier (6/73) Mar 15 2020 Have you seen this project?
- Peter Jacobs (13/23) Mar 16 2020 Yes, indeed, we use that OpenMPI wrapper, actually the version at
- Peter Jacobs (5/24) Mar 21 2020 Just to let interested parties know... I went with updating the
Given the recent thread that mentioned dpp for generating an interface module for a C library, I thought that I would give it a go on the openmpi headers. I am working in LinuxMint system and after making a symbolic link for libclang.so, I have had some success but not complete success. I have a module // mpi.dpp module mpi; #include <mpi.h> /** * Convert D's string[] args to a correct C char** argv. */ char** toArgv(string[] args) { auto argv = new char*[args.length + 1]; foreach(i, arg; args) { argv[i] = (arg.dup ~ '\0').ptr; } argv[args.length] = null; return argv.ptr; } enum MPI_DUMMY = 999; a main module // hello_d.d // hacked out of the hello_d example from the OpenMPI project. import std.stdio; import std.string; import std.array; import std.algorithm; import mpi; int main(string[] args) { int argc = cast(int)args.length; auto argv = args.toArgv(); int rank; int size; writeln("MPI_VERSION=", MPI_VERSION); MPI_Init(&argc, &argv); //MPI_Comm_rank(MPI_COMM_WORLD, &rank); //MPI_Comm_size(MPI_COMM_WORLD, &size); writefln("Hello, world, I am %d of %d", rank+1, size); writeln("MPI_DUMMY=", MPI_DUMMY); //MPI_Barrier(MPI_COMM_WORLD); MPI_Finalize(); return 0; } and a makefile hello_d : hello_d.d mpi.d dmd -ofhello_d hello_d.d mpi.d -L-lmpi mpi.d : mpi.dpp dub run dpp -- --include-path=/usr/include/openmpi \ --preprocess-only mpi.dpp The dpp program successfully generates an interface module mpi.d and the hello_d executable is built and it runs. However, not all of the needed elements for a real MPI program are present. For example, in the main module above, MPI_COMM_WORLD is not available so the lines calling MPI_Comm_rank, MPI_Comm_size and MPI_Barrier are commented out. It may be that I am not using dpp correctly, however, running dpp with the -- option gives a lot output that includes Cursor(MacroDefinition, "MPI_COMM_WORLD", Type(Invalid, "")) CAN SourceRange("/usr/include/openmpi/mpi.h", 1034:9, 1034:78) so I am guessing that dpp does not know how to compute its type and value. Thoughts or suggestions on how to proceed?
Mar 13 2020
On 13/03/2020 9:08 PM, Peter Jacobs wrote:
/**
* Convert D's string[] args to a correct C char** argv.
*/
char** toArgv(string[] args)
{
auto argv = new char*[args.length + 1];
foreach(i, arg; args)
{
argv[i] = (arg.dup ~ '\0').ptr;
}
argv[args.length] = null;
return argv.ptr;
}
https://dlang.org/phobos/core_runtime.html#.Runtime.cArgs
Mar 13 2020
Peter Jacobs <peterj mech.uq.edu.au> On Friday, 13 March 2020 at 08:13:44 UTC, rikki cattermole wrote:https://dlang.org/phobos/core_runtime.html#.Runtime.cArgsOh, that's neat. There's so much that I don't know...
Mar 13 2020
On Friday, 13 March 2020 at 08:08:17 UTC, Peter Jacobs wrote:Given the recent thread that mentioned dpp for generating an interface module for a C library, I thought that I would give it a go on the openmpi headers. I am working in LinuxMint system and after making a symbolic link for libclang.so, I have had some success but not complete success. [...]DPP cannot translate macros. Another problem is inlined functions. While DPP generates their headers, their implementations are missing in the binary files. In both cases you have to reimplement the functions in D. Kind regards Andre
Mar 13 2020
On Friday, 13 March 2020 at 10:44:08 UTC, Andre Pany wrote:DPP cannot translate macros. Another problem is inlined functions. While DPP generates their headers, their implementations are missing in the binary files. In both cases you have to reimplement the functions in D.With my crappy binding generator it might be possible to generate some of this, however it will likely just suck given its current shape. Especially compared to dpp. 100% natural pain & misery, enjoy. It is still might be somewhat useful to run it and gather the inlines. Maybe. https://github.com/Superbelko/ohmygentool
Mar 13 2020
On Friday, 13 March 2020 at 10:44:08 UTC, Andre Pany wrote:On Friday, 13 March 2020 at 08:08:17 UTC, Peter Jacobs wrote:Some of that can be done by dstep: https://github.com/jacob-carlborg/dstepGiven the recent thread that mentioned dpp for generating an interface module for a C library, I thought that I would give it a go on the openmpi headers. I am working in LinuxMint system and after making a symbolic link for libclang.so, I have had some success but not complete success. [...]DPP cannot translate macros. Another problem is inlined functions. While DPP generates their headers, their implementations are missing in the binary files. In both cases you have to reimplement the functions in D. Kind regards Andre
Mar 13 2020
On Friday, 13 March 2020 at 13:00:32 UTC, bachmeier wrote:On Friday, 13 March 2020 at 10:44:08 UTC, Andre Pany wrote:Tried the most recent dstep but was unsuccessful... $ ../dstep/bin/dstep /usr/include/mpi/mpi.h -o mpi.d /usr/include/mpi/mpi.h:351:8: warning: a type renamed to 'ompi_status_public_t_' due to the collision with the symbol declared in /usr/include/mpi/mpi.h:364:37 dstep: an unknown error occurred: core.exception.AssertError dstep/translator/Type.d(32): Assertion failure ---------------- ??:? _d_assertp [0x5625185d8659] dstep/translator/Type.d:32 dstep.translator.Output.SourceNode dstep.translator.Type.translateType(dstep.translator.Context.Context, clang.Cursor.Cursor, clang.Type.Type, bool, bool) [0x5625185c1941] --- snip --- clang/Visitor.d:66 extern (C) clang.c.Index.CXChildVisitResult clang.Visitor.Visitor.visitorFunction(clang.c.Index.CXCursor, clang.c.Index.CXCursor, void*) [0x562518562feb] ??:? [0x7f7d196f7bff]On Friday, 13 March 2020 at 08:08:17 UTC, Peter Jacobs wrote:Some of that can be done by dstep: https://github.com/jacob-carlborg/dstepGiven the recent thread that mentioned dpp for generating an interface module for a C library, I thought that I would give it a go on the openmpi headers.
Mar 15 2020
On 2020-03-15 23:40, Peter Jacobs wrote:Tried the most recent dstep but was unsuccessful... $ ../dstep/bin/dstep /usr/include/mpi/mpi.h -o mpi.d /usr/include/mpi/mpi.h:351:8: warning: a type renamed to 'ompi_status_public_t_' due to the collision with the symbol declared in /usr/include/mpi/mpi.h:364:37 dstep: an unknown error occurred: core.exception.AssertError dstep/translator/Type.d(32): Assertion failure ---------------- ??:? _d_assertp [0x5625185d8659] dstep/translator/Type.d:32 dstep.translator.Output.SourceNode dstep.translator.Type.translateType(dstep.translator.Context.Context, clang.Cursor.Cursor, clang.Type.Type, bool, bool) [0x5625185c1941] --- snip --- clang/Visitor.d:66 extern (C) clang.c.Index.CXChildVisitResult clang.Visitor.Visitor.visitorFunction(clang.c.Index.CXCursor, clang.c.Index.CXCursor, void*) [0x562518562feb] ??:? [0x7f7d196f7bff]Please report an issue here [1]. Seems to be similar to this issue [2] [1] https://github.com/jacob-carlborg/dstep/issues [2] https://github.com/jacob-carlborg/dstep/issues/249 -- /Jacob Carlborg
Mar 16 2020
On Friday, 13 March 2020 at 10:44:08 UTC, Andre Pany wrote:On Friday, 13 March 2020 at 08:08:17 UTC, Peter Jacobs wrote:It can translate some macros, but not all.Given the recent thread that mentioned dpp for generating an interface module for a C library, I thought that I would give it a go on the openmpi headers. I am working in LinuxMint system and after making a symbolic link for libclang.so, I have had some success but not complete success. [...]DPP cannot translate macros.Another problem is inlined functions.Yes. Technically I could add support for them by adding translation functions for all possible C AST nodes (I shudder to even consider C++ here), but it'd be quite a bit of work.While DPP generates their headers, their implementations are missing in the binary files. In both cases you have to reimplement the functions in D.Not quite - you can force a C compiler to emit code for them and link to it, using the dpp-translated declaration during compilation. It's how D supports calling C++ template functions now.
Mar 13 2020
On Friday, 13 March 2020 at 08:08:17 UTC, Peter Jacobs wrote:Given the recent thread that mentioned dpp for generating an interface module for a C library, I thought that I would give it a go on the openmpi headers. I am working in LinuxMint system and after making a symbolic link for libclang.so, I have had some success but not complete success. I have a module // mpi.dpp module mpi; #include <mpi.h> /** * Convert D's string[] args to a correct C char** argv. */ char** toArgv(string[] args) { auto argv = new char*[args.length + 1]; foreach(i, arg; args) { argv[i] = (arg.dup ~ '\0').ptr; } argv[args.length] = null; return argv.ptr; } enum MPI_DUMMY = 999; a main module // hello_d.d // hacked out of the hello_d example from the OpenMPI project. import std.stdio; import std.string; import std.array; import std.algorithm; import mpi; int main(string[] args) { int argc = cast(int)args.length; auto argv = args.toArgv(); int rank; int size; writeln("MPI_VERSION=", MPI_VERSION); MPI_Init(&argc, &argv); //MPI_Comm_rank(MPI_COMM_WORLD, &rank); //MPI_Comm_size(MPI_COMM_WORLD, &size); writefln("Hello, world, I am %d of %d", rank+1, size); writeln("MPI_DUMMY=", MPI_DUMMY); //MPI_Barrier(MPI_COMM_WORLD); MPI_Finalize(); return 0; } and a makefile hello_d : hello_d.d mpi.d dmd -ofhello_d hello_d.d mpi.d -L-lmpi mpi.d : mpi.dpp dub run dpp -- --include-path=/usr/include/openmpi \ --preprocess-only mpi.dpp The dpp program successfully generates an interface module mpi.d and the hello_d executable is built and it runs. However, not all of the needed elements for a real MPI program are present. For example, in the main module above, MPI_COMM_WORLD is not available so the lines calling MPI_Comm_rank, MPI_Comm_size and MPI_Barrier are commented out. It may be that I am not using dpp correctly, however, running dpp with the -- option gives a lot output that includes Cursor(MacroDefinition, "MPI_COMM_WORLD", Type(Invalid, "")) CAN SourceRange("/usr/include/openmpi/mpi.h", 1034:9, 1034:78) so I am guessing that dpp does not know how to compute its type and value. Thoughts or suggestions on how to proceed?Have you seen this project? https://github.com/1100110/OpenMPI As I think about this, OpenMPI is probably one of the most difficult libraries to deal with. It's such a complicated beast to build that you're supposed to use their wrapper compiler.
Mar 15 2020
On Monday, 16 March 2020 at 01:12:42 UTC, bachmeier wrote:On Friday, 13 March 2020 at 08:08:17 UTC, Peter Jacobs wrote:--- snip ---Given the recent thread that mentioned dpp for generating an interface module for a C library, I thought that I would give it a go on the openmpi headers. Thoughts or suggestions on how to proceed?Have you seen this project? https://github.com/1100110/OpenMPI As I think about this, OpenMPI is probably one of the most difficult libraries to deal with. It's such a complicated beast to build that you're supposed to use their wrapper compiler.Yes, indeed, we use that OpenMPI wrapper, actually the version at https://github.com/DlangScience/OpenMPI which has been maintained by John Colvin. A big thanks to both Jude Young and John Colvin. We have run our CFD flow solver across more than 1000 cores using MPI http://cfcfd.mechmining.uq.edu.au/eilmer/pdfs/cfh-seminar-oct-2019.pdf What prompted my experiments with dpp is that I have been getting reports that our code is not building with very recent editions of openmpi, presumably because the OpenMPI wrapper-generator code is surprised by the new header file content, and that I have seen dpp suggested by Atila Neves a couple of times.
Mar 16 2020
On Monday, 16 March 2020 at 10:40:09 UTC, Peter Jacobs wrote:On Monday, 16 March 2020 at 01:12:42 UTC, bachmeier wrote:Just to let interested parties know... I went with updating the interface in this project. It turned out to be a reasonably direct transcription of the MPI header into a D module. Now, it works for a range of modern openmpi versions.On Friday, 13 March 2020 at 08:08:17 UTC, Peter Jacobs wrote:--- snip ---Given the recent thread that mentioned dpp for generating an interface module for a C library, I thought that I would give it a go on the openmpi headers. Thoughts or suggestions on how to proceed?Have you seen this project? https://github.com/1100110/OpenMPI As I think about this, OpenMPI is probably one of the most difficult libraries to deal with. It's such a complicated beast to build that you're supposed to use their wrapper compiler.Yes, indeed, we use that OpenMPI wrapper, actually the version at https://github.com/DlangScience/OpenMPI which has been maintained by John Colvin.
Mar 21 2020