www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Using dpp on the openmpi headers

reply Peter Jacobs <peterj mech.uq.edu.au> writes:
Given the recent thread that mentioned dpp for generating an 
interface module for a C library, I thought that I would give it 
a go on the openmpi headers.  I am working in  LinuxMint system 
and after making a symbolic link for libclang.so, I have had some 
success but not complete success.

I have a module

// mpi.dpp

module mpi;

#include <mpi.h>

/**
  * Convert D's string[] args to a correct C char** argv.
  */
char** toArgv(string[] args)
{
     auto argv = new char*[args.length + 1];
     foreach(i, arg; args)
     {
         argv[i] = (arg.dup ~ '\0').ptr;
     }
     argv[args.length] = null;
     return argv.ptr;
}

enum MPI_DUMMY = 999;


a main module

// hello_d.d
// hacked out of the hello_d example from the OpenMPI project.

import std.stdio;
import std.string;
import std.array;
import std.algorithm;
import mpi;

int main(string[] args)
{
     int argc = cast(int)args.length;
     auto argv = args.toArgv();

     int rank;
     int size;
     writeln("MPI_VERSION=", MPI_VERSION);

     MPI_Init(&argc, &argv);
     //MPI_Comm_rank(MPI_COMM_WORLD, &rank);
     //MPI_Comm_size(MPI_COMM_WORLD, &size);
     writefln("Hello, world, I am %d of %d", rank+1, size);
     writeln("MPI_DUMMY=", MPI_DUMMY);
     //MPI_Barrier(MPI_COMM_WORLD);
     MPI_Finalize();

     return 0;
}

and a makefile



hello_d : hello_d.d mpi.d
	dmd -ofhello_d hello_d.d mpi.d -L-lmpi

mpi.d : mpi.dpp
	dub run dpp -- --include-path=/usr/include/openmpi \
		--preprocess-only mpi.dpp

The dpp program successfully generates an interface module mpi.d 
and the hello_d executable is built and it runs.  However, not 
all of the needed elements for a real MPI program are present.  
For example, in the main module above, MPI_COMM_WORLD is not 
available so the lines calling MPI_Comm_rank, MPI_Comm_size and 
MPI_Barrier are commented out.

It may be that I am not using dpp correctly, however, running dpp 
with the -- option gives a lot output that includes

Cursor(MacroDefinition, "MPI_COMM_WORLD", Type(Invalid, "")) CAN 
  SourceRange("/usr/include/openmpi/mpi.h", 1034:9, 1034:78)

so I am guessing that dpp does not know how to compute its type 
and value.

Thoughts or suggestions on how to proceed?
Mar 13
next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 13/03/2020 9:08 PM, Peter Jacobs wrote:
 /**
   * Convert D's string[] args to a correct C char** argv.
   */
 char** toArgv(string[] args)
 {
      auto argv = new char*[args.length + 1];
      foreach(i, arg; args)
      {
          argv[i] = (arg.dup ~ '\0').ptr;
      }
      argv[args.length] = null;
      return argv.ptr;
 }
https://dlang.org/phobos/core_runtime.html#.Runtime.cArgs
Mar 13
parent Peter Jacobs <peterj mech.uq.edu.au> writes:
On Friday, 13 March 2020 at 08:13:44 UTC, rikki cattermole wrote:
 https://dlang.org/phobos/core_runtime.html#.Runtime.cArgs
Oh, that's neat. There's so much that I don't know...
Mar 13
prev sibling next sibling parent reply Andre Pany <andre s-e-a-p.de> writes:
On Friday, 13 March 2020 at 08:08:17 UTC, Peter Jacobs wrote:
 Given the recent thread that mentioned dpp for generating an 
 interface module for a C library, I thought that I would give 
 it a go on the openmpi headers.  I am working in  LinuxMint 
 system and after making a symbolic link for libclang.so, I have 
 had some success but not complete success.

 [...]
DPP cannot translate macros. Another problem is inlined functions. While DPP generates their headers, their implementations are missing in the binary files. In both cases you have to reimplement the functions in D. Kind regards Andre
Mar 13
next sibling parent evilrat <evilrat666 gmail.com> writes:
On Friday, 13 March 2020 at 10:44:08 UTC, Andre Pany wrote:
 DPP cannot translate macros. Another problem is inlined 
 functions. While DPP generates their headers, their 
 implementations are missing in the binary files.
 In both cases you have to reimplement the functions in D.
With my crappy binding generator it might be possible to generate some of this, however it will likely just suck given its current shape. Especially compared to dpp. 100% natural pain & misery, enjoy. It is still might be somewhat useful to run it and gather the inlines. Maybe. https://github.com/Superbelko/ohmygentool
Mar 13
prev sibling next sibling parent reply bachmeier <no spam.net> writes:
On Friday, 13 March 2020 at 10:44:08 UTC, Andre Pany wrote:
 On Friday, 13 March 2020 at 08:08:17 UTC, Peter Jacobs wrote:
 Given the recent thread that mentioned dpp for generating an 
 interface module for a C library, I thought that I would give 
 it a go on the openmpi headers.  I am working in  LinuxMint 
 system and after making a symbolic link for libclang.so, I 
 have had some success but not complete success.

 [...]
DPP cannot translate macros. Another problem is inlined functions. While DPP generates their headers, their implementations are missing in the binary files. In both cases you have to reimplement the functions in D. Kind regards Andre
Some of that can be done by dstep: https://github.com/jacob-carlborg/dstep
Mar 13
parent reply Peter Jacobs <peterj mech.uq.edu.au> writes:
On Friday, 13 March 2020 at 13:00:32 UTC, bachmeier wrote:
 On Friday, 13 March 2020 at 10:44:08 UTC, Andre Pany wrote:
 On Friday, 13 March 2020 at 08:08:17 UTC, Peter Jacobs wrote:
 Given the recent thread that mentioned dpp for generating an 
 interface module for a C library, I thought that I would give 
 it a go on the openmpi headers.
Some of that can be done by dstep: https://github.com/jacob-carlborg/dstep
Tried the most recent dstep but was unsuccessful... $ ../dstep/bin/dstep /usr/include/mpi/mpi.h -o mpi.d /usr/include/mpi/mpi.h:351:8: warning: a type renamed to 'ompi_status_public_t_' due to the collision with the symbol declared in /usr/include/mpi/mpi.h:364:37 dstep: an unknown error occurred: core.exception.AssertError dstep/translator/Type.d(32): Assertion failure ---------------- ??:? _d_assertp [0x5625185d8659] dstep/translator/Type.d:32 dstep.translator.Output.SourceNode dstep.translator.Type.translateType(dstep.translator.Context.Context, clang.Cursor.Cursor, clang.Type.Type, bool, bool) [0x5625185c1941] --- snip --- clang/Visitor.d:66 extern (C) clang.c.Index.CXChildVisitResult clang.Visitor.Visitor.visitorFunction(clang.c.Index.CXCursor, clang.c.Index.CXCursor, void*) [0x562518562feb] ??:? [0x7f7d196f7bff]
Mar 15
parent Jacob Carlborg <doob me.com> writes:
On 2020-03-15 23:40, Peter Jacobs wrote:

 Tried the most recent dstep but was unsuccessful...
 
 $ ../dstep/bin/dstep /usr/include/mpi/mpi.h -o mpi.d
 /usr/include/mpi/mpi.h:351:8: warning: a type renamed to 
 'ompi_status_public_t_' due to the collision with the symbol declared in 
 /usr/include/mpi/mpi.h:364:37
 dstep: an unknown error occurred: 
 core.exception.AssertError dstep/translator/Type.d(32): Assertion failure
 ----------------
 ??:? _d_assertp [0x5625185d8659]
 dstep/translator/Type.d:32 dstep.translator.Output.SourceNode 
 dstep.translator.Type.translateType(dstep.translator.Context.Context, 
 clang.Cursor.Cursor, clang.Type.Type, bool, bool) [0x5625185c1941]
 
 --- snip ---
 
 clang/Visitor.d:66 extern (C) clang.c.Index.CXChildVisitResult 
 clang.Visitor.Visitor.visitorFunction(clang.c.Index.CXCursor, 
 clang.c.Index.CXCursor, void*) [0x562518562feb]
 ??:? [0x7f7d196f7bff]
Please report an issue here [1]. Seems to be similar to this issue [2] [1] https://github.com/jacob-carlborg/dstep/issues [2] https://github.com/jacob-carlborg/dstep/issues/249 -- /Jacob Carlborg
Mar 16
prev sibling parent Atila Neves <atila.neves gmail.com> writes:
On Friday, 13 March 2020 at 10:44:08 UTC, Andre Pany wrote:
 On Friday, 13 March 2020 at 08:08:17 UTC, Peter Jacobs wrote:
 Given the recent thread that mentioned dpp for generating an 
 interface module for a C library, I thought that I would give 
 it a go on the openmpi headers.  I am working in  LinuxMint 
 system and after making a symbolic link for libclang.so, I 
 have had some success but not complete success.

 [...]
DPP cannot translate macros.
It can translate some macros, but not all.
 Another problem is inlined functions.
Yes. Technically I could add support for them by adding translation functions for all possible C AST nodes (I shudder to even consider C++ here), but it'd be quite a bit of work.
 While DPP generates their headers, their implementations are 
 missing in the binary files.
 In both cases you have to reimplement the functions in D.
Not quite - you can force a C compiler to emit code for them and link to it, using the dpp-translated declaration during compilation. It's how D supports calling C++ template functions now.
Mar 13
prev sibling parent reply bachmeier <no spam.net> writes:
On Friday, 13 March 2020 at 08:08:17 UTC, Peter Jacobs wrote:
 Given the recent thread that mentioned dpp for generating an 
 interface module for a C library, I thought that I would give 
 it a go on the openmpi headers.  I am working in  LinuxMint 
 system and after making a symbolic link for libclang.so, I have 
 had some success but not complete success.

 I have a module

 // mpi.dpp

 module mpi;

 #include <mpi.h>

 /**
  * Convert D's string[] args to a correct C char** argv.
  */
 char** toArgv(string[] args)
 {
     auto argv = new char*[args.length + 1];
     foreach(i, arg; args)
     {
         argv[i] = (arg.dup ~ '\0').ptr;
     }
     argv[args.length] = null;
     return argv.ptr;
 }

 enum MPI_DUMMY = 999;


 a main module

 // hello_d.d
 // hacked out of the hello_d example from the OpenMPI project.

 import std.stdio;
 import std.string;
 import std.array;
 import std.algorithm;
 import mpi;

 int main(string[] args)
 {
     int argc = cast(int)args.length;
     auto argv = args.toArgv();

     int rank;
     int size;
     writeln("MPI_VERSION=", MPI_VERSION);

     MPI_Init(&argc, &argv);
     //MPI_Comm_rank(MPI_COMM_WORLD, &rank);
     //MPI_Comm_size(MPI_COMM_WORLD, &size);
     writefln("Hello, world, I am %d of %d", rank+1, size);
     writeln("MPI_DUMMY=", MPI_DUMMY);
     //MPI_Barrier(MPI_COMM_WORLD);
     MPI_Finalize();

     return 0;
 }

 and a makefile



 hello_d : hello_d.d mpi.d
 	dmd -ofhello_d hello_d.d mpi.d -L-lmpi

 mpi.d : mpi.dpp
 	dub run dpp -- --include-path=/usr/include/openmpi \
 		--preprocess-only mpi.dpp

 The dpp program successfully generates an interface module 
 mpi.d and the hello_d executable is built and it runs.  
 However, not all of the needed elements for a real MPI program 
 are present.  For example, in the main module above, 
 MPI_COMM_WORLD is not available so the lines calling 
 MPI_Comm_rank, MPI_Comm_size and MPI_Barrier are commented out.

 It may be that I am not using dpp correctly, however, running 
 dpp with the -- option gives a lot output that includes

 Cursor(MacroDefinition, "MPI_COMM_WORLD", Type(Invalid, "")) 
 CAN   SourceRange("/usr/include/openmpi/mpi.h", 1034:9, 1034:78)

 so I am guessing that dpp does not know how to compute its type 
 and value.

 Thoughts or suggestions on how to proceed?
Have you seen this project? https://github.com/1100110/OpenMPI As I think about this, OpenMPI is probably one of the most difficult libraries to deal with. It's such a complicated beast to build that you're supposed to use their wrapper compiler.
Mar 15
parent reply Peter Jacobs <peterj mech.uq.edu.au> writes:
On Monday, 16 March 2020 at 01:12:42 UTC, bachmeier wrote:
 On Friday, 13 March 2020 at 08:08:17 UTC, Peter Jacobs wrote:
 Given the recent thread that mentioned dpp for generating an 
 interface module for a C library, I thought that I would give 
 it a go on the openmpi headers.
 Thoughts or suggestions on how to proceed?
--- snip ---
 Have you seen this project?

 https://github.com/1100110/OpenMPI

 As I think about this, OpenMPI is probably one of the most 
 difficult libraries to deal with. It's such a complicated beast 
 to build that you're supposed to use their wrapper compiler.
Yes, indeed, we use that OpenMPI wrapper, actually the version at https://github.com/DlangScience/OpenMPI which has been maintained by John Colvin. A big thanks to both Jude Young and John Colvin. We have run our CFD flow solver across more than 1000 cores using MPI http://cfcfd.mechmining.uq.edu.au/eilmer/pdfs/cfh-seminar-oct-2019.pdf What prompted my experiments with dpp is that I have been getting reports that our code is not building with very recent editions of openmpi, presumably because the OpenMPI wrapper-generator code is surprised by the new header file content, and that I have seen dpp suggested by Atila Neves a couple of times.
Mar 16
parent Peter Jacobs <peterj mech.uq.edu.au> writes:
On Monday, 16 March 2020 at 10:40:09 UTC, Peter Jacobs wrote:
 On Monday, 16 March 2020 at 01:12:42 UTC, bachmeier wrote:
 On Friday, 13 March 2020 at 08:08:17 UTC, Peter Jacobs wrote:
 Given the recent thread that mentioned dpp for generating an 
 interface module for a C library, I thought that I would give 
 it a go on the openmpi headers.
 Thoughts or suggestions on how to proceed?
--- snip ---
 Have you seen this project?

 https://github.com/1100110/OpenMPI

 As I think about this, OpenMPI is probably one of the most 
 difficult libraries to deal with. It's such a complicated 
 beast to build that you're supposed to use their wrapper 
 compiler.
Yes, indeed, we use that OpenMPI wrapper, actually the version at https://github.com/DlangScience/OpenMPI which has been maintained by John Colvin.
Just to let interested parties know... I went with updating the interface in this project. It turned out to be a reasonably direct transcription of the MPI header into a D module. Now, it works for a range of modern openmpi versions.
Mar 21