www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - Linker issues with struct postblit

reply Thomas Gregory <charles.gregory940 topper.wku.edu> writes:
I am a maintainer of the 
[dhtslib](https://github.com/blachlylab/dhtslib) package and I 
have been running into issues with a new implementation of 
reference counting we are using.

Below is the implementation (which is basically our replacement 
for `RefCounted`).
```d
/// Template struct that wraps an htslib
/// pointer and reference counts it and then
/// destroys with destroyFun when it goes
/// truly out of scope
struct SafeHtslibPtr(T, alias destroyFun)
if(!isPointer!T && isSomeFunction!destroyFun)
{
      safe  nogc nothrow:

     /// data pointer
     T * ptr;
     /// reference counting
     shared int* refct;

     /// initialized?
     bool initialized;

     /// ctor that respects scope
     this(T * rawPtr)  trusted return scope
     {
         this.ptr = rawPtr;
         this.refct = cast(shared int *) calloc(int.sizeof,1);
         (*this.refct) = 1;
         this.initialized = true;
     }

     /// postblit that respects scope
     this(this)  trusted return scope
     {
         if(initialized)atomicOp!"+="(*this.refct, 1);
     }

     /// allow SafeHtslibPtr to be used as
     /// underlying ptr type
     alias getRef this;

     /// get underlying data pointer
      property nothrow pure  nogc
     ref inout(T*) getRef() inout return
     {
         return ptr;
     }

     /// take ownership of underlying data pointer
      property nothrow pure  nogc
     T* moveRef()
     {
         T * ptr;
         move(this.getRef, ptr);
         return ptr;
     }

     /// dtor that respects scope
     ~this()  trusted return scope
     {

         if(!this.initialized) return;
         if(atomicOp!"-="(*this.refct, 1)) return;
         if(this.ptr){
             free(cast(int*)this.refct);
             /// if destroy function return is void
             /// just destroy
             /// else if return is int
             /// destroy then check return value
             /// else don't compile
             static if(is(ReturnType!destroyFun == void))
                 destroyFun(this.ptr);
             else static if(is(ReturnType!destroyFun == int))
             {
                 auto err = destroyFun(this.ptr);
                 if(err != 0)
                     hts_log_errorNoGC!__FUNCTION__("Couldn't 
destroy/close "~T.stringof~" * data using function 
"~__traits(identifier, destroyFun));
             }else{
                 static assert(0, "HtslibPtr doesn't recognize 
destroy function return type");
             }
         }
     }
}
```
This can be used as such to reference count a pointer created 
from the c library [htslib](https://github.com/samtools/htslib) 
as such:
```d
/// bam1_t is a struct from c bindings
/// bam_destroy1 is a function to clean up a bam1_t *
/// that is created from the c bindings
alias Bam1 = SafeHtslibPtr!(bam1_t, bam_destroy1);
auto b = Bam1(bam_init1());
```

The issue presents with `SAMRecord`:
```d
struct SAMRecord
{
     /// Backing SAM/BAM row record
     Bam1 b;

     /// Corresponding SAM/BAM header data
     SAMHeader h;

```
dhtslib itself builds fine on both dmd and ldc compilers but when 
it is used as a dependency it seems to have issues building on 
any compiler that is not ldc > v1.24.0:
```
_D39TypeInfo_S7dhtslib3sam6record9SAMRecord6__initZ: error: 
undefined reference to 
`_D7dhtslib3sam6record9SAMRecord15__fieldPostblitMFNbNiNlNeZv'
```
Though I only experience this when trying to create an array of 
`SAMRecord`s.

One solution I have found is using `std.array.Appender` instead 
of arrays.
Another solution I have found is to define an explicit postblit 
for `SAMReader`:
```
this(this)
{
     this.h = h;
     this.b = b;
}
```

Looking through ldc changelogs, the closest thing I could 
attribute this to is this change for ldc-1.25.0:

 - Struct TypeInfos are emitted into referencing object files 
 only, and special TypeInfo member functions into the owning 



I suspect this is something to do with the alias'd function in 
`SafeHtslibPtr`. Is there something I should be doing differently?
Oct 27 2021
parent reply Imperatorn <johan_forsberg_86 hotmail.com> writes:
On Thursday, 28 October 2021 at 01:39:10 UTC, Thomas Gregory 
wrote:

 I am a maintainer of the 
 [dhtslib](https://github.com/blachlylab/dhtslib) package and I 
 have been running into issues with a new implementation of 
 reference counting we are using.

 [...]


Postblit?
Oct 29 2021
parent reply Stanislav Blinov <stanislav.blinov gmail.com> writes:
On Friday, 29 October 2021 at 11:05:14 UTC, Imperatorn wrote:

 On Thursday, 28 October 2021 at 01:39:10 UTC, Thomas Gregory 
 wrote:

 I am a maintainer of the 
 [dhtslib](https://github.com/blachlylab/dhtslib) package and I 
 have been running into issues with a new implementation of 
 reference counting we are using.

 [...]


 Postblit?


https://dlang.org/spec/struct.html#struct-postblit
Oct 29 2021
parent reply James Blachly <james.blachly gmail.com> writes:
On 10/29/21 7:10 AM, Stanislav Blinov wrote:

 On Friday, 29 October 2021 at 11:05:14 UTC, Imperatorn wrote:

 On Thursday, 28 October 2021 at 01:39:10 UTC, Thomas Gregory wrote:

 I am a maintainer of the 
 [dhtslib](https://github.com/blachlylab/dhtslib) package and I have 
 been running into issues with a new implementation of reference 
 counting we are using.

 [...]


 Postblit?


 
 https://dlang.org/spec/struct.html#struct-postblit


I imagine Imperatorn is quite familiar with postblit and was pointing 
out that it is strange to use postblit constructor, which is deprecated, 
in a "new implementation".

The original post was long and really buried the lede, but OP was 
pointing out what looks like a compiler bug: a linker error referencing 
a postblit symbol only shows up when compiling with DMD or with LDC 
earlier than 1.25.

Can someone give some insight?
Oct 30 2021
parent Thomas Gregory <charles.gregory940 topper.wku.edu> writes:
On Sunday, 31 October 2021 at 03:51:49 UTC, James Blachly wrote:

 On 10/29/21 7:10 AM, Stanislav Blinov wrote:

 On Friday, 29 October 2021 at 11:05:14 UTC, Imperatorn wrote:

 On Thursday, 28 October 2021 at 01:39:10 UTC, Thomas Gregory 
 wrote:

 I am a maintainer of the 
 [dhtslib](https://github.com/blachlylab/dhtslib) package and 
 I have been running into issues with a new implementation of 
 reference counting we are using.

 [...]


 Postblit?


 
 https://dlang.org/spec/struct.html#struct-postblit


 I imagine Imperatorn is quite familiar with postblit and was 
 pointing out that it is strange to use postblit constructor, 
 which is deprecated, in a "new implementation".

 The original post was long and really buried the lede, but OP 
 was pointing out what looks like a compiler bug: a linker error 
 referencing a postblit symbol only shows up when compiling with 
 DMD or with LDC earlier than 1.25.

 Can someone give some insight?


I apologize for the long-winded question.


 Postblit?






Yes, one solution would be to add a postblit.

However, the compiler should be able to generate default postblit 
and it does. I am only running into issues linking, which leads 
me to believe this is a compiler or language bug (fixed as of 
ldc-1.25).

Is there some way to to avoid writing an explicit postblit for 
every struct that uses `SafeHtslibPtr`? Many of `dhtslib`'s 
structs wrap c type pointers that are reference counted and owned 
by `SafeHtslibPtr` like so:

```D
// wrappedCPtr is an alias defined in another module as:
alias wrappedCPtr = SafeHtslibPtr!(c_type, destroy_c_type_fun);

// It is imported here
struct WrapperTypeName
{
     /// Backing C type pointer
     wrappedCPtr b;
...
}
```
Nov 01 2021