www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Implementing Half Floats in D

reply Walter Bright <newshound2 digitalmars.com> writes:
http://www.drdobbs.com/cpp/implementing-half-floats-in-d/240146674

Anyone care to do the reddit honors?
Jan 28 2013
next sibling parent reply "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Monday, 28 January 2013 at 23:11:11 UTC, Walter Bright wrote:

 http://www.drdobbs.com/cpp/implementing-half-floats-in-d/240146674

 Anyone care to do the reddit honors?



[quote]
   and crushed back down to 16 bytes for storage.
[/quote]

  Should be bits. Otherwise it looks really well done.
Jan 28 2013
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/28/2013 3:30 PM, Era Scarecrow wrote:

 On Monday, 28 January 2013 at 23:11:11 UTC, Walter Bright wrote:

 http://www.drdobbs.com/cpp/implementing-half-floats-in-d/240146674

 Anyone care to do the reddit honors?



 [quote]
    and crushed back down to 16 bytes for storage.
 [/quote]

   Should be bits. Otherwise it looks really well done.



thank you. Sorry that didn't get caught in review!
Jan 28 2013
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 01/29/2013 11:40 AM, TommiT wrote:

 On Tuesday, 29 January 2013 at 08:48:24 UTC, Simen Kjaeraas wrote:

 Except that's not a literal. 1.3.hf would be a function call, while
 hf!1.3f
 is a template instantiation.



 Can't help but point out, that some time ago, I made a suggestion to add
 a new function attribute to the language that would mean "evaluate any
 call to this function at compile time whenever it's possible". That
 would enable literals like:

 1.3.hf
 and
 ComplexToConstruct(2.5, 7.5)

 Given:
  property HalfFloat hf(float v)  aggressive_ctfe;
 and...
 struct ComplexToConstruct
 {
      this(float x, float y)  aggressive_ctfe;
 }



I'd call it  constfold.
Jan 29 2013
prev sibling next sibling parent reply Justin Whear <justin economicmodeling.com> writes:
On Mon, 28 Jan 2013 15:11:06 -0800, Walter Bright wrote:


 http://www.drdobbs.com/cpp/implementing-half-floats-in-d/240146674
 
 Anyone care to do the reddit honors?



Posted to /r/d_language, perhaps a x-post to /r/programming would be in 
order?

http://www.reddit.com/r/d_language/comments/17gkrk/
implementing_half_floats_in_d_dr_dobbs/
Jan 28 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/28/13 6:40 PM, Justin Whear wrote:

 On Mon, 28 Jan 2013 15:11:06 -0800, Walter Bright wrote:


 http://www.drdobbs.com/cpp/implementing-half-floats-in-d/240146674

 Anyone care to do the reddit honors?



 Posted to /r/d_language, perhaps a x-post to /r/programming would be in
 order?

 http://www.reddit.com/r/d_language/comments/17gkrk/
 implementing_half_floats_in_d_dr_dobbs/



Not sure what's going on, I tried to submit and got "already submitted" 
and then got redirected to 
http://www.reddit.com/r/programming/comments/16ycws/implementing_half_floats_in_d/

Hoever, I can't find the post in http://www.reddit.com/r/programming/new/


Andrei
Jan 28 2013
parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/28/2013 3:50 PM, Andrei Alexandrescu wrote:

 Not sure what's going on, I tried to submit and got "already submitted" and
then
 got redirected to
 http://www.reddit.com/r/programming/comments/16ycws/implementing_half_floats_in_d/



8 days old, durn, it's sunsetted.
Jan 28 2013
prev sibling parent Justin Whear <justin economicmodeling.com> writes:
On Mon, 28 Jan 2013 18:50:22 -0500, Andrei Alexandrescu wrote:


 On 1/28/13 6:40 PM, Justin Whear wrote:

 On Mon, 28 Jan 2013 15:11:06 -0800, Walter Bright wrote:


 http://www.drdobbs.com/cpp/implementing-half-floats-in-d/240146674

 Anyone care to do the reddit honors?



 Posted to /r/d_language, perhaps a x-post to /r/programming would be in
 order?

 http://www.reddit.com/r/d_language/comments/17gkrk/
 implementing_half_floats_in_d_dr_dobbs/



 Not sure what's going on, I tried to submit and got "already submitted"
 and then got redirected to
 http://www.reddit.com/r/programming/comments/16ycws/




 
 Hoever, I can't find the post in
 http://www.reddit.com/r/programming/new/
 
 
 Andrei



Possibly this: http://code.reddit.com/wiki/help/
faq#Whyisntmysubmissioncommentshowingup

Walter is an admin; maybe he can check on it?
Jan 28 2013
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Walter Bright:


 http://www.drdobbs.com/cpp/implementing-half-floats-in-d/240146674



 From the article:


Built-in types tend to run faster because the optimizer can take 
advantage of mathematical identities,<



Generally in D we like the compiler to verify user-applied 
annotations like pure and const. For the compiler of a normal 
language today it's probably too much difficult to prove the 
theorems coming from those identities on user defined types. But 
if the compiler assumes the programmer to be right on this 
(because such types are usually defined in well reviewed 
libraries), then I think it's not too much hard to invent a small 
set of  annotations that attached to a user defined struct tell 
the compiler to allow some optimizations typical of integral 
numbers, floating point numbers, complex numbers, gaussian 
integers, quaternions, octonions and few others, according to 
them having associative, commutative, etc, properties, or not 
having them.

Bye,
bearophile
Jan 28 2013
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Jan 29, 2013 at 01:44:20AM +0100, bearophile wrote:

 Walter Bright:
 

http://www.drdobbs.com/cpp/implementing-half-floats-in-d/240146674



 From the article:
 

Built-in types tend to run faster because the optimizer can take
advantage of mathematical identities,<



 Generally in D we like the compiler to verify user-applied
 annotations like pure and const. For the compiler of a normal
 language today it's probably too much difficult to prove the
 theorems coming from those identities on user defined types. But if
 the compiler assumes the programmer to be right on this (because
 such types are usually defined in well reviewed libraries), then I
 think it's not too much hard to invent a small set of  annotations
 that attached to a user defined struct tell the compiler to allow
 some optimizations typical of integral numbers, floating point
 numbers, complex numbers, gaussian integers, quaternions, octonions
 and few others, according to them having associative, commutative,
 etc, properties, or not having them.




+1.  I've always wanted a language capable of user-specified
optimizations for custom types. This would be very useful on things like
matrix types, which would otherwise require heavy trickery with
expression templates in order to produce efficient code.


T

-- 
May you live all the days of your life. -- Jonathan Swift
Jan 28 2013
prev sibling next sibling parent "Tove" <tove fransson.se> writes:
On Monday, 28 January 2013 at 23:58:40 UTC, Walter Bright wrote:

 On 1/28/2013 3:30 PM, Era Scarecrow wrote:

 On Monday, 28 January 2013 at 23:11:11 UTC, Walter Bright 
 wrote:

 http://www.drdobbs.com/cpp/implementing-half-floats-in-d/240146674

 Anyone care to do the reddit honors?



 [quote]
   and crushed back down to 16 bytes for storage.
 [/quote]

  Should be bits. Otherwise it looks really well done.



 thank you. Sorry that didn't get caught in review!



"HalfFloat h = hf!1.3f;"

Maybe you could also demonstrate that it's possible to implement 
another literal syntax?
HalfFloat h = 1.3.hf;

some people will prefer that for sure.
Jan 28 2013
prev sibling next sibling parent "Simen Kjaeraas" <simen.kjaras gmail.com> writes:
On 2013-49-29 08:01, Tove <tove fransson.se> wrote:


 On Monday, 28 January 2013 at 23:58:40 UTC, Walter Bright wrote:

 On 1/28/2013 3:30 PM, Era Scarecrow wrote:

 On Monday, 28 January 2013 at 23:11:11 UTC, Walter Bright wrote:

 http://www.drdobbs.com/cpp/implementing-half-floats-in-d/240146674

 Anyone care to do the reddit honors?



 [quote]
   and crushed back down to 16 bytes for storage.
 [/quote]

  Should be bits. Otherwise it looks really well done.



 thank you. Sorry that didn't get caught in review!



 "HalfFloat h = hf!1.3f;"

 Maybe you could also demonstrate that it's possible to implement another  
 literal syntax?
 HalfFloat h = 1.3.hf;

 some people will prefer that for sure.



Except that's not a literal. 1.3.hf would be a function call, while hf!1.3f
is a template instantiation.

-- 
Simen
Jan 29 2013
prev sibling next sibling parent "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Tuesday, 29 January 2013 at 08:48:24 UTC, Simen Kjaeraas wrote:

 On 2013-49-29 08:01, Tove <tove fransson.se> wrote:

 Maybe you could also demonstrate that it's possible to 
 implement another literal syntax?
 HalfFloat h = 1.3.hf;

 some people will prefer that for sure.



 Except that's not a literal. 1.3.hf would be a function call, 
 while hf!1.3f is a template instantiation.



  Unless it's an enum or something that requires it to know it 
during compile-time.

  enum h = 1.3.hf; //potentially correct

  But I doubt it would be that useful. More likely during 
optimization static values would be equal to it.
Jan 29 2013
prev sibling next sibling parent "TommiT" <tommitissari hotmail.com> writes:
On Tuesday, 29 January 2013 at 08:48:24 UTC, Simen Kjaeraas wrote:

 Except that's not a literal. 1.3.hf would be a function call, 
 while hf!1.3f
 is a template instantiation.



Can't help but point out, that some time ago, I made a suggestion 
to add a new function attribute to the language that would mean 
"evaluate any call to this function at compile time whenever it's 
possible". That would enable literals like:

1.3.hf
and
ComplexToConstruct(2.5, 7.5)

Given:
 property HalfFloat hf(float v)  aggressive_ctfe;
and...
struct ComplexToConstruct
{
     this(float x, float y)  aggressive_ctfe;
}
Jan 29 2013
prev sibling next sibling parent reply "Stephan" <stephan_schiffels mac.com> writes:
On Monday, 28 January 2013 at 23:11:11 UTC, Walter Bright wrote:

 http://www.drdobbs.com/cpp/implementing-half-floats-in-d/240146674

 Anyone care to do the reddit honors?



Very nice. I learnt quite a bit there.
btw:
"In order to restrict it to only accepting floats, turn it into a 
template and check the type with static if:" - you actually use a 
static assert. But while we're at it: Why *don't* you use a 
signature constraint with if? I feel it would be more fitting to 
constrain the constructor usage.

Stephan
Jan 29 2013
parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/29/2013 2:43 AM, Stephan wrote:

 Why *don't* you use a signature constraint with if?



To get a custom error message.
Jan 29 2013
prev sibling next sibling parent "TommiT" <tommitissari hotmail.com> writes:
On Tuesday, 29 January 2013 at 11:26:00 UTC, Timon Gehr wrote:

 I'd call it  constfold.



That's a good name. Descriptive and short.

I'm not going to try to push this suggestion any harder, since it 
got no support on the first time I tried it. But I'd like to 
mention just one thing that came to mind:

The generic solution without any extra function attributes:

template ct(alias expr)
{
     enum ct = expr;
}

Literal of type TypeWithNonTrivialConstructor:

ct!(TypeWithNonTrivialConstructor(2.5, 7.5))

The above solution throws the responsibility of the correct use 
of TypeWithNonTrivialConstructor on the end-user. Whereas a 
 constfold attribute on the constructor of 
TypeWithNonTrivialConstructor automates the correct behaviour.
Jan 29 2013
prev sibling next sibling parent reply "Don" <don nospam.com> writes:
On Monday, 28 January 2013 at 23:11:11 UTC, Walter Bright wrote:

 http://www.drdobbs.com/cpp/implementing-half-floats-in-d/240146674




Since it got lost in the old thread on this topic, I'll repost my 
versions of floatToshort and shortToFloat, which are extremely 
fast (no unpredictable branches, no lookup tables) and respect 
the current rounding mode:

-----------------

float shortToFloat(ushort s)
{
     // note this is a signed shift, so sign bit gets smeared all 
the way into the int bit!
     uint u = ((cast(int)cast(short)s) << 13);

     if ( (s & EXPMASK) == 0 )
     {   // Subnormal or 0.
         // The simple conversion is wrong in two ways:
         // (1) it's added an implicit bit. This has value 0x1p-15.
         // (2) the mantissa bits got shifted along to make space 
for the hidden bit.
         //     So we need to multiply the result by 2.
         // Note that 0x3880_0000 means 0x1p-14.

         uint v = (u & 0x0FFF_FFFF ) + 0x3880_0000;
         float f = *cast(float *)&v - 0x1p-14;
         u = (u & 0x8000_0000) | *cast(uint *)&f;
         return *cast(float *)&u;
     }

     u = (u & 0x8FFF_FFFF) + 0x3800_0000;

     if ( (s & EXPMASK) == EXPMASK )
     {   // infinity or NaN
         u |= FEXPMASK;
     }
     return *cast(float *)&u;
}


-----------------
NOTE: this only works on 64-bit runtime, for 32bit or CTFE with 
80-bit intermediates, the constants need to be changed. 
Unfortunately I don't know of a nice way to detect the size of 
the intermediates.
-----

ushort floatToShort(float f)
{
     // Remember the sign
     uint x = *cast(uint *)&f;

     ushort sgn = (x >> 16) & 0x8000;

     // Need to treat NaN and Inf specially, otherwise the
     // mantissa shortening step would generate a new NaN.
     if ( (x & FEXPMASK) == FEXPMASK)
         return ( (x >> 13) & 0x7FFF) | sgn;

     // Shorten the mantissa, rounding it according to the current 
rounding mode

     f = (f * (1.0f + 0x1p-13f) -f) * 0x1p13f;

     // Force large numbers to overflow my moving near float.max

     f *= 0x1p112f;
     f *= 0x1p-112f; // Then undo it

     // Force small numbers to underflow, and shift into position

     f *= 0x1p-112f;

     uint u = *cast(uint *)&f;

     return ((u>>13) & 0x7FFF) | sgn;
}
Jan 29 2013
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/29/2013 4:31 AM, Don wrote:

 Since it got lost in the old thread on this topic, I'll repost my versions of
 floatToshort and shortToFloat, which are extremely fast (no unpredictable
 branches, no lookup tables) and respect the current rounding mode:



I was waiting on incorporating your improvements until after std.halffloat was 
merged into Phobos. This is because the old ones were easier to understand, and 
I wanted them in the repository history.
Jan 29 2013
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-01-29 20:06, Walter Bright wrote:


 I was waiting on incorporating your improvements until after
 std.halffloat was merged into Phobos. This is because the old ones were
 easier to understand, and I wanted them in the repository history.



Oh, please no, not another module at the "std" level.

-- 
/Jacob Carlborg
Jan 29 2013
parent reply Dejan Lekic <dejan.lekic gmail.com> writes:
Jacob Carlborg wrote:


 On 2013-01-29 20:06, Walter Bright wrote:
 

 I was waiting on incorporating your improvements until after
 std.halffloat was merged into Phobos. This is because the old ones were
 easier to understand, and I wanted them in the repository history.



 Oh, please no, not another module at the "std" level.
 



+11111111
I can't agree more to this, Jacob...
Seriously people... "std.halffloat" ???

-- 
Dejan Lekic
dejan.lekic (a) gmail.com
http://dejan.lekic.org
Jan 29 2013
parent 1100110 <0b1100110 gmail.com> writes:
On 01/30/2013 05:27 PM, Era Scarecrow wrote:

 On Tuesday, 29 January 2013 at 22:17:30 UTC, Dejan Lekic wrote:

 Jacob Carlborg wrote:

 Oh, please no, not another module at the "std" level.



 +11111111
 I can't agree more to this, Jacob...
 Seriously people... "std.halffloat" ???



 I'll agree too. HalfFloats are more a special case with certain circles
 of programmers that need those. The majority of programmers (95%+) won't
 care about half floats, nor would it be needed; It would be akin to
 making std.bitop instad of core.bitop.

 You could HalfFloats to std.math; Right next to cos, sin and tan.
 std.mathspecial doesn't seem like it would be closely enough tied to
 make it an option.

 Alternatively make a std.customtypes for related misc types like this as
 they are added to phobos.



std.typecons?

Seriously,  std.type.cons, tuple, halffloat, bigint, etc...
Jan 30 2013
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/29/2013 2:27 PM, Iain Buclaw wrote:

 I'm sure this was mentioned before, but what does halffloat have over
 std.numeric.CustomFloat?



There was a long discussion about it, I don't remember if it was in the ng or
on 
github.
Jan 29 2013
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/31/13 5:18 AM, Don wrote:

 std.numeric is not superficially flawed, it's fundamentally flawed. What
 is it for? What is its theme? The problem is, std.numeric is one of the
 few good names which are left as a possible package name, after C
 insulted the mathematical community by creating a module called 'math'.



Guilty as charged. I've put stuff in std.numeric as I was working on my 
thesis. I recall you added some stuff there too. As I'm sure you 
remember the state of D in 2007 was rather different than that of today. 
Overall no need to get agitated here, we're all on the same boat and 
aiming for the same shore.

Let's see what we have there:

entropy
CustomFloat
kullbackLeiblerDivergence
Fft
gapWeightedSimilarityIncremental
gapWeightedSimilarity
gapWeightedSimilarityNormalized
FPTemporary
findRoot
euclideanDistance
dotProduct
cosineSimilarity
gcd
jensenShannonDivergence
normalize
secantMethod

The general theme is obvious - numeric algorithms and data structures. 
Many are obvious and with obvious utility to one interested in numerics: 
entropy, various distance and similarity measures. I think you wrote 
findRoot.

The gapWeightedSimilarity algorithms are string kernels. They are 
somewhat niche but quite powerful to anyone interested in string 
similarity (technically they are string edit distance on steroids). They 
might belong in std.string but I figured they have enough numeric 
algorithm flavor to put them in there.

So let's itemize the grievances and see how we can sort this out.


Thanks,

Andrei
Jan 31 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/31/13 10:38 AM, Don wrote:

 The basic problem is that there are hundreds of potential numeric
 algorithms and data structures of equal importance to these ones. In
 fact, the total number of mathematical algorithms is probably a
 substantial fraction of the total algorithms in computer science!

 Even a module which contained only FFT, could be quite large, once it
 included all the important related transforms.



A possible solution would be to make std.numeric a knot for several 
others, or convert it to a package.


 I'm not sure that we can solve this without addressing the high-level
 question: What is the scope of Phobos?

 How big will it eventually get? Twice its current size? Ten times? A
 hundred times?



I think the current trend is to go for large libraries.


 Both SmallPhobos and LargePhobos are reasonable, but we do have to pick
 one. Currently we have aspects of both approaches, but they aren't
 compatible.



LargePhobos. Agreed that we need to package-ize some of the current modules.


Andrei
Jan 31 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-02-01 20:15, Paul D. Anderson wrote:


 Another consideration is the process for a package to become "standard".
 Current practice is to implement a package and go through a review in
 this forum. That's a good thing, but I'd like to see a more interactive
 approach, where the community was involved sooner:

 1. A package is proposed for future inclusion in the library. Presumably
 this is done by someone with an interest in the subject, ideally this is
 someone who wants to spearhead the effort and may already have a draft
 implementation.

 2. If there's enough interest to move forward, community members could
 bicker about all the bikeshed stuff: structures, methods,
 implementations, API, etc. until there's enough agreement about what the
 result should look like.

 3. Interested parties could collaborate on the implementation and release.

 4. There would have to be time limits. Many (most?) D projects start
 with a bang but then fizzle out before they're complete. There would
 need to be (monthly?) progress reviews. If the discussion or the
 implementation has dwindled, a reconsideration of the project could
 occur. If unsuccessful, the project would be moved off the active list.
 If there is renewed interest or if somebody else wants to take a crack
 at it the previous work can be a resource, but the project should
 probably go back to step 1.

 You all see the problem with this: the very short attention span of the
 D community. But we've got a good start between the DIP process and the
 review queue. The only thing really different would be the progress
 reviews, which would hopefully prevent projects falling of the end of
 the earth.

 This is just one possibility and there are probably better ways to do
 this. But I think some sort of maintenance of active projects is needed.



Isn't this what the whole community is about and what we're already 
doing. Just not very formal or official.

-- 
/Jacob Carlborg
Feb 01 2013
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-02-02 14:42, Joseph Rushton Wakeling wrote:


 Speaking as a researcher, I've on occasion had cause to call on stuff
 from such "2nd party" libraries, and it's not necessarily a happy
 experience.

 A while ago I had cause to call on a package in CRAN, the contributed
 package archive for the R statistical programming language.  It wasn't
 one of the modules that's packaged for Debian et al., so I had to pull
 things in directly from R's own package management and build and install
 locally.

 In and of itself, R makes this very easy, but what I wasn't prepared for
 was that in installing the one package I was interested in, it would
 pull in and install dozens of other CRAN packages.  And then, when I
 looked inside the code, actually finding out what it was doing was
 nightmarish, because the module of interest was built on top of several
 other contributions by the same authors. Understanding the code involved
 a massive wild goose chase through all those other contributed modules
 to find what functions were being called and what they did.

 I don't know if this is typical of CRAN packages, because I tend to use
 R infrequently, but my strong impression was of code that had been
 built, thrown over the wall and then built on top of without any
 attention to design, integration or performance (when I re-implemented
 the algorithms using Octave/MATLAB they were much faster, and I doubt
 this was down to the superiority of the language or interpreter).

 And this code wasn't built by stupid people -- they were very good
 statisticians.  In their defence, I suspect the reason they built
 higgledy-piggledy as they did was because they knew their earlier
 modules _worked_ and reliability was the most important thing for them.

 That's the cost of 2nd-party libraries -- they are very hit-and-miss in
 terms of design, sustainability and hence, reliability.

 Now, that said, I think a "2nd-party" repository for D could be a great
 project, but what I _wouldn't_ like to see was that repository being
 considered an adequate replacement for carefully designed standard
 library functionality.  One of the things I love about D is precisely
 the breadth of Phobos' support, and it feels like the solution to
 Phobos' problems is a better design and review process rather than
 ringfencing a too-small set of core functionality.

 Of course, a 2nd-party repository could be part of the prototyping and
 experimentation behind new standard-library work, just as Boost is for C++.



I can tell you this about Ruby. Its package manager, RubyGems, is one 
its greatest assets.

Just because a library has many dependencies doesn't mean it's badly 
designed. It could just mean that it's flexible and modular.

When Ruby on Rails 3.0 (a web framework for Ruby) was released it had a 
lot more dependencies than Rails 2.0. One reason was it was more modular 
designed. They extracted package that could be used without the rest of 
Rails, like ActiveRecord (database access), ActiveSupport (a general 
utility package) and so on.

Do you rather prefer how most C and C++ library works. They all 
implement everything from scratch. String classes, containers, 
algorithms and so on. Or they bundle a third party library within their 
own library forcing me to use it even though I already have it installed 
out of the box.

-- 
/Jacob Carlborg
Feb 02 2013
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/2/13 11:50 AM, Jacob Carlborg wrote:

 I can tell you this about Ruby. Its package manager, RubyGems, is one
 its greatest assets.

 Just because a library has many dependencies doesn't mean it's badly
 designed. It could just mean that it's flexible and modular.

 When Ruby on Rails 3.0 (a web framework for Ruby) was released it had a
 lot more dependencies than Rails 2.0. One reason was it was more modular
 designed. They extracted package that could be used without the rest of
 Rails, like ActiveRecord (database access), ActiveSupport (a general
 utility package) and so on.

 Do you rather prefer how most C and C++ library works. They all
 implement everything from scratch. String classes, containers,
 algorithms and so on. Or they bundle a third party library within their
 own library forcing me to use it even though I already have it installed
 out of the box.



I agree with all of the above. While knowing very little about design of 
package managers, I really appreciate using such (either at Facebook or 
for other languages).

It would be really meaningful to the future of D if a strongly motivated 
expert in package management would want to propose what it takes to get 
us where we should be in that regard.

As a simple question - what would be an existing design that would be 
most appropriate for us to take inspiration from?


Andrei
Feb 02 2013
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Jan 31, 2013 at 07:53:58PM +0100, Zach the Mystic wrote:

 On Thursday, 31 January 2013 at 15:38:04 UTC, Don wrote:

I'm not sure that we can solve this without addressing the
high-level question: What is the scope of Phobos?

How big will it eventually get? Twice its current size? Ten times?
A hundred times?



 Purely by chance I read an article on the blog of Jarrett
 Billingsley about this topic:
 
 http://www.jfbillingsley.com/blog/?p=206



Interesting. I think Phobos can be easily divided into two parts, with
core stuff like std.algorithm, std.range, std.stdio, etc., in one, and
less common but still widely applicable stuff like numeric algorithms in
a "2nd party" library (or libraries).

Even if we don't actually split Phobos up in that way, this may help us
reorganize Phobos in a better way. The current flat std hierarchy is
slowly approaching the point where it's starting to tear at the seams.
We should consider starting a 3rd level of hierarchy sooner rather than
later.


T

-- 
Help a man when he is in trouble and he will remember you when he is in trouble
again.
Jan 31 2013
prev sibling next sibling parent "Zach the Mystic" <reachBUTMINUSTHISzach gOOGLYmail.com> writes:
On Thursday, 31 January 2013 at 19:04:05 UTC, H. S. Teoh wrote:

 On Thu, Jan 31, 2013 at 07:53:58PM +0100, Zach the Mystic wrote:

 On Thursday, 31 January 2013 at 15:38:04 UTC, Don wrote:

I'm not sure that we can solve this without addressing the
high-level question: What is the scope of Phobos?

How big will it eventually get? Twice its current size? Ten 
times?
A hundred times?



 Purely by chance I read an article on the blog of Jarrett
 Billingsley about this topic:
 
 http://www.jfbillingsley.com/blog/?p=206



 Interesting. I think Phobos can be easily divided into two 
 parts, with
 core stuff like std.algorithm, std.range, std.stdio, etc., in 
 one, and
 less common but still widely applicable stuff like numeric 
 algorithms in
 a "2nd party" library (or libraries).

 Even if we don't actually split Phobos up in that way, this may 
 help us
 reorganize Phobos in a better way. The current flat std 
 hierarchy is
 slowly approaching the point where it's starting to tear at the 
 seams.
 We should consider starting a 3rd level of hierarchy sooner 
 rather than
 later.


 T



I don't know if this is relevant, but I find Jarret's writings to 
be quite brilliant. He designed a Lua-like language called miniD, 
based partly on D, but renamed Croc after he left the D 
community. I feel like I've learned a lot about programming just 
by reading the spec for his language.
Feb 01 2013
prev sibling next sibling parent "Zach the Mystic" <reachBUTMINUSTHISzach gOOGLYmail.com> writes:
On Thursday, 31 January 2013 at 19:04:05 UTC, H. S. Teoh wrote:

 Interesting. I think Phobos can be easily divided into two 
 parts, with
 core stuff like std.algorithm, std.range, std.stdio, etc., in 
 one, and
 less common but still widely applicable stuff like numeric 
 algorithms in
 a "2nd party" library (or libraries).

 Even if we don't actually split Phobos up in that way, this may 
 help us
 reorganize Phobos in a better way. The current flat std 
 hierarchy is
 slowly approaching the point where it's starting to tear at the 
 seams.
 We should consider starting a 3rd level of hierarchy sooner 
 rather than
 later.


 T



Whoops. Forgot to actually give the site:
http://www.jfbillingsley.com/croc/
Feb 01 2013
prev sibling next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 01/31/2013 08:02 PM, H. S. Teoh wrote:

 Interesting. I think Phobos can be easily divided into two parts, with
 core stuff like std.algorithm, std.range, std.stdio, etc., in one, and
 less common but still widely applicable stuff like numeric algorithms in
 a "2nd party" library (or libraries).



Speaking as a researcher, I've on occasion had cause to call on stuff from such 
"2nd party" libraries, and it's not necessarily a happy experience.

A while ago I had cause to call on a package in CRAN, the contributed package 
archive for the R statistical programming language.  It wasn't one of the 
modules that's packaged for Debian et al., so I had to pull things in directly 
from R's own package management and build and install locally.

In and of itself, R makes this very easy, but what I wasn't prepared for was 
that in installing the one package I was interested in, it would pull in and 
install dozens of other CRAN packages.  And then, when I looked inside the
code, 
actually finding out what it was doing was nightmarish, because the module of 
interest was built on top of several other contributions by the same authors. 
Understanding the code involved a massive wild goose chase through all those 
other contributed modules to find what functions were being called and what
they 
did.

I don't know if this is typical of CRAN packages, because I tend to use R 
infrequently, but my strong impression was of code that had been built, thrown 
over the wall and then built on top of without any attention to design, 
integration or performance (when I re-implemented the algorithms using 
Octave/MATLAB they were much faster, and I doubt this was down to the 
superiority of the language or interpreter).

And this code wasn't built by stupid people -- they were very good 
statisticians.  In their defence, I suspect the reason they built 
higgledy-piggledy as they did was because they knew their earlier modules 
_worked_ and reliability was the most important thing for them.

That's the cost of 2nd-party libraries -- they are very hit-and-miss in terms
of 
design, sustainability and hence, reliability.

Now, that said, I think a "2nd-party" repository for D could be a great
project, 
but what I _wouldn't_ like to see was that repository being considered an 
adequate replacement for carefully designed standard library functionality. 
One 
of the things I love about D is precisely the breadth of Phobos' support, and
it 
feels like the solution to Phobos' problems is a better design and review 
process rather than ringfencing a too-small set of core functionality.

Of course, a 2nd-party repository could be part of the prototyping and 
experimentation behind new standard-library work, just as Boost is for C++.
Feb 02 2013
prev sibling next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 02/02/2013 02:42 PM, Joseph Rushton Wakeling wrote:

 I've on occasion had cause to call on ....

 A while ago I had cause to call on ....



Oh bugger.  Overly verbose _and_ repetitive. :-P
Feb 02 2013
prev sibling next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 02/02/2013 05:50 PM, Jacob Carlborg wrote:

 I can tell you this about Ruby. Its package manager, RubyGems, is one its
 greatest assets.



Don't get me wrong.  I'm not speaking out against a package library per se, I 
think that would be an excellent idea.  I'm just against the idea of narrowing 
Phobos' scope too strongly in favour of various things being provided by 
contributed libraries.


 Just because a library has many dependencies doesn't mean it's badly designed.
 It could just mean that it's flexible and modular.



I agree, but the example I was talking about really was just frustrating and
opaque.


 Do you rather prefer how most C and C++ library works. They all implement
 everything from scratch. String classes, containers, algorithms and so on. Or
 they bundle a third party library within their own library forcing me to use it
 even though I already have it installed out of the box.



No, I don't like the typical C/C++ library situation -- that's why I'd like to 
see as much functionality as possible in a well-designed and supported standard 
library implementation.
Feb 02 2013
prev sibling parent "Peter Sommerfeld" <noreply rubrica.at> writes:
Joseph Rushton Wakeling <joseph.wakeling webdrake.net> wrote:


 No, I don't like the typical C/C++ library situation -- that's
 why I'd like to see as much functionality as possible in a well-designed  
 and supported standard library implementation.



As d-newcomer I want to support this! A large standard library
is an  attractive  argument for the language because it makes
sure that its features will be supported as the language evolves.

Its organization is currently rather confusing and I suggest
to seriously think about a deeper nested structure like it has
been suggested for std.numeric: std.numeric.~, std.concurrent.~,
std.io.~ etc. The pain of transformation is once, to keep it,
it is recurring.

Peter
Feb 02 2013
prev sibling next sibling parent "TommiT" <tommitissari hotmail.com> writes:
On Monday, 28 January 2013 at 23:11:11 UTC, Walter Bright wrote:

 http://www.drdobbs.com/cpp/implementing-half-floats-in-d/240146674



In the heat of coding, I think, the end-user is quite likely to 
make the honest mistake of writing:

HalfFloat h = HalfFloat(1.3f);

...when he should be writing:

HalfFloat h = hf!1.3f;
Jan 29 2013
prev sibling next sibling parent "David Nadlinger" <see klickverbot.at> writes:
On Tuesday, 29 January 2013 at 19:51:03 UTC, Jacob Carlborg wrote:

 On 2013-01-29 20:06, Walter Bright wrote:


 I was waiting on incorporating your improvements until after
 std.halffloat was merged into Phobos. This is because the old 
 ones were
 easier to understand, and I wanted them in the repository 
 history.



 Oh, please no, not another module at the "std" level.



+1

David
Jan 29 2013
prev sibling next sibling parent Iain Buclaw <ibuclaw ubuntu.com> writes:
--e89a8f921a1ee9ed0204d474e4ef
Content-Type: text/plain; charset=ISO-8859-1

On 29 January 2013 19:06, Walter Bright <newshound2 digitalmars.com> wrote:


 On 1/29/2013 4:31 AM, Don wrote:


 Since it got lost in the old thread on this topic, I'll repost my
 versions of
 floatToshort and shortToFloat, which are extremely fast (no unpredictable
 branches, no lookup tables) and respect the current rounding mode:



 I was waiting on incorporating your improvements until after std.halffloat
 was merged into Phobos. This is because the old ones were easier to
 understand, and I wanted them in the repository history.



I'm sure this was mentioned before, but what does halffloat have over
std.numeric.CustomFloat?


Thanks
-- 
Iain Buclaw

*(p < e ? p++ : p) = (c & 0x0f) + '0';

--e89a8f921a1ee9ed0204d474e4ef
Content-Type: text/html; charset=ISO-8859-1

<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On 29 January
2013 19:06, Walter Bright <span dir="ltr">&lt;<a
href="mailto:newshound2 digitalmars.com"
target="_blank">newshound2 digitalmars.com</a>&gt;</span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex"><div class="im">On 1/29/2013 4:31 AM, Don wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
Since it got lost in the old thread on this topic, I&#39;ll repost my versions
of<br>
floatToshort and shortToFloat, which are extremely fast (no unpredictable<br>
branches, no lookup tables) and respect the current rounding mode:<br>
</blockquote>
<br></div>
I was waiting on incorporating your improvements until after std.halffloat was
merged into Phobos. This is because the old ones were easier to understand, and
I wanted them in the repository history.<br>
<br>
</blockquote></div><br><br></div><div class="gmail_extra">I&#39;m sure this was
mentioned before, but what does halffloat have over
std.numeric.CustomFloat?<br></div><div class="gmail_extra"><br><br
clear="all"></div><div class="gmail_extra">
Thanks<br></div><div class="gmail_extra">-- <br>Iain Buclaw<br><br>*(p &lt; e ?
p++ : p) = (c &amp; 0x0f) + &#39;0&#39;;
</div></div>

--e89a8f921a1ee9ed0204d474e4ef--
Jan 29 2013
prev sibling next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 01/29/2013 11:27 PM, Iain Buclaw wrote:

 I'm sure this was mentioned before, but what does halffloat have over
 std.numeric.CustomFloat?



"These formats are for storage only; all operations on them are performed by 
first implicitly extracting them to real first."

... whereas HalfFloat extracts to float, and so should be faster, which is 
probably one of the main goals here.

But since std.numeric was mentioned, why not add HalfFloat there rather than
its 
own separate module?
Jan 29 2013
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Jan 29, 2013 at 04:36:48PM -0800, Walter Bright wrote:

 On 1/29/2013 2:27 PM, Iain Buclaw wrote:

I'm sure this was mentioned before, but what does halffloat have over
std.numeric.CustomFloat?



 There was a long discussion about it, I don't remember if it was in
 the ng or on github.



It was on github. But IIRC no conclusion was ever reached.


T

-- 
Живёшь только однажды.
Jan 29 2013
prev sibling next sibling parent "Mehrdad" <wfunction hotmail.com> writes:
On Tuesday, 29 January 2013 at 22:49:02 UTC, Joseph Rushton 
Wakeling wrote:

 But since std.numeric was mentioned, why not add HalfFloat 
 there rather than its own separate module?



+1
Jan 29 2013
prev sibling next sibling parent "Don" <don nospam.com> writes:
On Tuesday, 29 January 2013 at 22:49:02 UTC, Joseph Rushton 
Wakeling wrote:

 On 01/29/2013 11:27 PM, Iain Buclaw wrote:

 I'm sure this was mentioned before, but what does halffloat 
 have over
 std.numeric.CustomFloat?



 "These formats are for storage only; all operations on them are 
 performed by first implicitly extracting them to real first."

 ... whereas HalfFloat extracts to float, and so should be 
 faster, which is probably one of the main goals here.

 But since std.numeric was mentioned, why not add HalfFloat 
 there rather than its own separate module?




The discussion we had on github agreed that std.halffloat isn't a 
good place.
But OTOH std.numeric needs a complete overhaul, it's a mess.
It would be a mistake to throw it in there.
Jan 30 2013
prev sibling next sibling parent "Simen Kjaeraas" <simen.kjaras gmail.com> writes:
On 2013-01-30, 09:26, Don wrote:


 The discussion we had on github agreed that std.halffloat isn't a good  
 place.
 But OTOH std.numeric needs a complete overhaul, it's a mess.
 It would be a mistake to throw it in there.



So put it somewhere else for the moment, and move it to std.numeric after
a while? Boy, does that ever sound stupid.

I say stuff it in std.numeric, and announce properly that someone should
probably have look at fixing std.numeric, and that HalfFloat in there may
be the only thing salvageable (not that I know if it is, but that's how
you make it sound).

-- 
Simen
Jan 30 2013
prev sibling next sibling parent "Era Scarecrow" <rtcvb32 yahoo.com> writes:
On Tuesday, 29 January 2013 at 22:17:30 UTC, Dejan Lekic wrote:

 Jacob Carlborg wrote:

 Oh, please no, not another module at the "std" level.



 +11111111
 I can't agree more to this, Jacob...
 Seriously people... "std.halffloat" ???



  I'll agree too. HalfFloats are more a special case with certain 
circles of programmers that need those. The majority of 
programmers (95%+) won't care about half floats, nor would it be 
needed; It would be akin to making std.bitop instad of core.bitop.

  You could HalfFloats to std.math; Right next to cos, sin and 
tan. std.mathspecial doesn't seem like it would be closely enough 
tied to make it an option.

  Alternatively make a std.customtypes for related misc types like 
this as they are added to phobos.
Jan 30 2013
prev sibling next sibling parent "Don" <don nospam.com> writes:
On Wednesday, 30 January 2013 at 12:51:18 UTC, Simen Kjaeraas 
wrote:

 On 2013-01-30, 09:26, Don wrote:


 The discussion we had on github agreed that std.halffloat 
 isn't a good place.
 But OTOH std.numeric needs a complete overhaul, it's a mess.
 It would be a mistake to throw it in there.



 So put it somewhere else for the moment, and move it to 
 std.numeric after
 a while? Boy, does that ever sound stupid.



Yes, of course that would be stupid. It will never be in 
std.numeric.


 I say stuff it in std.numeric, and announce properly that 
 someone should
 probably have look at fixing std.numeric, and that HalfFloat in 
 there may
 be the only thing salvageable (not that I know if it is, but 
 that's how
 you make it sound).



std.numeric is not superficially flawed, it's fundamentally 
flawed. What is it for? What is its theme? The problem is, 
std.numeric is one of the few good names which are left as a 
possible package name, after C insulted the mathematical 
community by creating a module called 'math'.
Jan 31 2013
prev sibling next sibling parent "Simen Kjaeraas" <simen.kjaras gmail.com> writes:
On 2013-18-31 11:01, Don <don nospam.com> wrote:


 std.numeric is not superficially flawed, it's fundamentally flawed. What  
 is it for? What is its theme? The problem is, std.numeric is one of the  
 few good names which are left as a possible package name, after C  
 insulted the mathematical community by creating a module called 'math'.



Ah. Now that makes sense.

Suppose we do scrap std.numeric as it is, and find new homes for the
contents that can be salvaged (I sorta like 1100110's suggestion of
std.type.halffloat, std.type.bigint, etc). What would you want in a
refurbished std.numeric?

-- 
Simen
Jan 31 2013
prev sibling next sibling parent Jens Mueller <jens.k.mueller gmx.de> writes:
Don wrote:

 On Wednesday, 30 January 2013 at 12:51:18 UTC, Simen Kjaeraas wrote:

On 2013-01-30, 09:26, Don wrote:


The discussion we had on github agreed that std.halffloat isn't
a good place.
But OTOH std.numeric needs a complete overhaul, it's a mess.
It would be a mistake to throw it in there.



So put it somewhere else for the moment, and move it to
std.numeric after
a while? Boy, does that ever sound stupid.



 Yes, of course that would be stupid. It will never be in
 std.numeric.
 

I say stuff it in std.numeric, and announce properly that someone
should
probably have look at fixing std.numeric, and that HalfFloat in
there may
be the only thing salvageable (not that I know if it is, but
that's how
you make it sound).



 std.numeric is not superficially flawed, it's fundamentally flawed.
 What is it for? What is its theme? The problem is, std.numeric is
 one of the few good names which are left as a possible package name,
 after C insulted the mathematical community by creating a module
 called 'math'.



Dimitry said this once. Why not start from scratch with package called
std.numerics?
// for numeric types
std.numerics.halffloat
std.numerics.complex
std.numerics.quaternion
...
// algorithms working with numeric types
std.numerics.algorithms // even std.math could make sense

or something similar.
Wouldn't this work?

Jens
Jan 31 2013
prev sibling next sibling parent Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 01/31/2013 11:18 AM, Don wrote:

 std.numeric is not superficially flawed, it's fundamentally flawed. What is it
 for? What is its theme? The problem is, std.numeric is one of the few good
names
 which are left as a possible package name



std.numerical ... ?
Jan 31 2013
prev sibling next sibling parent "Don" <don nospam.com> writes:
On Thursday, 31 January 2013 at 13:41:13 UTC, Andrei Alexandrescu 
wrote:

 On 1/31/13 5:18 AM, Don wrote:

 std.numeric is not superficially flawed, it's fundamentally 
 flawed. What
 is it for? What is its theme? The problem is, std.numeric is 
 one of the
 few good names which are left as a possible package name, 
 after C
 insulted the mathematical community by creating a module 
 called 'math'.



 Guilty as charged. I've put stuff in std.numeric as I was 
 working on my thesis. I recall you added some stuff there too. 
 As I'm sure you remember the state of D in 2007 was rather 
 different than that of today. Overall no need to get agitated 
 here, we're all on the same boat and aiming for the same shore.



Sorry if that came across as agitated, it wasn't intended to be.
As you noted, I have code in there as well.
It's just one of those old modules that needs to be cleaned up, 
though it reveals a deeper issue - see below.


 Let's see what we have there:

 entropy
 CustomFloat
 kullbackLeiblerDivergence
 Fft
 gapWeightedSimilarityIncremental
 gapWeightedSimilarity
 gapWeightedSimilarityNormalized
 FPTemporary
 findRoot
 euclideanDistance
 dotProduct
 cosineSimilarity
 gcd
 jensenShannonDivergence
 normalize
 secantMethod

 The general theme is obvious - numeric algorithms and data 
 structures. Many are obvious and with obvious utility to one 
 interested in numerics: entropy, various distance and 
 similarity measures. I think you wrote findRoot.



Yes.
The basic problem is that there are hundreds of potential numeric 
algorithms and data structures of equal importance to these ones. 
In fact, the total number of mathematical algorithms is probably 
a substantial fraction of the total algorithms in computer 
science!

Even a module which contained only FFT, could be quite large, 
once it included all the important related transforms.


 The gapWeightedSimilarity algorithms are string kernels. They 
 are somewhat niche but quite powerful to anyone interested in 
 string similarity (technically they are string edit distance on 
 steroids). They might belong in std.string but I figured they 
 have enough numeric algorithm flavor to put them in there.

 So let's itemize the grievances and see how we can sort this 
 out.



I'm not sure that we can solve this without addressing the 
high-level question: What is the scope of Phobos?

How big will it eventually get? Twice its current size? Ten 
times? A hundred times?

Both SmallPhobos and LargePhobos are reasonable, but we do have 
to pick one. Currently we have aspects of both approaches, but 
they aren't compatible.

The current approach of putting everything directly into a single 
level in std doesn't scale very far -- it will look very clumsy 
once it gets more than (say) three times larger. This argues for 
SmallPhobos.

But if it doesn't get to be at least ten times larger, some of 
this niche stuff shouldn't be in there, they are functions from 
LargePhobos. If we go with SmallPhobos then we need to move the 
niche stuff somewhere else.
Jan 31 2013
prev sibling next sibling parent Sean Kelly <sean invisibleduck.org> writes:
On Jan 31, 2013, at 7:38 AM, "Don" <don nospam.com> wrote:


 But if it doesn't get to be at least ten times larger, some of this niche s=



th SmallPhobos then we need to move the niche stuff somewhere else.

If D had a package distribution system, the solution would be obvious. I'd a=
rgue for SmallPhobos plus that. Everything in Phobos proper has to be mainta=
ined as a part of Phobos, after all.=20=
Jan 31 2013
prev sibling next sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Thursday, 31 January 2013 at 15:38:04 UTC, Don wrote:

 On Thursday, 31 January 2013 at 13:41:13 UTC, Andrei 
 Alexandrescu wrote:

 On 1/31/13 5:18 AM, Don wrote:

 std.numeric is not superficially flawed, it's fundamentally 
 flawed. What
 is it for? What is its theme? The problem is, std.numeric is 
 one of the
 few good names which are left as a possible package name, 
 after C
 insulted the mathematical community by creating a module 
 called 'math'.



 Guilty as charged. I've put stuff in std.numeric as I was 
 working on my thesis. I recall you added some stuff there too. 
 As I'm sure you remember the state of D in 2007 was rather 
 different than that of today. Overall no need to get agitated 
 here, we're all on the same boat and aiming for the same shore.



 Sorry if that came across as agitated, it wasn't intended to be.
 As you noted, I have code in there as well.
 It's just one of those old modules that needs to be cleaned up, 
 though it reveals a deeper issue - see below.


 Let's see what we have there:

 entropy
 CustomFloat
 kullbackLeiblerDivergence
 Fft
 gapWeightedSimilarityIncremental
 gapWeightedSimilarity
 gapWeightedSimilarityNormalized
 FPTemporary
 findRoot
 euclideanDistance
 dotProduct
 cosineSimilarity
 gcd
 jensenShannonDivergence
 normalize
 secantMethod

 The general theme is obvious - numeric algorithms and data 
 structures. Many are obvious and with obvious utility to one 
 interested in numerics: entropy, various distance and 
 similarity measures. I think you wrote findRoot.



 Yes.
 The basic problem is that there are hundreds of potential 
 numeric algorithms and data structures of equal importance to 
 these ones. In fact, the total number of mathematical 
 algorithms is probably a substantial fraction of the total 
 algorithms in computer science!

 Even a module which contained only FFT, could be quite large, 
 once it included all the important related transforms.


 The gapWeightedSimilarity algorithms are string kernels. They 
 are somewhat niche but quite powerful to anyone interested in 
 string similarity (technically they are string edit distance 
 on steroids). They might belong in std.string but I figured 
 they have enough numeric algorithm flavor to put them in there.

 So let's itemize the grievances and see how we can sort this 
 out.



 I'm not sure that we can solve this without addressing the 
 high-level question: What is the scope of Phobos?

 How big will it eventually get? Twice its current size? Ten 
 times? A hundred times?

 Both SmallPhobos and LargePhobos are reasonable, but we do have 
 to pick one. Currently we have aspects of both approaches, but 
 they aren't compatible.

 The current approach of putting everything directly into a 
 single level in std doesn't scale very far -- it will look very 
 clumsy once it gets more than (say) three times larger. This 
 argues for SmallPhobos.

 But if it doesn't get to be at least ten times larger, some of 
 this niche stuff shouldn't be in there, they are functions from 
 LargePhobos. If we go with SmallPhobos then we need to move the 
 niche stuff somewhere else.




I think having a large standard library inspires confidence in 
developers. Rightly or wrongly, code in a standard library has an 
appearance of permanence, as opposed to being someone's personal 
project that may or may not disappear/cease to be maintained 
tomorrow.
Jan 31 2013
prev sibling next sibling parent "Zach the Mystic" <reachBUTMINUSTHISzach gOOGLYmail.com> writes:
On Thursday, 31 January 2013 at 15:38:04 UTC, Don wrote:

 I'm not sure that we can solve this without addressing the 
 high-level question: What is the scope of Phobos?

 How big will it eventually get? Twice its current size? Ten 
 times? A hundred times?



Purely by chance I read an article on the blog of Jarrett 
Billingsley about this topic:

http://www.jfbillingsley.com/blog/?p=206
Jan 31 2013
prev sibling next sibling parent "Paul D. Anderson" <paul.d.removethis.anderson comcast.andthis.net> writes:
I agree that we need a distinct package for numerics, both 
numbers and algorithms.

I proposed a birds-of-a-feather discussion on this topic for 
DConf2013, but haven't heard back yet. If not officially, we can 
still discuss this there, perhaps?

Paul D. Anderson
Feb 01 2013
prev sibling next sibling parent "Paul D. Anderson" <paul.d.removethis.anderson comcast.andthis.net> writes:
On Thursday, 31 January 2013 at 17:05:50 UTC, Andrei Alexandrescu 
wrote:

 On 1/31/13 10:38 AM, Don wrote:

 The basic problem is that there are hundreds of potential 
 numeric
 algorithms and data structures of equal importance to these 
 ones. In
 fact, the total number of mathematical algorithms is probably a
 substantial fraction of the total algorithms in computer 
 science!

 Even a module which contained only FFT, could be quite large, 
 once it
 included all the important related transforms.



 A possible solution would be to make std.numeric a knot for 
 several others, or convert it to a package.


 I'm not sure that we can solve this without addressing the 
 high-level
 question: What is the scope of Phobos?

 How big will it eventually get? Twice its current size? Ten 
 times? A
 hundred times?



 I think the current trend is to go for large libraries.


 Both SmallPhobos and LargePhobos are reasonable, but we do 
 have to pick
 one. Currently we have aspects of both approaches, but they 
 aren't
 compatible.



 LargePhobos. Agreed that we need to package-ize some of the 
 current modules.


 Andrei



Another consideration is the process for a package to become 
"standard". Current practice is to implement a package and go 
through a review in this forum. That's a good thing, but I'd like 
to see a more interactive approach, where the community was 
involved sooner:

1. A package is proposed for future inclusion in the library. 
Presumably this is done by someone with an interest in the 
subject, ideally this is someone who wants to spearhead the 
effort and may already have a draft implementation.

2. If there's enough interest to move forward, community members 
could bicker about all the bikeshed stuff: structures, methods, 
implementations, API, etc. until there's enough agreement about 
what the result should look like.

3. Interested parties could collaborate on the implementation and 
release.

4. There would have to be time limits. Many (most?) D projects 
start with a bang but then fizzle out before they're complete. 
There would need to be (monthly?) progress reviews. If the 
discussion or the implementation has dwindled, a reconsideration 
of the project could occur. If unsuccessful, the project would be 
moved off the active list. If there is renewed interest or if 
somebody else wants to take a crack at it the previous work can 
be a resource, but the project should probably go back to step 1.

You all see the problem with this: the very short attention span 
of the D community. But we've got a good start between the DIP 
process and the review queue. The only thing really different 
would be the progress reviews, which would hopefully prevent 
projects falling of the end of the earth.

This is just one possibility and there are probably better ways 
to do this. But I think some sort of maintenance of active 
projects is needed.

Paul
Feb 01 2013
prev sibling next sibling parent "Paul D. Anderson" <paul.d.removethis.anderson comcast.andthis.net> writes:
On Friday, 1 February 2013 at 20:04:47 UTC, Jacob Carlborg wrote:

 Isn't this what the whole community is about and what we're 
 already doing. Just not very formal or official.



You're right, but what I was looking for was something that 
actually was more formal.

I think the only thing that's missing is someone to keep track of 
and check up on development efforts. So I'll revise my proposal: 
I'll volunteer to try to gather information on the projects that 
are in-work and see what I can do to find their status and gauge 
their progress. If doing this turns out to be too difficult to 
manage or of little value I'll announce that and shut up. But if 
it works, I thik it will add value to our efforts.

Paul
Feb 01 2013
prev sibling next sibling parent "SomeDude" <lovelydear mailmetrash.com> writes:
On Thursday, 31 January 2013 at 15:38:04 UTC, Don wrote:

 The current approach of putting everything directly into a 
 single level in std doesn't scale very far -- it will look very 
 clumsy once it gets more than (say) three times larger. This 
 argues for SmallPhobos.

 But if it doesn't get to be at least ten times larger, some of 
 this niche stuff shouldn't be in there, they are functions from 
 LargePhobos. If we go with SmallPhobos then we need to move the 
 niche stuff somewhere else.



I am all for a "small" Phobos. And specialized libraries in 
Deimos, which should get the same level of attention as Phobos.
Feb 02 2013
prev sibling parent "David Nadlinger" <see klickverbot.at> writes:
On Sunday, 3 February 2013 at 07:43:57 UTC, SomeDude wrote:

 I am all for a "small" Phobos. And specialized libraries in 
 Deimos, which should get the same level of attention as Phobos.



Huh? Deimos is specifically for C headers translated to D.

David
Feb 03 2013