www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - DIP80: phobos additions

reply "Robert burner Schadek" <rburners gmail.com> writes:
Phobos is awesome, the libs of go, python and rust only have 
better marketing.
As discussed on dconf, phobos needs to become big and blow the 
rest out of the sky.

http://wiki.dlang.org/DIP80

lets get OT, please discuss
Jun 07 2015
next sibling parent "Dennis Ritchie" <dennis.ritchie mail.ru> writes:
On Sunday, 7 June 2015 at 18:27:16 UTC, Robert burner Schadek 
wrote:
 Phobos is awesome, the libs of go, python and rust only have 
 better marketing.
 As discussed on dconf, phobos needs to become big and blow the 
 rest out of the sky.

 http://wiki.dlang.org/DIP80

 lets get OT, please discuss
Yes, it's a great DIP's discussion. I just for the expansion of Phobos! You also need to consider Hana to copy some very useful elements in the Phobos: http://ldionne.com/hana/
Jun 07 2015
prev sibling next sibling parent reply "weaselcat" <weaselcat gmail.com> writes:
On Sunday, 7 June 2015 at 18:27:16 UTC, Robert burner Schadek 
wrote:
 Phobos is awesome, the libs of go, python and rust only have 
 better marketing.
 As discussed on dconf, phobos needs to become big and blow the 
 rest out of the sky.

 http://wiki.dlang.org/DIP80

 lets get OT, please discuss
can we discuss the downside of making phobos huge? I actively avoid adding phobos libs to my projects because it bloats my binaries and increases compile times by massive amounts.
Jun 07 2015
next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Monday, 8 June 2015 at 00:05:58 UTC, weaselcat wrote:
 I actively avoid adding phobos libs to my projects because it 
 bloats my binaries and increases compile times by massive 
 amounts.
Me too... but that's not actually a problem of huge library. It is more a problem of an interconnected library - if you write independent modules an import should only pull them with little from other ones. There's a difference with classes because of Object.factory, all of them are pulled in, but modules with functions, structs, and templates are cool, shouldn't be a problem.
Jun 07 2015
prev sibling parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Monday, 8 June 2015 at 00:05:58 UTC, weaselcat wrote:
 can we discuss the downside of making phobos huge?

 I actively avoid adding phobos libs to my projects because it 
 bloats my binaries and increases compile times by massive 
 amounts.
Andrei has already stated that we are definitely going to make Phobos large. We are _not_ going for the minimalistic approach, and pretty much no other language is at this point either. So, Phobos _will_ continue to grow in size. Now, as Adam points out, we can should do a better job of making it so that different pieces of Phobos don't depend on each other if they don't need to, but it's a given at this point that Phobos is only going to get larger. And if unnecessary dependencies are kept to a minimum, then it really shouldn't hurt your compilation times (and I'm sure that we'll have further compiler improvements in that area anyway). - Jonathan M Davis
Jun 07 2015
parent "weaselcat" <weaselcat gmail.com> writes:
On Monday, 8 June 2015 at 01:39:33 UTC, Jonathan M Davis wrote:
 On Monday, 8 June 2015 at 00:05:58 UTC, weaselcat wrote:
 can we discuss the downside of making phobos huge?

 I actively avoid adding phobos libs to my projects because it 
 bloats my binaries and increases compile times by massive 
 amounts.
Andrei has already stated that we are definitely going to make Phobos large. We are _not_ going for the minimalistic approach, and pretty much no other language is at this point either. So, Phobos _will_ continue to grow in size. Now, as Adam points out, we can should do a better job of making it so that different pieces of Phobos don't depend on each other if they don't need to, but it's a given at this point that Phobos is only going to get larger. And if unnecessary dependencies are kept to a minimum, then it really shouldn't hurt your compilation times (and I'm sure that we'll have further compiler improvements in that area anyway). - Jonathan M Davis
I wasn't arguing against a large library(in fact, I prefer it.) I just think the effort should be put towards making phobos more modular before adding more stuff on top of it and making the problem worse. bye,
Jun 07 2015
prev sibling next sibling parent reply "Tofu Ninja" <emmons0 purdue.edu> writes:
On Sunday, 7 June 2015 at 18:27:16 UTC, Robert burner Schadek 
wrote:
 Phobos is awesome, the libs of go, python and rust only have 
 better marketing.
 As discussed on dconf, phobos needs to become big and blow the 
 rest out of the sky.

 http://wiki.dlang.org/DIP80

 lets get OT, please discuss
Would love some kind of color implementation in Phobos, simple linear algebra(vectors, matrices), image manipulation.
Jun 07 2015
parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 8/06/2015 2:50 p.m., Tofu Ninja wrote:
 On Sunday, 7 June 2015 at 18:27:16 UTC, Robert burner Schadek wrote:
 Phobos is awesome, the libs of go, python and rust only have better
 marketing.
 As discussed on dconf, phobos needs to become big and blow the rest
 out of the sky.

 http://wiki.dlang.org/DIP80

 lets get OT, please discuss
Would love some kind of color implementation in Phobos, simple linear algebra(vectors, matrices), image manipulation.
https://github.com/D-Programming-Language/phobos/pull/2845 Heyyyy Manu, hows it going? Gl3n should be a candidate as it is old code and good one at that. https://github.com/Dav1dde/gl3n But it seems like it is no longer maintained. Can anyone contact the author regarding license to boost? Image manipulation blocked by color.
Jun 07 2015
next sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 8 June 2015 at 13:08, Rikki Cattermole via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 8/06/2015 2:50 p.m., Tofu Ninja wrote:
 On Sunday, 7 June 2015 at 18:27:16 UTC, Robert burner Schadek wrote:
 Phobos is awesome, the libs of go, python and rust only have better
 marketing.
 As discussed on dconf, phobos needs to become big and blow the rest
 out of the sky.

 http://wiki.dlang.org/DIP80

 lets get OT, please discuss
Would love some kind of color implementation in Phobos, simple linear algebra(vectors, matrices), image manipulation.
https://github.com/D-Programming-Language/phobos/pull/2845 Heyyyy Manu, hows it going?
I've kinda just been working on it on the side for my own use. I wasn't happy with the layout, and restructured it a lot. If there's an active demand for it, I'll give it top priority...?
Jun 07 2015
next sibling parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 8/06/2015 3:48 p.m., Manu via Digitalmars-d wrote:
 On 8 June 2015 at 13:08, Rikki Cattermole via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 8/06/2015 2:50 p.m., Tofu Ninja wrote:
 On Sunday, 7 June 2015 at 18:27:16 UTC, Robert burner Schadek wrote:
 Phobos is awesome, the libs of go, python and rust only have better
 marketing.
 As discussed on dconf, phobos needs to become big and blow the rest
 out of the sky.

 http://wiki.dlang.org/DIP80

 lets get OT, please discuss
Would love some kind of color implementation in Phobos, simple linear algebra(vectors, matrices), image manipulation.
https://github.com/D-Programming-Language/phobos/pull/2845 Heyyyy Manu, hows it going?
I've kinda just been working on it on the side for my own use. I wasn't happy with the layout, and restructured it a lot. If there's an active demand for it, I'll give it top priority...?
Like I said its a blocker for an image library. There's no point implementing an image library with a half baked color definition meant for phobos. The long term issue is that we cannot really move forward with anything related to GUI or game development into phobos without it. So preferably we can get it into phobos by the end of the year :)
Jun 07 2015
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 8 June 2015 at 13:54, Rikki Cattermole via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 8/06/2015 3:48 p.m., Manu via Digitalmars-d wrote:
 On 8 June 2015 at 13:08, Rikki Cattermole via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 8/06/2015 2:50 p.m., Tofu Ninja wrote:
 On Sunday, 7 June 2015 at 18:27:16 UTC, Robert burner Schadek wrote:
 Phobos is awesome, the libs of go, python and rust only have better
 marketing.
 As discussed on dconf, phobos needs to become big and blow the rest
 out of the sky.

 http://wiki.dlang.org/DIP80

 lets get OT, please discuss
Would love some kind of color implementation in Phobos, simple linear algebra(vectors, matrices), image manipulation.
https://github.com/D-Programming-Language/phobos/pull/2845 Heyyyy Manu, hows it going?
I've kinda just been working on it on the side for my own use. I wasn't happy with the layout, and restructured it a lot. If there's an active demand for it, I'll give it top priority...?
Like I said its a blocker for an image library. There's no point implementing an image library with a half baked color definition meant for phobos.
Yeah, that's fine. Is there an initiative for a phobos image library? I have said before that I'm dubious about it's worth; the trouble with an image library is that it will be almost impossible to decide on API, whereas a colour is fairly unambiguous in terms of design merits.
 The long term issue is that we cannot really move forward with anything
 related to GUI or game development into phobos without it.
 So preferably we can get it into phobos by the end of the year :)
Yeah, I agree it's a sore missing point, which is why I started working on it ;) ... I'll make it high priority. I recently finished up various work on premake5, so I can work on this now.
Jun 07 2015
next sibling parent Rikki Cattermole <alphaglosined gmail.com> writes:
On 8/06/2015 4:05 p.m., Manu via Digitalmars-d wrote:
 On 8 June 2015 at 13:54, Rikki Cattermole via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 8/06/2015 3:48 p.m., Manu via Digitalmars-d wrote:
 On 8 June 2015 at 13:08, Rikki Cattermole via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 8/06/2015 2:50 p.m., Tofu Ninja wrote:
 On Sunday, 7 June 2015 at 18:27:16 UTC, Robert burner Schadek wrote:
 Phobos is awesome, the libs of go, python and rust only have better
 marketing.
 As discussed on dconf, phobos needs to become big and blow the rest
 out of the sky.

 http://wiki.dlang.org/DIP80

 lets get OT, please discuss
Would love some kind of color implementation in Phobos, simple linear algebra(vectors, matrices), image manipulation.
https://github.com/D-Programming-Language/phobos/pull/2845 Heyyyy Manu, hows it going?
I've kinda just been working on it on the side for my own use. I wasn't happy with the layout, and restructured it a lot. If there's an active demand for it, I'll give it top priority...?
Like I said its a blocker for an image library. There's no point implementing an image library with a half baked color definition meant for phobos.
Yeah, that's fine. Is there an initiative for a phobos image library? I have said before that I'm dubious about it's worth; the trouble with an image library is that it will be almost impossible to decide on API, whereas a colour is fairly unambiguous in terms of design merits.
I agree that it is. But we will need to move past this for the betterment of our ecosystem. Without it we will suffer too much. As it is, Devisualization.Image will have a new interface once std.image.color is pulled. So it'll be a contender for std.image.
 The long term issue is that we cannot really move forward with anything
 related to GUI or game development into phobos without it.
 So preferably we can get it into phobos by the end of the year :)
Yeah, I agree it's a sore missing point, which is why I started working on it ;) ... I'll make it high priority. I recently finished up various work on premake5, so I can work on this now.
Sounds good, I was getting worried that you had stopped altogether.
Jun 07 2015
prev sibling parent reply "Tofu Ninja" <emmons0 purdue.edu> writes:
On Monday, 8 June 2015 at 04:05:23 UTC, Manu wrote:
 Yeah, that's fine. Is there an initiative for a phobos image 
 library?
 I have said before that I'm dubious about it's worth; the 
 trouble with
 an image library is that it will be almost impossible to decide 
 on
 API, whereas a colour is fairly unambiguous in terms of design 
 merits.
Personally I would just be happy with a d wrapper for something like freeimage being included.
Jun 07 2015
parent reply "Mike" <none none.com> writes:
On Monday, 8 June 2015 at 04:21:45 UTC, Tofu Ninja wrote:
 Personally I would just be happy with a d wrapper for something 
 like freeimage being included.
That's what Deimos is for (https://github.com/D-Programming-Deimos/FreeImage). Mike
Jun 07 2015
next sibling parent reply "Tofu Ninja" <emmons0 purdue.edu> writes:
On Monday, 8 June 2015 at 04:22:56 UTC, Mike wrote:
 On Monday, 8 June 2015 at 04:21:45 UTC, Tofu Ninja wrote:
 Personally I would just be happy with a d wrapper for 
 something like freeimage being included.
That's what Deimos is for (https://github.com/D-Programming-Deimos/FreeImage). Mike
I guess I meant to use it as a base for image loading and storing and to build some kind of d image lib on top of it. I see no point in us trying to implement all the various image formats if we try to make a image lib for phobos.
Jun 07 2015
parent Rikki Cattermole <alphaglosined gmail.com> writes:
On 8/06/2015 4:34 p.m., Tofu Ninja wrote:
 On Monday, 8 June 2015 at 04:22:56 UTC, Mike wrote:
 On Monday, 8 June 2015 at 04:21:45 UTC, Tofu Ninja wrote:
 Personally I would just be happy with a d wrapper for something like
 freeimage being included.
That's what Deimos is for (https://github.com/D-Programming-Deimos/FreeImage). Mike
I guess I meant to use it as a base for image loading and storing and to build some kind of d image lib on top of it. I see no point in us trying to implement all the various image formats if we try to make a image lib for phobos.
Atleast my idea behind Devisualization.Image was mostly this. The implementation can be swapped out with another easily. But the actual interface used is well made. So while a Phobos image library might have a few formats such as PNG, it probably wouldn't include a vast array of them. So then its just a matter of allowing 3rd party libraries to add them transparently.
Jun 07 2015
prev sibling parent reply "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Monday, 8 June 2015 at 04:22:56 UTC, Mike wrote:
 On Monday, 8 June 2015 at 04:21:45 UTC, Tofu Ninja wrote:
 Personally I would just be happy with a d wrapper for 
 something like freeimage being included.
That's what Deimos is for (https://github.com/D-Programming-Deimos/FreeImage).
Yeah. After the problems with linking in curl, I think that we more or less decided that including stuff in Phobos which has to link against 3rd party libraries isn't a great idea. Maybe we'll end up doing it again, but in general, it just makes more sense for those to be done as 3rd party libraries and put in code.dlang.org. - Jonathan M Davis
Jun 07 2015
parent "Tofu Ninja" <emmons0 purdue.edu> writes:
On Monday, 8 June 2015 at 04:34:56 UTC, Jonathan M Davis wrote:
 On Monday, 8 June 2015 at 04:22:56 UTC, Mike wrote:
 On Monday, 8 June 2015 at 04:21:45 UTC, Tofu Ninja wrote:
 Personally I would just be happy with a d wrapper for 
 something like freeimage being included.
That's what Deimos is for (https://github.com/D-Programming-Deimos/FreeImage).
Yeah. After the problems with linking in curl, I think that we more or less decided that including stuff in Phobos which has to link against 3rd party libraries isn't a great idea. Maybe we'll end up doing it again, but in general, it just makes more sense for those to be done as 3rd party libraries and put in code.dlang.org. - Jonathan M Davis
I think that is probably pretty sad if that is actually the current stance. There are so many great libraries that Phobos could benefit from and is not because of a packaging issue.
Jun 08 2015
prev sibling parent "Mike" <none none.com> writes:
On Monday, 8 June 2015 at 03:48:14 UTC, Manu wrote:

 I've kinda just been working on it on the side for my own use.
 I wasn't happy with the layout, and restructured it a lot.
 If there's an active demand for it, I'll give it top 
 priority...?
I'm interested in this library as well. Mike
Jun 07 2015
prev sibling parent "Mike" <none none.com> writes:
On Monday, 8 June 2015 at 03:08:46 UTC, Rikki Cattermole wrote:

 Gl3n should be a candidate as it is old code and good one at 
 that.
 https://github.com/Dav1dde/gl3n
 But it seems like it is no longer maintained.
Looks like it's been getting a couple commits monthly, so I think it's being maintained.
Jun 08 2015
prev sibling next sibling parent reply "weaselcat" <weaselcat gmail.com> writes:
On Sunday, 7 June 2015 at 18:27:16 UTC, Robert burner Schadek 
wrote:
 Phobos is awesome, the libs of go, python and rust only have 
 better marketing.
 As discussed on dconf, phobos needs to become big and blow the 
 rest out of the sky.

 http://wiki.dlang.org/DIP80

 lets get OT, please discuss
I think a std.bindings or something similar for ubiquitous C libraries would go a long way - _quality_(not just a wrapper) SDL, OpenGL, etc bindings. D is very attractive to game developers, I think with a little push it would get a lot of traction from this.
Jun 07 2015
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 8 June 2015 at 13:15, weaselcat via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Sunday, 7 June 2015 at 18:27:16 UTC, Robert burner Schadek wrote:
 Phobos is awesome, the libs of go, python and rust only have better
 marketing.
 As discussed on dconf, phobos needs to become big and blow the rest out of
 the sky.

 http://wiki.dlang.org/DIP80

 lets get OT, please discuss
I think a std.bindings or something similar for ubiquitous C libraries would go a long way - _quality_(not just a wrapper) SDL, OpenGL, etc bindings. D is very attractive to game developers, I think with a little push it would get a lot of traction from this.
I've been humoring the idea of porting my engine to D. It's about 15 years of development, better/cleaner than most proprietary engines I've used at game studios. I wonder if there would be interest in this? Problem is, I need all the cross compilers to exist before I pull the plug on the C code... a game engine is no good if it's not portable to all the consoles under the sun. That said, I think it would be a good case-study to get the cross compilers working against.
Jun 07 2015
next sibling parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 8/06/2015 3:53 p.m., Manu via Digitalmars-d wrote:
 On 8 June 2015 at 13:15, weaselcat via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On Sunday, 7 June 2015 at 18:27:16 UTC, Robert burner Schadek wrote:
 Phobos is awesome, the libs of go, python and rust only have better
 marketing.
 As discussed on dconf, phobos needs to become big and blow the rest out of
 the sky.

 http://wiki.dlang.org/DIP80

 lets get OT, please discuss
I think a std.bindings or something similar for ubiquitous C libraries would go a long way - _quality_(not just a wrapper) SDL, OpenGL, etc bindings. D is very attractive to game developers, I think with a little push it would get a lot of traction from this.
I've been humoring the idea of porting my engine to D. It's about 15 years of development, better/cleaner than most proprietary engines I've used at game studios. I wonder if there would be interest in this? Problem is, I need all the cross compilers to exist before I pull the plug on the C code... a game engine is no good if it's not portable to all the consoles under the sun. That said, I think it would be a good case-study to get the cross compilers working against.
I'm definitely interested. Imagine getting something like that into phobos! Would be utterly amazing for us. Or atleast parts of it, once D-ified. Although might be worth doing tests using e.g. ldc to see how many platforms you can actually get working. Then perhaps an acceptance criteria before you port it?
Jun 07 2015
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 8 June 2015 at 13:59, Rikki Cattermole via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 8/06/2015 3:53 p.m., Manu via Digitalmars-d wrote:
 On 8 June 2015 at 13:15, weaselcat via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On Sunday, 7 June 2015 at 18:27:16 UTC, Robert burner Schadek wrote:
 Phobos is awesome, the libs of go, python and rust only have better
 marketing.
 As discussed on dconf, phobos needs to become big and blow the rest out
 of
 the sky.

 http://wiki.dlang.org/DIP80

 lets get OT, please discuss
I think a std.bindings or something similar for ubiquitous C libraries would go a long way - _quality_(not just a wrapper) SDL, OpenGL, etc bindings. D is very attractive to game developers, I think with a little push it would get a lot of traction from this.
I've been humoring the idea of porting my engine to D. It's about 15 years of development, better/cleaner than most proprietary engines I've used at game studios. I wonder if there would be interest in this? Problem is, I need all the cross compilers to exist before I pull the plug on the C code... a game engine is no good if it's not portable to all the consoles under the sun. That said, I think it would be a good case-study to get the cross compilers working against.
I'm definitely interested. Imagine getting something like that into phobos! Would be utterly amazing for us. Or atleast parts of it, once D-ified.
I can't really see a place for many parts in phobos... large parts of it are hardware/platform abstraction; would depend on many system library bindings present in phobos.
 Although might be worth doing tests using e.g. ldc to see how many platforms
 you can actually get working.
 Then perhaps an acceptance criteria before you port it?
Yeah, it's a lot of work to do unit tests for parallel runtime systems that depend almost exclusively on user input or large bodies of external data... and where many of the outputs don't naturally feedback for analysis (render output, audio output). I can see a unit test framework being more code than most parts of the engine ;) .. not that it would be bad (it would be awesome!), I just can't imagine a simple/acceptable design. The thing I'm most happy about with Fuji is how relatively minimal it is (considering its scope and capability).
Jun 07 2015
parent Rikki Cattermole <alphaglosined gmail.com> writes:
On 8/06/2015 4:12 p.m., Manu via Digitalmars-d wrote:
 On 8 June 2015 at 13:59, Rikki Cattermole via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 8/06/2015 3:53 p.m., Manu via Digitalmars-d wrote:
 On 8 June 2015 at 13:15, weaselcat via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On Sunday, 7 June 2015 at 18:27:16 UTC, Robert burner Schadek wrote:
 Phobos is awesome, the libs of go, python and rust only have better
 marketing.
 As discussed on dconf, phobos needs to become big and blow the rest out
 of
 the sky.

 http://wiki.dlang.org/DIP80

 lets get OT, please discuss
I think a std.bindings or something similar for ubiquitous C libraries would go a long way - _quality_(not just a wrapper) SDL, OpenGL, etc bindings. D is very attractive to game developers, I think with a little push it would get a lot of traction from this.
I've been humoring the idea of porting my engine to D. It's about 15 years of development, better/cleaner than most proprietary engines I've used at game studios. I wonder if there would be interest in this? Problem is, I need all the cross compilers to exist before I pull the plug on the C code... a game engine is no good if it's not portable to all the consoles under the sun. That said, I think it would be a good case-study to get the cross compilers working against.
I'm definitely interested. Imagine getting something like that into phobos! Would be utterly amazing for us. Or atleast parts of it, once D-ified.
I can't really see a place for many parts in phobos... large parts of it are hardware/platform abstraction; would depend on many system library bindings present in phobos.
 Although might be worth doing tests using e.g. ldc to see how many platforms
 you can actually get working.
 Then perhaps an acceptance criteria before you port it?
Yeah, it's a lot of work to do unit tests for parallel runtime systems that depend almost exclusively on user input or large bodies of external data... and where many of the outputs don't naturally feedback for analysis (render output, audio output). I can see a unit test framework being more code than most parts of the engine ;) .. not that it would be bad (it would be awesome!), I just can't imagine a simple/acceptable design. The thing I'm most happy about with Fuji is how relatively minimal it is (considering its scope and capability).
They would have to be manual tests. So e.g. throws exceptions happily and uses threads kind of thing. But where you load it up and run it. It could help the ldc and gdc guys know what is still missing for this use case.
Jun 07 2015
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 6/7/2015 8:53 PM, Manu via Digitalmars-d wrote:
 I've been humoring the idea of porting my engine to D. It's about 15
 years of development, better/cleaner than most proprietary engines
 I've used at game studios.
 I wonder if there would be interest in this? Problem is, I need all
 the cross compilers to exist before I pull the plug on the C code... a
 game engine is no good if it's not portable to all the consoles under
 the sun. That said, I think it would be a good case-study to get the
 cross compilers working against.
It's a chicken-and-egg thing. Somebody's got to start and not wait for the others.
Jun 08 2015
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2015-06-08 05:53, Manu via Digitalmars-d wrote:

 I've been humoring the idea of porting my engine to D. It's about 15
 years of development, better/cleaner than most proprietary engines
 I've used at game studios.
Perhaps you could try using magicport. -- /Jacob Carlborg
Jun 08 2015
prev sibling parent reply "Joakim" <dlang joakim.fea.st> writes:
On Monday, 8 June 2015 at 03:53:52 UTC, Manu wrote:
 I've been humoring the idea of porting my engine to D. It's 
 about 15
 years of development, better/cleaner than most proprietary 
 engines
 I've used at game studios.
 I wonder if there would be interest in this? Problem is, I need 
 all
 the cross compilers to exist before I pull the plug on the C 
 code... a
 game engine is no good if it's not portable to all the consoles 
 under
 the sun. That said, I think it would be a good case-study to 
 get the
 cross compilers working against.
What cross-compilers are you waiting for? Nobody is working on XBone or PS4 as far as I know, but Dan's work on iOS seems pretty far along, if you want to try that out.
Jun 09 2015
parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 9 June 2015 at 17:32, Joakim via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 8 June 2015 at 03:53:52 UTC, Manu wrote:
 I've been humoring the idea of porting my engine to D. It's about 15
 years of development, better/cleaner than most proprietary engines
 I've used at game studios.
 I wonder if there would be interest in this? Problem is, I need all
 the cross compilers to exist before I pull the plug on the C code... a
 game engine is no good if it's not portable to all the consoles under
 the sun. That said, I think it would be a good case-study to get the
 cross compilers working against.
What cross-compilers are you waiting for? Nobody is working on XBone or PS4 as far as I know, but Dan's work on iOS seems pretty far along, if you want to try that out.
XBone works. PS4 is probably easy or already working. Android, iOS are critical. Nintendo platforms also exist. I would hope we'll see Enscripten and NaCl at some point; I could use them at work right now. The phones do appear to be moving recently, which is really encouraging.
Jun 09 2015
prev sibling next sibling parent reply "ezneh" <petitv.isat gmail.com> writes:
On Sunday, 7 June 2015 at 18:27:16 UTC, Robert burner Schadek 
wrote:
 Phobos is awesome, the libs of go, python and rust only have 
 better marketing.
 As discussed on dconf, phobos needs to become big and blow the 
 rest out of the sky.

 http://wiki.dlang.org/DIP80

 lets get OT, please discuss
IMHO, Phobos could include thinks like this as a standard : - OAuth (1 & 2), at least it would be useful for projects like vibe.d - Create / read QR codes, maybe ? It seems we see more and more QR Codes here and there, so it could potentially be worth it - Better / full OS bindings (winapi, x11, ect.), but it would (sadly) require a very large amount of work to do so. - + what has been said already. I guess we can try to think about integrating a large amount of things that are widely used and are considered as "standards"
Jun 08 2015
parent reply Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On 06/08/2015 03:55 AM, ezneh wrote:
 - Create / read QR codes, maybe ? It seems we see more and more QR Codes
 here and there, so it could potentially be worth it
I see them everywhere, but does anyone ever actually use them? Usually it's just an obvious link to some company's marketing/advertising. It's basically just like the old CueCat, if anyone remembers it: <https://en.wikipedia.org/wiki/CueCat> Only time I've ever seen *anyone* actually using a QR code is when *I* use a "display QR link for this page" FF plugin to send the webpage I'm looking at to my phone. Maybe I'm just not seeing it, but I suspect QR is more someone that companies *want* people to care about, rather than something anyone actually uses.
Jun 13 2015
next sibling parent ketmar <ketmar ketmar.no-ip.org> writes:
On Sat, 13 Jun 2015 11:46:41 -0400, Nick Sabalausky wrote:

 Maybe I'm just not seeing it, but I suspect QR is more someone that
 companies *want* people to care about, rather than something anyone
 actually uses.
same for me.=
Jun 13 2015
prev sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/13/15 11:46 AM, Nick Sabalausky wrote:
 On 06/08/2015 03:55 AM, ezneh wrote:
 - Create / read QR codes, maybe ? It seems we see more and more QR Codes
 here and there, so it could potentially be worth it
I see them everywhere, but does anyone ever actually use them? Usually it's just an obvious link to some company's marketing/advertising. It's basically just like the old CueCat, if anyone remembers it: <https://en.wikipedia.org/wiki/CueCat> Only time I've ever seen *anyone* actually using a QR code is when *I* use a "display QR link for this page" FF plugin to send the webpage I'm looking at to my phone. Maybe I'm just not seeing it, but I suspect QR is more someone that companies *want* people to care about, rather than something anyone actually uses.
A rather cool usage of QR code I saw was a sticker on a device that was a link to the PDF of the manual. -Steve
Jun 13 2015
next sibling parent ketmar <ketmar ketmar.no-ip.org> writes:
On Sat, 13 Jun 2015 21:57:42 -0400, Steven Schveighoffer wrote:

 A rather cool usage of QR code I saw was a sticker on a device that was
 a link to the PDF of the manual.
it's k001, but i'll take a printed URL for it in any time. the old good=20 URL that i can read with my eyes.=
Jun 13 2015
prev sibling parent reply "Joakim" <dlang joakim.fea.st> writes:
On Sunday, 14 June 2015 at 01:57:37 UTC, Steven Schveighoffer 
wrote:
 On 6/13/15 11:46 AM, Nick Sabalausky wrote:
 On 06/08/2015 03:55 AM, ezneh wrote:
 - Create / read QR codes, maybe ? It seems we see more and 
 more QR Codes
 here and there, so it could potentially be worth it
I see them everywhere, but does anyone ever actually use them? Usually it's just an obvious link to some company's marketing/advertising. It's basically just like the old CueCat, if anyone remembers it: <https://en.wikipedia.org/wiki/CueCat> Only time I've ever seen *anyone* actually using a QR code is when *I* use a "display QR link for this page" FF plugin to send the webpage I'm looking at to my phone. Maybe I'm just not seeing it, but I suspect QR is more someone that companies *want* people to care about, rather than something anyone actually uses.
A rather cool usage of QR code I saw was a sticker on a device that was a link to the PDF of the manual.
Then there's always this: http://www.theverge.com/2015/6/19/8811425/heinz-ketchup-qr-code-porn-site-fundorado Not the fault of the QR code of course, just an expired domain name, but still funny. :)
Jun 19 2015
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/19/15 9:50 PM, Joakim wrote:

 Then there's always this:

 http://www.theverge.com/2015/6/19/8811425/heinz-ketchup-qr-code-porn-site-fundorado


 Not the fault of the QR code of course, just an expired domain name, but
 still funny. :)
Oh man. Note to marketing department -- all QR codes must point to ourcompany.com, you can redirect from there!!! -Steve
Jun 21 2015
prev sibling next sibling parent "ponce" <contact gam3sfrommars.fr> writes:
On Sunday, 7 June 2015 at 18:27:16 UTC, Robert burner Schadek 
wrote:
 Phobos is awesome, the libs of go, python and rust only have 
 better marketing.
 As discussed on dconf, phobos needs to become big and blow the 
 rest out of the sky.

 http://wiki.dlang.org/DIP80

 lets get OT, please discuss
What I'd like in phobos: - OS bindings (more complete win32, Cocoa etc) - DerelictUtil - allocators That's about it.
Jun 08 2015
prev sibling next sibling parent "Per =?UTF-8?B?Tm9yZGzDtnci?= <per.nordlow gmail.com> writes:
On Sunday, 7 June 2015 at 18:27:16 UTC, Robert burner Schadek 
wrote:
 http://wiki.dlang.org/DIP80

 lets get OT, please discuss
Automatic randomizer for builtins, ranges, etc. Used to generate data for tests. Here's a start: https://github.com/nordlow/justd/blob/master/random_ex.d
Jun 08 2015
prev sibling next sibling parent reply "Ilya Yaroshenko" <ilyayaroshenko gmail.com> writes:
On Sunday, 7 June 2015 at 18:27:16 UTC, Robert burner Schadek 
wrote:
 Phobos is awesome, the libs of go, python and rust only have 
 better marketing.
 As discussed on dconf, phobos needs to become big and blow the 
 rest out of the sky.

 http://wiki.dlang.org/DIP80

 lets get OT, please discuss
There are https://github.com/9il/simple_matrix and https://github.com/9il/cblas . I will try to rework them for Phobos. Any ideas and suggestions? Some notes about portability: 1. OS X has Accelerated framework builtin. 2. Linux has blast by default or it can be easily installed. However default blast is very slow. The openBLAS is preferred. 3. Looks like there is no simple way to have BLAS support on Windows. Should we provide BLAS library with DMD for Windows and maybe Linux? Regards, Ilya
Jun 08 2015
next sibling parent "Ilya Yaroshenko" <ilyayaroshenko gmail.com> writes:
On Tuesday, 9 June 2015 at 03:26:25 UTC, Ilya Yaroshenko wrote:
 On Sunday, 7 June 2015 at 18:27:16 UTC, Robert burner Schadek 
 wrote:
 Phobos is awesome, the libs of go, python and rust only have 
 better marketing.
 As discussed on dconf, phobos needs to become big and blow the 
 rest out of the sky.

 http://wiki.dlang.org/DIP80

 lets get OT, please discuss
There are https://github.com/9il/simple_matrix and https://github.com/9il/cblas . I will try to rework them for Phobos. Any ideas and suggestions? Some notes about portability: 1. OS X has Accelerated framework builtin. 2. Linux has blast by default or it can be easily installed. However default blast is very slow. The openBLAS is preferred. 3. Looks like there is no simple way to have BLAS support on Windows. Should we provide BLAS library with DMD for Windows and maybe Linux? Regards, Ilya
... probably std.container.Array is good template to start.
Jun 08 2015
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/8/15 8:26 PM, Ilya Yaroshenko wrote:
 On Sunday, 7 June 2015 at 18:27:16 UTC, Robert burner Schadek wrote:
 Phobos is awesome, the libs of go, python and rust only have better
 marketing.
 As discussed on dconf, phobos needs to become big and blow the rest
 out of the sky.

 http://wiki.dlang.org/DIP80

 lets get OT, please discuss
There are https://github.com/9il/simple_matrix and https://github.com/9il/cblas . I will try to rework them for Phobos. Any ideas and suggestions? Some notes about portability: 1. OS X has Accelerated framework builtin. 2. Linux has blast by default or it can be easily installed. However default blast is very slow. The openBLAS is preferred. 3. Looks like there is no simple way to have BLAS support on Windows. Should we provide BLAS library with DMD for Windows and maybe Linux?
I think licensing matters would make this difficult. What I do think we can do is: (a) Provide standard data layouts in std.array for the typical shapes supported by linear algebra libs: row major, column major, alongside with striding primitives. (b) Provide signatures for C and Fortran libraries so people who have them can use them easily with D. (c) Provide high-level wrappers on top of those functions. Andrei
Jun 08 2015
parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Tuesday, 9 June 2015 at 06:59:07 UTC, Andrei Alexandrescu 
wrote:
 On 6/8/15 8:26 PM, Ilya Yaroshenko wrote:
 On Sunday, 7 June 2015 at 18:27:16 UTC, Robert burner Schadek 
 wrote:
 Phobos is awesome, the libs of go, python and rust only have 
 better
 marketing.
 As discussed on dconf, phobos needs to become big and blow 
 the rest
 out of the sky.

 http://wiki.dlang.org/DIP80

 lets get OT, please discuss
There are https://github.com/9il/simple_matrix and https://github.com/9il/cblas . I will try to rework them for Phobos. Any ideas and suggestions? Some notes about portability: 1. OS X has Accelerated framework builtin. 2. Linux has blast by default or it can be easily installed. However default blast is very slow. The openBLAS is preferred. 3. Looks like there is no simple way to have BLAS support on Windows. Should we provide BLAS library with DMD for Windows and maybe Linux?
I think licensing matters would make this difficult. What I do think we can do is: (a) Provide standard data layouts in std.array for the typical shapes supported by linear algebra libs: row major, column major, alongside with striding primitives.
I don't think this is quite the right approach. Multidimensional arrays and matrices are about accessing and iteration over data, not data structures themselves. The standard layouts are common special cases.
 (b) Provide signatures for C and Fortran libraries so people 
 who have them can use them easily with D.

 (c) Provide high-level wrappers on top of those functions.


 Andrei
That is how e.g. numpy works and it's OK, but D can do better. Ilya, I'm very interested in discussing this further with you. I have a reasonable idea and implementation of how I would want the generic n-dimensional types in D to work, but you seem to have more experience with BLAS and LAPACK than me* and of course interfacing with them is critical. *I rarely interact with them directly.
Jun 09 2015
next sibling parent reply "Ilya Yaroshenko" <ilyayaroshenko gmail.com> writes:
On Tuesday, 9 June 2015 at 08:50:16 UTC, John Colvin wrote:
 On Tuesday, 9 June 2015 at 06:59:07 UTC, Andrei Alexandrescu 
 wrote:
 On 6/8/15 8:26 PM, Ilya Yaroshenko wrote:
 On Sunday, 7 June 2015 at 18:27:16 UTC, Robert burner Schadek 
 wrote:
 Phobos is awesome, the libs of go, python and rust only have 
 better
 marketing.
 As discussed on dconf, phobos needs to become big and blow 
 the rest
 out of the sky.

 http://wiki.dlang.org/DIP80

 lets get OT, please discuss
There are https://github.com/9il/simple_matrix and https://github.com/9il/cblas . I will try to rework them for Phobos. Any ideas and suggestions? Some notes about portability: 1. OS X has Accelerated framework builtin. 2. Linux has blast by default or it can be easily installed. However default blast is very slow. The openBLAS is preferred. 3. Looks like there is no simple way to have BLAS support on Windows. Should we provide BLAS library with DMD for Windows and maybe Linux?
I think licensing matters would make this difficult. What I do think we can do is: (a) Provide standard data layouts in std.array for the typical shapes supported by linear algebra libs: row major, column major, alongside with striding primitives.
I don't think this is quite the right approach. Multidimensional arrays and matrices are about accessing and iteration over data, not data structures themselves. The standard layouts are common special cases.
Probably we need both approaches: [1]. Multidimensional random access slices (ranges, not only arrays) We can do it easily: size_t anyNumber; auto ar = new int[3 * 8 * 9 + anyNumber]; auto slice = Slice[0..3, 4..8, 1..9]; assert(ar.canBeSlicedWith(slice)); //checks that ar.length <= 3 * 8 * 9 auto tensor = ar.sliced(slice); tensor[0, 1, 2] = 4; auto matrix = tensor[0..$, 1, 0..$]; assert(matrix[0, 2] == 4); [2]. BLAS Transposed.no/yes and Major.row/column (naming can be changed) flags for plain 2D matrixes based on 2.1 D arrays (both GC and manual memory management) 2.2 std.container.array (RefCounted) RowMajor and RowMinor are not needed if Transposed is already defined. But this stuff helps engineers implement software in terms of corresponding mathematical documentation. I hope to create nogc versions for 2.1 and 2.2 (because GC is not needed for slices). Furthermore [2] can be based on [1].
 (b) Provide signatures for C and Fortran libraries so people 
 who have them can use them easily with D.

 (c) Provide high-level wrappers on top of those functions.


 Andrei
That is how e.g. numpy works and it's OK, but D can do better. Ilya, I'm very interested in discussing this further with you. I have a reasonable idea and implementation of how I would want the generic n-dimensional types in D to work, but you seem to have more experience with BLAS and LAPACK than me* and of course interfacing with them is critical. *I rarely interact with them directly.
John, please describe your ideas and use cases. I think github issues is more convenient place. You have opened https://github.com/kyllingstad/scid/issues/24 . So I think we can past our code examples at SciD issue.
Jun 09 2015
parent "Ilya Yaroshenko" <ilyayaroshenko gmail.com> writes:
 size_t anyNumber;
 auto ar = new int[3 * 8 * 9 + anyNumber];
 auto slice = Slice[0..3, 4..8, 1..9];
 assert(ar.canBeSlicedWith(slice)); //checks that ar.length <= 3 
 * 8 * 9

 auto tensor = ar.sliced(slice);
 tensor[0, 1, 2] = 4;

 auto matrix = tensor[0..$, 1, 0..$];
 assert(matrix[0, 2] == 4);
assert(&matrix[0, 2] is &tensor[0, 1, 2]);
Jun 09 2015
prev sibling next sibling parent "Ilya Yaroshenko" <ilyayaroshenko gmail.com> writes:
 Ilya, I'm very interested in discussing this further with you. 
 I have a reasonable idea and implementation of how I would want 
 the generic n-dimensional types in D to work, but you seem to 
 have more experience with BLAS and LAPACK than me* and of 
 course interfacing with them is critical.

 *I rarely interact with them directly.
I have created Phobos PR. Now we can discuss it at GitHub. https://github.com/D-Programming-Language/phobos/pull/3397
Jun 09 2015
prev sibling next sibling parent reply "Dennis Ritchie" <dennis.ritchie mail.ru> writes:
On Tuesday, 9 June 2015 at 08:50:16 UTC, John Colvin wrote:
 I don't think this is quite the right approach. 
 Multidimensional arrays and matrices are about accessing and 
 iteration over data, not data structures themselves. The 
 standard layouts are common special cases.
Yes, I really want to D supports multidimensional arrays, matrices, rational numbers and quaternions. I believe that Phobos must support some common methods of linear algebra and general mathematics. I have no desire to join D with Fortran libraries :) Rational numbers and quaternions have already been implemented are: https://github.com/k3kaimu/carbon/blob/master/source/carbon/rational.d https://github.com/k3kaimu/carbon/blob/master/source/carbon/quaternion.d Satisfactory developments with matrices have here: https://github.com/k3kaimu/carbon/blob/master/source/carbon/linear.d
Jun 09 2015
parent reply "Ilya Yaroshenko" <ilyayaroshenko gmail.com> writes:
 I believe that Phobos must support some common methods of 
 linear algebra and general mathematics. I have no desire to 
 join D with Fortran libraries :)
D definitely needs BLAS API support for matrix multiplication. Best BLAS libraries are written in assembler like openBLAS. Otherwise D will have last position in corresponding math benchmarks.
Jun 09 2015
next sibling parent reply "Dennis Ritchie" <dennis.ritchie mail.ru> writes:
On Tuesday, 9 June 2015 at 15:26:43 UTC, Ilya Yaroshenko wrote:
 D definitely needs BLAS API support for matrix multiplication. 
 Best BLAS libraries are written in assembler like openBLAS. 
 Otherwise D will have last position in corresponding math 
 benchmarks.
Yes, those programs on D, is clearly lagging behind the programmers Wolfram Mathematica :) https://projecteuler.net/language=D https://projecteuler.net/language=Mathematica To solve these problems you need something like Blas. Perhaps BLAS - it's more practical way to enrich D techniques for working with matrices.
Jun 09 2015
next sibling parent reply "Dennis Ritchie" <dennis.ritchie mail.ru> writes:
On Tuesday, 9 June 2015 at 16:14:24 UTC, Dennis Ritchie wrote:
 To solve these problems you need something like Blas. Perhaps 
 BLAS - it's more practical way to enrich D techniques for 
 working with matrices.
Actually, that's what you need to realize in D: http://www.boost.org/doc/libs/1_58_0/libs/numeric/ublas/doc/index.html
Jun 09 2015
next sibling parent "Ilya Yaroshenko" <ilyayaroshenko gmail.com> writes:
On Tuesday, 9 June 2015 at 16:16:39 UTC, Dennis Ritchie wrote:
 On Tuesday, 9 June 2015 at 16:14:24 UTC, Dennis Ritchie wrote:
 To solve these problems you need something like Blas. Perhaps 
 BLAS - it's more practical way to enrich D techniques for 
 working with matrices.
Actually, that's what you need to realize in D: http://www.boost.org/doc/libs/1_58_0/libs/numeric/ublas/doc/index.html
This is very good stuff. However I want to create something more simple: [1]. n-dimensional slices (without matrix multiplication, "RowMajor/..." and other math features) [2]. netlib like standart CBLAS API at `etc.blas.cblas` [3]. High level bindings to connect [1] and 1-2D subset of [2].
Jun 09 2015
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/9/15 9:16 AM, Dennis Ritchie wrote:
 On Tuesday, 9 June 2015 at 16:14:24 UTC, Dennis Ritchie wrote:
 To solve these problems you need something like Blas. Perhaps BLAS -
 it's more practical way to enrich D techniques for working with matrices.
Actually, that's what you need to realize in D: http://www.boost.org/doc/libs/1_58_0/libs/numeric/ublas/doc/index.html
"And finally uBLAS offers good (but not outstanding) performance." -- Andrei
Jun 09 2015
parent reply "Dennis Ritchie" <dennis.ritchie mail.ru> writes:
On Tuesday, 9 June 2015 at 17:19:28 UTC, Andrei Alexandrescu 
wrote:
 On 6/9/15 9:16 AM, Dennis Ritchie wrote:
 On Tuesday, 9 June 2015 at 16:14:24 UTC, Dennis Ritchie wrote:
 To solve these problems you need something like Blas. Perhaps 
 BLAS -
 it's more practical way to enrich D techniques for working 
 with matrices.
Actually, that's what you need to realize in D: http://www.boost.org/doc/libs/1_58_0/libs/numeric/ublas/doc/index.html
"And finally uBLAS offers good (but not outstanding) performance." -- Andrei
OK, but... Same thing I can say about BigInt in Phobos. "And finally `std.bigint` offers good (but not outstanding) performance." I decided 17 math problems and for most of them I needed a `BigInt`: http://i.imgur.com/CmOSm7V.png https://projecteuler.net/language=D If in D would not be `BigInt`, I probably would have used to Boost.Multiprekison on C++: http://www.boost.org/doc/libs/1_58_0/libs/multiprecision/doc/html/index.html Or do some slow Python. Maybe all this and does not give a huge performance, but for a wide range of mathematical problems it all helps. Thus, it is better to have something than nothing :) And BLAS is more than something...
Jun 09 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/9/15 11:42 AM, Dennis Ritchie wrote:
 "And finally `std.bigint` offers good (but not outstanding) performance."
BigInt should use reference counting. Its current approach to allocating new memory for everything is a liability. Could someone file a report for this please. -- Andrei
Jun 09 2015
next sibling parent reply "Dennis Ritchie" <dennis.ritchie mail.ru> writes:
On Tuesday, 9 June 2015 at 18:58:56 UTC, Andrei Alexandrescu 
wrote:
 On 6/9/15 11:42 AM, Dennis Ritchie wrote:
 "And finally `std.bigint` offers good (but not outstanding) 
 performance."
BigInt should use reference counting. Its current approach to allocating new memory for everything is a liability. Could someone file a report for this please. -- Andrei
Done: https://issues.dlang.org/show_bug.cgi?id=14673
Jun 09 2015
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/9/15 12:21 PM, Dennis Ritchie wrote:
 On Tuesday, 9 June 2015 at 18:58:56 UTC, Andrei Alexandrescu wrote:
 On 6/9/15 11:42 AM, Dennis Ritchie wrote:
 "And finally `std.bigint` offers good (but not outstanding)
 performance."
BigInt should use reference counting. Its current approach to allocating new memory for everything is a liability. Could someone file a report for this please. -- Andrei
Done: https://issues.dlang.org/show_bug.cgi?id=14673
Thanks! -- Andrei
Jun 09 2015
prev sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/9/15 2:59 PM, Andrei Alexandrescu wrote:
 On 6/9/15 11:42 AM, Dennis Ritchie wrote:
 "And finally `std.bigint` offers good (but not outstanding) performance."
BigInt should use reference counting. Its current approach to allocating new memory for everything is a liability. Could someone file a report for this please. -- Andrei
Slightly OT, but this reminds me. RefCounted is not viable when using the GC, because any references on the heap may race against stack-based references. Can we make RefCounted use atomicInc and atomicDec? It will hurt performance a bit, but the current state is not good. I spoke with Erik about this, as he was planning on using RefCounted, but didn't know about the hairy issues with the GC. If we get to a point where we can have a thread-local GC, we can remove the implementation detail of using atomic operations when possible. -Steve
Jun 09 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/9/15 1:53 PM, Steven Schveighoffer wrote:
 On 6/9/15 2:59 PM, Andrei Alexandrescu wrote:
 On 6/9/15 11:42 AM, Dennis Ritchie wrote:
 "And finally `std.bigint` offers good (but not outstanding)
 performance."
BigInt should use reference counting. Its current approach to allocating new memory for everything is a liability. Could someone file a report for this please. -- Andrei
Slightly OT, but this reminds me. RefCounted is not viable when using the GC, because any references on the heap may race against stack-based references.
How do you mean that?
 Can we make RefCounted use atomicInc and atomicDec? It will hurt
 performance a bit, but the current state is not good.

 I spoke with Erik about this, as he was planning on using RefCounted,
 but didn't know about the hairy issues with the GC.

 If we get to a point where we can have a thread-local GC, we can remove
 the implementation detail of using atomic operations when possible.
The obvious solution that comes to mind is adding a Flag!"interlocked". -- Andrei
Jun 09 2015
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/9/15 5:46 PM, Andrei Alexandrescu wrote:
 On 6/9/15 1:53 PM, Steven Schveighoffer wrote:
 On 6/9/15 2:59 PM, Andrei Alexandrescu wrote:
 On 6/9/15 11:42 AM, Dennis Ritchie wrote:
 "And finally `std.bigint` offers good (but not outstanding)
 performance."
BigInt should use reference counting. Its current approach to allocating new memory for everything is a liability. Could someone file a report for this please. -- Andrei
Slightly OT, but this reminds me. RefCounted is not viable when using the GC, because any references on the heap may race against stack-based references.
How do you mean that?
If you add an instance of RefCounted to a GC-destructed type (either in an array, or as a member of a class), there is the potential that the GC will run the dtor of the RefCounted item in a different thread, opening up the possibility of races.
 Can we make RefCounted use atomicInc and atomicDec? It will hurt
 performance a bit, but the current state is not good.

 I spoke with Erik about this, as he was planning on using RefCounted,
 but didn't know about the hairy issues with the GC.

 If we get to a point where we can have a thread-local GC, we can remove
 the implementation detail of using atomic operations when possible.
The obvious solution that comes to mind is adding a Flag!"interlocked".
Can you explain it further? It's not obvious to me. -Steve
Jun 10 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/10/15 3:52 AM, Steven Schveighoffer wrote:
 On 6/9/15 5:46 PM, Andrei Alexandrescu wrote:
 On 6/9/15 1:53 PM, Steven Schveighoffer wrote:
 On 6/9/15 2:59 PM, Andrei Alexandrescu wrote:
 On 6/9/15 11:42 AM, Dennis Ritchie wrote:
 "And finally `std.bigint` offers good (but not outstanding)
 performance."
BigInt should use reference counting. Its current approach to allocating new memory for everything is a liability. Could someone file a report for this please. -- Andrei
Slightly OT, but this reminds me. RefCounted is not viable when using the GC, because any references on the heap may race against stack-based references.
How do you mean that?
If you add an instance of RefCounted to a GC-destructed type (either in an array, or as a member of a class), there is the potential that the GC will run the dtor of the RefCounted item in a different thread, opening up the possibility of races.
That's a problem with the GC. Collected memory must be deallocated in the thread that allocated it. It's not really that complicated to implement, either - the collection process puts the memory to deallocate in a per-thread freelist; then when each thread wakes up and tries to allocate things, it first allocates from the freelist.
 Can we make RefCounted use atomicInc and atomicDec? It will hurt
 performance a bit, but the current state is not good.

 I spoke with Erik about this, as he was planning on using RefCounted,
 but didn't know about the hairy issues with the GC.

 If we get to a point where we can have a thread-local GC, we can remove
 the implementation detail of using atomic operations when possible.
The obvious solution that comes to mind is adding a Flag!"interlocked".
Can you explain it further? It's not obvious to me.
The RefCounted type could have a flag as a template parameter. Andrei
Jun 10 2015
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/10/15 11:49 AM, Andrei Alexandrescu wrote:
 On 6/10/15 3:52 AM, Steven Schveighoffer wrote:
 On 6/9/15 5:46 PM, Andrei Alexandrescu wrote:
 On 6/9/15 1:53 PM, Steven Schveighoffer wrote:
 On 6/9/15 2:59 PM, Andrei Alexandrescu wrote:
 On 6/9/15 11:42 AM, Dennis Ritchie wrote:
 "And finally `std.bigint` offers good (but not outstanding)
 performance."
BigInt should use reference counting. Its current approach to allocating new memory for everything is a liability. Could someone file a report for this please. -- Andrei
Slightly OT, but this reminds me. RefCounted is not viable when using the GC, because any references on the heap may race against stack-based references.
How do you mean that?
If you add an instance of RefCounted to a GC-destructed type (either in an array, or as a member of a class), there is the potential that the GC will run the dtor of the RefCounted item in a different thread, opening up the possibility of races.
That's a problem with the GC. Collected memory must be deallocated in the thread that allocated it. It's not really that complicated to implement, either - the collection process puts the memory to deallocate in a per-thread freelist; then when each thread wakes up and tries to allocate things, it first allocates from the freelist.
I agree it's a problem with the GC, but not that it's a simple fix. It's not just a freelist -- the dtor needs to be run in the thread also. But the amount of affected code (i.e. any code that uses GC) makes this a very high risk change, whereas changing RefCounted is a 2-line change that is easy to prove/review. I will make the RefCounted atomic PR if you can accept that.
 Can we make RefCounted use atomicInc and atomicDec? It will hurt
 performance a bit, but the current state is not good.

 I spoke with Erik about this, as he was planning on using RefCounted,
 but didn't know about the hairy issues with the GC.

 If we get to a point where we can have a thread-local GC, we can remove
 the implementation detail of using atomic operations when possible.
The obvious solution that comes to mind is adding a Flag!"interlocked".
Can you explain it further? It's not obvious to me.
The RefCounted type could have a flag as a template parameter.
OK, thanks for the explanation. I'd do it the other way around: Flag!"threadlocal", since we should be safe by default. -Steve
Jun 10 2015
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Wednesday, 10 June 2015 at 20:31:52 UTC, Steven Schveighoffer 
wrote:
 OK, thanks for the explanation. I'd do it the other way around: 
 Flag!"threadlocal", since we should be safe by default.
`RefCounted!T` is also thread-local by default, only `shared(RefCounted!T)` needs to use atomic operations.
Jun 11 2015
parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 6/11/15 4:15 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" 
wrote:
 On Wednesday, 10 June 2015 at 20:31:52 UTC, Steven Schveighoffer wrote:
 OK, thanks for the explanation. I'd do it the other way around:
 Flag!"threadlocal", since we should be safe by default.
`RefCounted!T` is also thread-local by default, only `shared(RefCounted!T)` needs to use atomic operations.
I may have misunderstood Andrei. We can't just use a flag to fix this problem, all allocations are in danger of races (even thread-local ones). But maybe he meant *after* we fix the GC we could add a flag? I'm not sure. A flag at this point would be a band-aid fix, allowing one to optimize if one knows that his code never puts RefCounted instances on the heap. Hard to prove... -Steve
Jun 11 2015
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/11/15 5:17 AM, Steven Schveighoffer wrote:
 On 6/11/15 4:15 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>"
 wrote:
 On Wednesday, 10 June 2015 at 20:31:52 UTC, Steven Schveighoffer wrote:
 OK, thanks for the explanation. I'd do it the other way around:
 Flag!"threadlocal", since we should be safe by default.
`RefCounted!T` is also thread-local by default, only `shared(RefCounted!T)` needs to use atomic operations.
I may have misunderstood Andrei. We can't just use a flag to fix this problem, all allocations are in danger of races (even thread-local ones). But maybe he meant *after* we fix the GC we could add a flag? I'm not sure.
Yes, we definitely need to fix the GC. -- Andrei
Jun 11 2015
prev sibling parent reply "ixid" <adamsibson hotmail.com> writes:
On Tuesday, 9 June 2015 at 16:14:24 UTC, Dennis Ritchie wrote:
 On Tuesday, 9 June 2015 at 15:26:43 UTC, Ilya Yaroshenko wrote:
 D definitely needs BLAS API support for matrix multiplication. 
 Best BLAS libraries are written in assembler like openBLAS. 
 Otherwise D will have last position in corresponding math 
 benchmarks.
Yes, those programs on D, is clearly lagging behind the programmers Wolfram Mathematica :) https://projecteuler.net/language=D https://projecteuler.net/language=Mathematica To solve these problems you need something like Blas. Perhaps BLAS - it's more practical way to enrich D techniques for working with matrices.
I suspect this is more about who the Mathematica and D users are as Project Euler is mostly mathematical rather than code optimization. More of the Mathematica users would have strong maths backgrounds. I haven't felt held back by D at all, it's only been my own lack of ability. I'm in 2nd place atm for D users.
Jun 10 2015
next sibling parent "Dennis Ritchie" <dennis.ritchie mail.ru> writes:
On Wednesday, 10 June 2015 at 08:39:12 UTC, ixid wrote:
 I suspect this is more about who the Mathematica and D users 
 are as Project Euler is mostly mathematical rather than code 
 optimization. More of the Mathematica users would have strong 
 maths backgrounds. I haven't felt held back by D at all, it's 
 only been my own lack of ability. I'm in 2nd place atm for D 
 users.
OK, if D is at least BLAS, I will try to overtake you :)
Jun 10 2015
prev sibling parent reply "Dennis Ritchie" <dennis.ritchie mail.ru> writes:
On Wednesday, 10 June 2015 at 08:39:12 UTC, ixid wrote:
 I suspect this is more about who the Mathematica and D users 
 are as Project Euler is mostly mathematical rather than code 
 optimization.
Here and I say that despite the fact that in D BigInt not optimized very well, it helps me to solve a wide range of tasks that do not require high performance, so I want to BLAS or something similar was in D. Something is better than nothing!
Jun 10 2015
parent reply "ixid" <adamsibson hotmail.com> writes:
On Wednesday, 10 June 2015 at 08:50:31 UTC, Dennis Ritchie wrote:
 On Wednesday, 10 June 2015 at 08:39:12 UTC, ixid wrote:
 I suspect this is more about who the Mathematica and D users 
 are as Project Euler is mostly mathematical rather than code 
 optimization.
Here and I say that despite the fact that in D BigInt not optimized very well, it helps me to solve a wide range of tasks that do not require high performance, so I want to BLAS or something similar was in D. Something is better than nothing!
You rarely need to use BigInt for heavy lifting though, often it's just summing, not that I would argue against optimization. I think speed is absolutely vital and one of the most powerful things we could do to promote D would be to run the best benchmarks site for all language comers and make sure D does very well. Every time there's a benchmark contest it seems to unearth D performance issues that can be greatly improved upon. I'm sure you will beat me pretty quickly, as I said my maths isn't very good but it might motivate me to solve some more! =)
Jun 10 2015
parent "Dennis Ritchie" <dennis.ritchie mail.ru> writes:
On Wednesday, 10 June 2015 at 09:43:47 UTC, ixid wrote:
 You rarely need to use BigInt for heavy lifting though, often 
 it's just summing, not that I would argue against optimization. 
 I think speed is absolutely vital and one of the most powerful 
 things we could do to promote D would be to run the best 
 benchmarks site for all language comers and make sure D does 
 very well. Every time there's a benchmark contest it seems to 
 unearth D performance issues that can be greatly improved upon.
Yes it is. Many are trying to find performance problems D. And sometimes it turns out.
 I'm sure you will beat me pretty quickly, as I said my maths 
 isn't very good but it might motivate me to solve some more! =)
No, I will start to beat you until next year, because, unfortunately, I will not have a full year of access to the computer. We can say that this is something like a long vacation :)
Jun 10 2015
prev sibling next sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 10 June 2015 at 01:26, Ilya Yaroshenko via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 I believe that Phobos must support some common methods of linear algebra
 and general mathematics. I have no desire to join D with Fortran libraries
 :)
D definitely needs BLAS API support for matrix multiplication. Best BLAS libraries are written in assembler like openBLAS. Otherwise D will have last position in corresponding math benchmarks.
A complication for linear algebra (or other mathsy things in general) is the inability to detect and implement compound operations. We don't declare mathematical operators to be algebraic operations, which I think is a lost opportunity. If we defined the properties along with their properties (commutativity, transitivity, invertibility, etc), then the compiler could potentially do an algebraic simplification on expressions before performing codegen and optimisation. There are a lot of situations where the optimiser can't simplify expressions because it runs into an arbitrary function call, and I've never seen an optimiser that understands exp/log/roots, etc, to the point where it can reduce those expressions properly. To compete with maths benchmarks, we need some means to simplify expressions properly.
Jun 09 2015
next sibling parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Tuesday, 9 June 2015 at 16:18:06 UTC, Manu wrote:
 On 10 June 2015 at 01:26, Ilya Yaroshenko via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:
 I believe that Phobos must support some common methods of 
 linear algebra and general mathematics. I have no desire to 
 join D with Fortran libraries :)
D definitely needs BLAS API support for matrix multiplication. Best BLAS libraries are written in assembler like openBLAS. Otherwise D will have last position in corresponding math benchmarks.
A complication for linear algebra (or other mathsy things in general) is the inability to detect and implement compound operations. We don't declare mathematical operators to be algebraic operations, which I think is a lost opportunity. If we defined the properties along with their properties (commutativity, transitivity, invertibility, etc), then the compiler could potentially do an algebraic simplification on expressions before performing codegen and optimisation. There are a lot of situations where the optimiser can't simplify expressions because it runs into an arbitrary function call, and I've never seen an optimiser that understands exp/log/roots, etc, to the point where it can reduce those expressions properly. To compete with maths benchmarks, we need some means to simplify expressions properly.
Optimising floating point is a massive pain because of precision concerns and IEEE-754 conformance. Just because something is analytically the same doesn't mean you want the optimiser to go ahead and make the switch for you. Of the things that can be done, lazy operations should make it easier/possible for the optimiser to spot.
Jun 09 2015
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 10 June 2015 at 02:32, John Colvin via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Tuesday, 9 June 2015 at 16:18:06 UTC, Manu wrote:
 On 10 June 2015 at 01:26, Ilya Yaroshenko via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 I believe that Phobos must support some common methods of linear algebra
 and general mathematics. I have no desire to join D with Fortran libraries
 :)
D definitely needs BLAS API support for matrix multiplication. Best BLAS libraries are written in assembler like openBLAS. Otherwise D will have last position in corresponding math benchmarks.
A complication for linear algebra (or other mathsy things in general) is the inability to detect and implement compound operations. We don't declare mathematical operators to be algebraic operations, which I think is a lost opportunity. If we defined the properties along with their properties (commutativity, transitivity, invertibility, etc), then the compiler could potentially do an algebraic simplification on expressions before performing codegen and optimisation. There are a lot of situations where the optimiser can't simplify expressions because it runs into an arbitrary function call, and I've never seen an optimiser that understands exp/log/roots, etc, to the point where it can reduce those expressions properly. To compete with maths benchmarks, we need some means to simplify expressions properly.
Optimising floating point is a massive pain because of precision concerns and IEEE-754 conformance. Just because something is analytically the same doesn't mean you want the optimiser to go ahead and make the switch for you.
We have flags to control this sort of thing (fast-math, strict ieee, etc). I will worry about my precision, I just want the optimiser to do its job and do the very best it possibly can. In the case of linear algebra, the optimiser generally fails and I must manually simplify expressions as much as possible. In the event the expressions emerge as a result of a series of inlines, or generic code (the sort that appears frequently as a result of stream/range based programming), then there's nothing you can do except to flatten and unroll your work loops yourself.
 Of the things that can be done, lazy operations should make it
 easier/possible for the optimiser to spot.
My experience is that they possibly make it harder, although I don't know why. I find the compiler becomes very unpredictable optimising deep lazy expressions. The backend inline heuristics may not be tuned for typical D expressions of this type? I often wish I could address common compound operations myself, by implementing something like a compound operator which I can special case with an optimised path for particular expressions. But I can't think of any reasonable ways to approach that.
Jun 09 2015
parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Tuesday, 9 June 2015 at 16:45:33 UTC, Manu wrote:
 On 10 June 2015 at 02:32, John Colvin via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:
 On Tuesday, 9 June 2015 at 16:18:06 UTC, Manu wrote:
 On 10 June 2015 at 01:26, Ilya Yaroshenko via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:
 [...]
A complication for linear algebra (or other mathsy things in general) is the inability to detect and implement compound operations. We don't declare mathematical operators to be algebraic operations, which I think is a lost opportunity. If we defined the properties along with their properties (commutativity, transitivity, invertibility, etc), then the compiler could potentially do an algebraic simplification on expressions before performing codegen and optimisation. There are a lot of situations where the optimiser can't simplify expressions because it runs into an arbitrary function call, and I've never seen an optimiser that understands exp/log/roots, etc, to the point where it can reduce those expressions properly. To compete with maths benchmarks, we need some means to simplify expressions properly.
Optimising floating point is a massive pain because of precision concerns and IEEE-754 conformance. Just because something is analytically the same doesn't mean you want the optimiser to go ahead and make the switch for you.
We have flags to control this sort of thing (fast-math, strict ieee, etc). I will worry about my precision, I just want the optimiser to do its job and do the very best it possibly can. In the case of linear algebra, the optimiser generally fails and I must manually simplify expressions as much as possible.
If the compiler is free to rewrite by analytical rules then "I will worry about my precision" is equivalent to either "I don't care about my precision" or "I have checked the codegen". A simple rearrangement of an expression can easily turn a perfectly good result in to complete garbage. It would be great if compilers were even better at fast-math mode, but an awful lot of applications can't use it.
Jun 09 2015
parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 10 June 2015 at 03:04, John Colvin via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Tuesday, 9 June 2015 at 16:45:33 UTC, Manu wrote:
 On 10 June 2015 at 02:32, John Colvin via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On Tuesday, 9 June 2015 at 16:18:06 UTC, Manu wrote:
 On 10 June 2015 at 01:26, Ilya Yaroshenko via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 [...]
A complication for linear algebra (or other mathsy things in general) is the inability to detect and implement compound operations. We don't declare mathematical operators to be algebraic operations, which I think is a lost opportunity. If we defined the properties along with their properties (commutativity, transitivity, invertibility, etc), then the compiler could potentially do an algebraic simplification on expressions before performing codegen and optimisation. There are a lot of situations where the optimiser can't simplify expressions because it runs into an arbitrary function call, and I've never seen an optimiser that understands exp/log/roots, etc, to the point where it can reduce those expressions properly. To compete with maths benchmarks, we need some means to simplify expressions properly.
Optimising floating point is a massive pain because of precision concerns and IEEE-754 conformance. Just because something is analytically the same doesn't mean you want the optimiser to go ahead and make the switch for you.
We have flags to control this sort of thing (fast-math, strict ieee, etc). I will worry about my precision, I just want the optimiser to do its job and do the very best it possibly can. In the case of linear algebra, the optimiser generally fails and I must manually simplify expressions as much as possible.
If the compiler is free to rewrite by analytical rules then "I will worry about my precision" is equivalent to either "I don't care about my precision" or "I have checked the codegen". A simple rearrangement of an expression can easily turn a perfectly good result in to complete garbage. It would be great if compilers were even better at fast-math mode, but an awful lot of applications can't use it.
This is fine, those applications would continue not to use it. Personally, I've never written code in 20 years where I didn't want fast-math.
Jun 11 2015
prev sibling parent reply "Ilya Yaroshenko" <ilyayaroshenko gmail.com> writes:
On Tuesday, 9 June 2015 at 16:18:06 UTC, Manu wrote:
 On 10 June 2015 at 01:26, Ilya Yaroshenko via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 I believe that Phobos must support some common methods of 
 linear algebra
 and general mathematics. I have no desire to join D with 
 Fortran libraries
 :)
D definitely needs BLAS API support for matrix multiplication. Best BLAS libraries are written in assembler like openBLAS. Otherwise D will have last position in corresponding math benchmarks.
A complication for linear algebra (or other mathsy things in general) is the inability to detect and implement compound operations. We don't declare mathematical operators to be algebraic operations, which I think is a lost opportunity. If we defined the properties along with their properties (commutativity, transitivity, invertibility, etc), then the compiler could potentially do an algebraic simplification on expressions before performing codegen and optimisation. There are a lot of situations where the optimiser can't simplify expressions because it runs into an arbitrary function call, and I've never seen an optimiser that understands exp/log/roots, etc, to the point where it can reduce those expressions properly. To compete with maths benchmarks, we need some means to simplify expressions properly.
Simplified expressions would help because 1. On matrix (hight) level optimisation can be done very well by programer (algorithms with matrixes in terms of count of matrix multiplications are small). 2. Low level optimisation requires specific CPU/Cache optimisation. Modern implementations are optimised for all cache levels. See work by KAZUSHIGE GOTO http://www.cs.utexas.edu/users/pingali/CS378/2008sp/papers/gotoPaper.pdf
Jun 09 2015
next sibling parent "Ilya Yaroshenko" <ilyayaroshenko gmail.com> writes:
On Tuesday, 9 June 2015 at 16:40:56 UTC, Ilya Yaroshenko wrote:
 On Tuesday, 9 June 2015 at 16:18:06 UTC, Manu wrote:
 On 10 June 2015 at 01:26, Ilya Yaroshenko via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 I believe that Phobos must support some common methods of 
 linear algebra
 and general mathematics. I have no desire to join D with 
 Fortran libraries
 :)
D definitely needs BLAS API support for matrix multiplication. Best BLAS libraries are written in assembler like openBLAS. Otherwise D will have last position in corresponding math benchmarks.
A complication for linear algebra (or other mathsy things in general) is the inability to detect and implement compound operations. We don't declare mathematical operators to be algebraic operations, which I think is a lost opportunity. If we defined the properties along with their properties (commutativity, transitivity, invertibility, etc), then the compiler could potentially do an algebraic simplification on expressions before performing codegen and optimisation. There are a lot of situations where the optimiser can't simplify expressions because it runs into an arbitrary function call, and I've never seen an optimiser that understands exp/log/roots, etc, to the point where it can reduce those expressions properly. To compete with maths benchmarks, we need some means to simplify expressions properly.
Simplified expressions would help because 1. On matrix (hight) level optimisation can be done very well by programer (algorithms with matrixes in terms of count of matrix multiplications are small). 2. Low level optimisation requires specific CPU/Cache optimisation. Modern implementations are optimised for all cache levels. See work by KAZUSHIGE GOTO http://www.cs.utexas.edu/users/pingali/CS378/2008sp/papers/gotoPaper.pdf
EDIT: would NOT help
Jun 09 2015
prev sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 10 June 2015 at 02:40, Ilya Yaroshenko via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Tuesday, 9 June 2015 at 16:18:06 UTC, Manu wrote:
 On 10 June 2015 at 01:26, Ilya Yaroshenko via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 I believe that Phobos must support some common methods of linear algebra
 and general mathematics. I have no desire to join D with Fortran
 libraries
 :)
D definitely needs BLAS API support for matrix multiplication. Best BLAS libraries are written in assembler like openBLAS. Otherwise D will have last position in corresponding math benchmarks.
A complication for linear algebra (or other mathsy things in general) is the inability to detect and implement compound operations. We don't declare mathematical operators to be algebraic operations, which I think is a lost opportunity. If we defined the properties along with their properties (commutativity, transitivity, invertibility, etc), then the compiler could potentially do an algebraic simplification on expressions before performing codegen and optimisation. There are a lot of situations where the optimiser can't simplify expressions because it runs into an arbitrary function call, and I've never seen an optimiser that understands exp/log/roots, etc, to the point where it can reduce those expressions properly. To compete with maths benchmarks, we need some means to simplify expressions properly.
Simplified expressions would [NOT] help because 1. On matrix (hight) level optimisation can be done very well by programer (algorithms with matrixes in terms of count of matrix multiplications are small).
Perhaps you've never worked with incompetent programmers (in my experience, >50% of the professional workforce). Programmers, on average, don't know maths. They literally have no idea how to simplify an algebraic expression. I think there are about 3-4 (being generous!) people in my office (of 30-40) that could do it properly, and without spending heaps of time on it.
 2. Low level optimisation requires specific CPU/Cache optimisation. Modern
 implementations are optimised for all cache levels. See work by KAZUSHIGE
 GOTO
 http://www.cs.utexas.edu/users/pingali/CS378/2008sp/papers/gotoPaper.pdf
Low-level optimisation is a sliding scale, not a binary position. Reaching 'optimal' state definitely requires careful consideration of all the details you refer to, but there are a lot of improvements that can be gained from quickly written code without full low-level optimisation. A lot of basic low-level optimisations (like just using appropriate opcodes, or eliding redundant operations; ie, squares followed by sqrt) can't be applied without first simplifying expressions.
Jun 11 2015
next sibling parent "Dennis Ritchie" <dennis.ritchie mail.ru> writes:
On Friday, 12 June 2015 at 00:51:04 UTC, Manu wrote:
 Perhaps you've never worked with incompetent programmers (in my
 experience, >50% of the professional workforce).
 Programmers, on average, don't know maths. They literally have 
 no idea
 how to simplify an algebraic expression.
 I think there are about 3-4 (being generous!) people in my 
 office (of
 30-40) that could do it properly, and without spending heaps of 
 time
 on it.
But you don't think you need to look up to programmers who are not able to quickly simplify an algebraic expression? :) For example, I'm a little addicted to sports programming. And I could really use matrix and other math in the standard library.
Jun 11 2015
prev sibling parent reply "Ilya Yaroshenko" <ilyayaroshenko gmail.com> writes:
On Friday, 12 June 2015 at 00:51:04 UTC, Manu wrote:
 On 10 June 2015 at 02:40, Ilya Yaroshenko via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On Tuesday, 9 June 2015 at 16:18:06 UTC, Manu wrote:
 On 10 June 2015 at 01:26, Ilya Yaroshenko via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 I believe that Phobos must support some common methods of 
 linear algebra
 and general mathematics. I have no desire to join D with 
 Fortran
 libraries
 :)
D definitely needs BLAS API support for matrix multiplication. Best BLAS libraries are written in assembler like openBLAS. Otherwise D will have last position in corresponding math benchmarks.
A complication for linear algebra (or other mathsy things in general) is the inability to detect and implement compound operations. We don't declare mathematical operators to be algebraic operations, which I think is a lost opportunity. If we defined the properties along with their properties (commutativity, transitivity, invertibility, etc), then the compiler could potentially do an algebraic simplification on expressions before performing codegen and optimisation. There are a lot of situations where the optimiser can't simplify expressions because it runs into an arbitrary function call, and I've never seen an optimiser that understands exp/log/roots, etc, to the point where it can reduce those expressions properly. To compete with maths benchmarks, we need some means to simplify expressions properly.
Simplified expressions would [NOT] help because 1. On matrix (hight) level optimisation can be done very well by programer (algorithms with matrixes in terms of count of matrix multiplications are small).
Perhaps you've never worked with incompetent programmers (in my experience, >50% of the professional workforce). Programmers, on average, don't know maths. They literally have no idea how to simplify an algebraic expression. I think there are about 3-4 (being generous!) people in my office (of 30-40) that could do it properly, and without spending heaps of time on it.
 2. Low level optimisation requires specific CPU/Cache 
 optimisation. Modern
 implementations are optimised for all cache levels. See work 
 by KAZUSHIGE
 GOTO
 http://www.cs.utexas.edu/users/pingali/CS378/2008sp/papers/gotoPaper.pdf
Low-level optimisation is a sliding scale, not a binary position. Reaching 'optimal' state definitely requires careful consideration of all the details you refer to, but there are a lot of improvements that can be gained from quickly written code without full low-level optimisation. A lot of basic low-level optimisations (like just using appropriate opcodes, or eliding redundant operations; ie, squares followed by sqrt) can't be applied without first simplifying expressions.
OK, generally you are talking about something we can name MathD. I understand the reasons. However I am strictly against algebraic operations (or eliding redundant operations for floating points) for basic routines in system programming language. Even float/double internal conversion to real in math expressions is a huge headache when math algorithms are implemented (see first two comments at https://github.com/D-Programming-Language/phobos/pull/2991 ). In system PL sqrt(x)^2 should compiles as is. Such optimisations can be implemented over the basic routines (pow, sqrt, gemv, gemm, etc). We can use approach similar to D compile time regexp. Best, Ilya
Jun 11 2015
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 12 June 2015 at 15:22, Ilya Yaroshenko via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Friday, 12 June 2015 at 00:51:04 UTC, Manu wrote:
 On 10 June 2015 at 02:40, Ilya Yaroshenko via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On Tuesday, 9 June 2015 at 16:18:06 UTC, Manu wrote:
 On 10 June 2015 at 01:26, Ilya Yaroshenko via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 I believe that Phobos must support some common methods of linear
 algebra
 and general mathematics. I have no desire to join D with Fortran
 libraries
 :)
D definitely needs BLAS API support for matrix multiplication. Best BLAS libraries are written in assembler like openBLAS. Otherwise D will have last position in corresponding math benchmarks.
A complication for linear algebra (or other mathsy things in general) is the inability to detect and implement compound operations. We don't declare mathematical operators to be algebraic operations, which I think is a lost opportunity. If we defined the properties along with their properties (commutativity, transitivity, invertibility, etc), then the compiler could potentially do an algebraic simplification on expressions before performing codegen and optimisation. There are a lot of situations where the optimiser can't simplify expressions because it runs into an arbitrary function call, and I've never seen an optimiser that understands exp/log/roots, etc, to the point where it can reduce those expressions properly. To compete with maths benchmarks, we need some means to simplify expressions properly.
Simplified expressions would [NOT] help because 1. On matrix (hight) level optimisation can be done very well by programer (algorithms with matrixes in terms of count of matrix multiplications are small).
Perhaps you've never worked with incompetent programmers (in my experience, >50% of the professional workforce). Programmers, on average, don't know maths. They literally have no idea how to simplify an algebraic expression. I think there are about 3-4 (being generous!) people in my office (of 30-40) that could do it properly, and without spending heaps of time on it.
 2. Low level optimisation requires specific CPU/Cache optimisation.
 Modern
 implementations are optimised for all cache levels. See work by KAZUSHIGE
 GOTO
 http://www.cs.utexas.edu/users/pingali/CS378/2008sp/papers/gotoPaper.pdf
Low-level optimisation is a sliding scale, not a binary position. Reaching 'optimal' state definitely requires careful consideration of all the details you refer to, but there are a lot of improvements that can be gained from quickly written code without full low-level optimisation. A lot of basic low-level optimisations (like just using appropriate opcodes, or eliding redundant operations; ie, squares followed by sqrt) can't be applied without first simplifying expressions.
OK, generally you are talking about something we can name MathD. I understand the reasons. However I am strictly against algebraic operations (or eliding redundant operations for floating points) for basic routines in system programming language.
That's nice... I'm all for it :) Perhaps if there were some distinction between a base type and an algebraic type? I wonder if it would be possible to express an algebraic expression like a lazy range, and then capture the expression at the end and simplify it with some fancy template... I'd call that an abomination, but it might be possible. Hopefully nobody in their right mind would ever use that ;)
 Even float/double internal conversion to real
 in math expressions is a huge headache when math algorithms are implemented
 (see first two comments at
 https://github.com/D-Programming-Language/phobos/pull/2991 ). In system PL
 sqrt(x)^2  should compiles as is.
Yeah... unless you -fast-math, in which case I want the compiler to do whatever it can. Incidentally, I don't think I've ever run into a case in practise where precision was lost by doing _less_ operations.
 Such optimisations can be implemented over the basic routines (pow, sqrt,
 gemv, gemm, etc). We can use approach similar to D compile time regexp.
Not really. The main trouble is that many of these patterns only emerge when inlining is performed. It would be particularly awkward to express such expressions in some DSL that spanned across conventional API boundaries.
Jun 12 2015
parent "Ilya Yaroshenko" <ilyayaroshenko gmail.com> writes:
On Friday, 12 June 2015 at 11:00:20 UTC, Manu wrote:
 Low-level optimisation is a sliding scale, not a binary 
 position.
 Reaching 'optimal' state definitely requires careful 
 consideration of
 all the details you refer to, but there are a lot of 
 improvements that
 can be gained from quickly written code without full low-level
 optimisation. A lot of basic low-level optimisations (like 
 just using
 appropriate opcodes, or eliding redundant operations; ie, 
 squares
 followed by sqrt) can't be applied without first simplifying
 expressions.
OK, generally you are talking about something we can name MathD. I understand the reasons. However I am strictly against algebraic operations (or eliding redundant operations for floating points) for basic routines in system programming language.
That's nice... I'm all for it :) Perhaps if there were some distinction between a base type and an algebraic type? I wonder if it would be possible to express an algebraic expression like a lazy range, and then capture the expression at the end and simplify it with some fancy template... I'd call that an abomination, but it might be possible. Hopefully nobody in their right mind would ever use that ;)
... for example we can optimise matrix chain multiplication https://en.wikipedia.org/wiki/Matrix_chain_multiplication ---- //calls `this(MatrixExp!double chain)` Matrix!double = m1*m2*m3*m4; ----
 Even float/double internal conversion to real
 in math expressions is a huge headache when math algorithms 
 are implemented
 (see first two comments at
 https://github.com/D-Programming-Language/phobos/pull/2991 ). 
 In system PL
 sqrt(x)^2  should compiles as is.
Yeah... unless you -fast-math, in which case I want the compiler to do whatever it can. Incidentally, I don't think I've ever run into a case in practise where precision was lost by doing _less_ operations.
Mathematics functions requires concrete order of operations http://www.netlib.org/cephes/ (std.mathspecial and a bit of std.math/std.numeric are based on cephes).
 Such optimisations can be implemented over the basic routines 
 (pow, sqrt,
 gemv, gemm, etc). We can use approach similar to D compile 
 time regexp.
Not really. The main trouble is that many of these patterns only emerge when inlining is performed. It would be particularly awkward to express such expressions in some DSL that spanned across conventional API boundaries.
If I am not wrong in both LLVM and GCC `fast-math` attribute can be defined for functions. This feature can be implemented in D.
Jun 12 2015
prev sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 10 June 2015 at 02:17, Manu <turkeyman gmail.com> wrote:
 ... If we defined the properties along with their properties ...
*operators* along with their properties
Jun 09 2015
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/9/15 1:50 AM, John Colvin wrote:
 On Tuesday, 9 June 2015 at 06:59:07 UTC, Andrei Alexandrescu wrote:
 (a) Provide standard data layouts in std.array for the typical shapes
 supported by linear algebra libs: row major, column major, alongside
 with striding primitives.
I don't think this is quite the right approach. Multidimensional arrays and matrices are about accessing and iteration over data, not data structures themselves. The standard layouts are common special cases.
I see. So what would be the primitives necessary? Strides (in the form of e.g. special ranges)? What are the things that would make a library vendor or user go, "OK, now I know what steps to take to use my code with D"?
 (b) Provide signatures for C and Fortran libraries so people who have
 them can use them easily with D.

 (c) Provide high-level wrappers on top of those functions.


 Andrei
That is how e.g. numpy works and it's OK, but D can do better. Ilya, I'm very interested in discussing this further with you. I have a reasonable idea and implementation of how I would want the generic n-dimensional types in D to work, but you seem to have more experience with BLAS and LAPACK than me* and of course interfacing with them is critical. *I rarely interact with them directly.
Color me interested. This is another of those domains that hold great promise for D, but sadly a strong champion has been missing. Or two :o). Andrei
Jun 09 2015
parent "Ilya Yaroshenko" <ilyayaroshenko gmail.com> writes:
On Tuesday, 9 June 2015 at 16:08:40 UTC, Andrei Alexandrescu 
wrote:
 On 6/9/15 1:50 AM, John Colvin wrote:
 On Tuesday, 9 June 2015 at 06:59:07 UTC, Andrei Alexandrescu 
 wrote:
 (a) Provide standard data layouts in std.array for the 
 typical shapes
 supported by linear algebra libs: row major, column major, 
 alongside
 with striding primitives.
I don't think this is quite the right approach. Multidimensional arrays and matrices are about accessing and iteration over data, not data structures themselves. The standard layouts are common special cases.
I see. So what would be the primitives necessary? Strides (in the form of e.g. special ranges)? What are the things that would make a library vendor or user go, "OK, now I know what steps to take to use my code with D"?
N-dimensional slices can be expressed as N slices and N shifts. Where shift equals count of elements in source range between front elements of neighboring sub-slices on corresponding slice-level. private struct Slice(size_t N, Range) { size_t[2][N] slices; size_t[N] shifts; Range range; }
Jun 09 2015
prev sibling parent reply "jmh530" <john.michael.hall gmail.com> writes:
On Tuesday, 9 June 2015 at 03:26:25 UTC, Ilya Yaroshenko wrote:

 There are
 https://github.com/9il/simple_matrix and
 https://github.com/9il/cblas .
 I will try to rework them for Phobos.

 Any ideas and suggestions?
A well-supported matrix math library would definitely lead to me using D more. I would definitely applaud any work being done on this subject, but I still feel there are some enhancements (most seemingly minor) that would really make a matrix math library easy/fun to use. Most of what I discuss below is just syntactical sugar for some stuff that could be accomplished with loops or std.algorithm, but having it built-in would make practical use of a matrix math library much easier. I think Armadillo implements some of these as member functions, whereas other languages like R and Matlab have them more built-in. Disclaimer: I don't consider myself a D expert, so I could be horribly wrong on some of this stuff. 1) There is no support for assignment to arrays based on the values of another array. int[] A = [-1, 1, 5]; int[] B = [1, 2]; int[] C = A[B]; You would have to use int[] C = A[1..2];. In this simple example, it’s not really a big deal, but if I have a function that returns B, then I can’t just throw B in there. I would have to loop through B and assign it to C. So the type of assignment is possible, but if you’re frequently doing this type of array manipulation, then the number of loops you need starts increasing. 2) Along the same lines, there is no support for replacing the B above with an array of bools like bool[] B = [false, true, true]; or auto B = A.map!(a => a < 0); Again, it is doable with a loop, but this form of logical indexing is a pretty common idiom for people who use Matlab or R quite a bit. 3) In addition to being able to index by a range of values or bools, you would want to be able to make assignments based on this. So something like A[B] = c; This is a very common operation in R or Matlab. 4) Along the lines of #2, as an alternative to map, there is no support for array comparison operators. Something like int[3] B; B[] = A[] + 5; works, but bool[3] B; B[] = A[] > 0; doesn’t (I’m also not sure why I can’t just write auto B[] = A[] + 5;, but that’s neither here nor there). Moreover, it seems like only the mathematical operators work in this way. Mathematical functions from std.math, like exp, don’t seem to work. You have to use map (or a loop) with exp to get the result. I don’t have an issue with map, per se, but it seems inconsistent when some things work but not others. 5) You can only assign scalars to slices of arrays. There doesn’t seem to be an ability to assign an array to a slice. For instance, in #1, I couldn’t write A[0..1] = B; or A[0, 1] = B; instead of what I had written for C. 6) std.range and std.algorithm seem to have much better support for one dimensional containers than if you want to treat a container as two-dimensional. If you have a two-dimensional array and want to use map on every element, then there’s no issue. However, if you want to apply a function to each column or row, then you’d have to use a for loop (not even foreach). This seems to be a more difficult problem to solve than the others. I’m not sure what the best approach is, but it makes sense to look at other languages/libraries. In R, you have apply, which can operate on any dimensional array. Matlab has arrayfun. Numpy has apply_along_axis. Armadillo has .each_col and .each_row (one other thing about Armadillo is that you can switch between what underlying matrix math library is being used, like OpenBlas vs. Intel MKL).
Jun 11 2015
next sibling parent reply "Wyatt" <wyatt.epp gmail.com> writes:
On Thursday, 11 June 2015 at 21:30:22 UTC, jmh530 wrote:
 Most of what I discuss below is just syntactical sugar for some 
 stuff that could be accomplished with loops or std.algorithm,
Your post reminds me of two things I've considered attempting in the past: 1) a set of operators that have no meaning unless an overload is specifically provided (for dot product, dyadic transpose, etc.) and 2) a library implementing features of array-oriented languages to the extent it's possible (APL functions, rank awareness, trivial reshaping, aggregate lifting, et al). Syntax sugar can be important. -Wyatt
Jun 11 2015
parent reply "jmh530" <john.michael.hall gmail.com> writes:
On Thursday, 11 June 2015 at 22:36:28 UTC, Wyatt wrote:

 1) a set of operators that have no meaning unless an overload 
 is specifically provided (for dot product, dyadic transpose, 
 etc.) and
I see your point, but I think it might be a bit risky if you allow too much freedom for overloading operators. For instance, what if two people implement separate packages for matrix multiplication, one adopts the syntax of R (%*%) and one adopts the new Python syntax ( ). It may lead to some confusion.
Jun 11 2015
parent reply "Wyatt" <wyatt.epp gmail.com> writes:
On Friday, 12 June 2015 at 00:11:16 UTC, jmh530 wrote:
 On Thursday, 11 June 2015 at 22:36:28 UTC, Wyatt wrote:

 1) a set of operators that have no meaning unless an overload 
 is specifically provided (for dot product, dyadic transpose, 
 etc.) and
I see your point, but I think it might be a bit risky if you allow too much freedom for overloading operators. For instance, what if two people implement separate packages for matrix multiplication, one adopts the syntax of R (%*%) and one adopts the new Python syntax ( ). It may lead to some confusion.
From the outset, my thought was to strictly define the set of (eight or so?) symbols for this. If memory serves, it was right around the time Walter's rejected wholesale user-defined operators because of exactly the problem you mention. (Compounded by Unicode-- what the hell is "2 🐵 8" supposed to be!?) I strongly suspect you don't need many simultaneous extra operators on a type to cover most cases. -Wyatt
Jun 11 2015
next sibling parent reply "Tofu Ninja" <emmons0 purdue.edu> writes:
On Friday, 12 June 2015 at 01:55:15 UTC, Wyatt wrote:
 From the outset, my thought was to strictly define the set of 
 (eight or so?) symbols for this.  If memory serves, it was 
 right around the time Walter's rejected wholesale user-defined 
 operators because of exactly the problem you mention. 
 (Compounded by Unicode-- what the hell is "2 🐵 8" supposed to 
 be!?)  I strongly suspect you don't need many simultaneous 
 extra operators on a type to cover most cases.

 -Wyatt
What would the new order of operations be for these new operators?
Jun 11 2015
parent "Wyatt" <wyatt.epp gmail.com> writes:
On Friday, 12 June 2015 at 03:18:31 UTC, Tofu Ninja wrote:
 What would the new order of operations be for these new 
 operators?
Hadn't honestly thought that far. Like I said, it was more of a nascent idea than a coherent proposal (probably with a DIP and many more words). It's an interesting question, though. I think the approach taken by F# and OCaml may hit at the right notes, though: precedence and fixity are determined by the base operator. In my head, extra operators would be represented in code by some annotation or affix on a built-in operator... say, braces around it or something (e.g. [*] or {+}, though this is just an example that sets a baseline for visibility). -Wyatt
Jun 12 2015
prev sibling parent reply "Tofu Ninja" <emmons0 purdue.edu> writes:
On Friday, 12 June 2015 at 01:55:15 UTC, Wyatt wrote:
 From the outset, my thought was to strictly define the set of 
 (eight or so?) symbols for this.  If memory serves, it was 
 right around the time Walter's rejected wholesale user-defined 
 operators because of exactly the problem you mention. 
 (Compounded by Unicode-- what the hell is "2 🐵 8" supposed to 
 be!?)  I strongly suspect you don't need many simultaneous 
 extra operators on a type to cover most cases.

 -Wyatt
I actually thought about it more, and D does have a bunch of binary operators that no ones uses. You can make all sorts of weird operators like +*, *~, +++, ---, *--, /++, ~~, ~-, -~,
--, &++, ^^+, in++, |-, %~, ect...
void main(string[] args){ test a; test b; a +* b; } struct test{ private struct testAlpha{ test payload; } testAlpha opUnary(string s : "*")(){ return testAlpha(this); } void opBinary(string op : "+")(test rhs){ writeln("+"); } void opBinary(string op : "+")(testAlpha rhs){ writeln("+*"); } }
Jun 17 2015
next sibling parent reply "Dominikus Dittes Scherkl" writes:
On Wednesday, 17 June 2015 at 09:28:00 UTC, Tofu Ninja wrote:
 I actually thought about it more, and D does have a bunch of 
 binary operators that no ones uses. You can make all sorts of 
 weird operators like +*, *~, +++, ---, *--, /++, ~~, ~-, -~,
--, &++, ^^+, in++, |-, %~, ect...
+* is a specially bad idea, as I would read that as "a + (*b)", which is quite usual in C. But in general very cool. I love ~~ and |- the most :-)
Jun 23 2015
parent "Tofu Ninja" <emmons0 purdue.edu> writes:
On Tuesday, 23 June 2015 at 16:33:29 UTC, Dominikus Dittes 
Scherkl wrote:
 On Wednesday, 17 June 2015 at 09:28:00 UTC, Tofu Ninja wrote:
 I actually thought about it more, and D does have a bunch of 
 binary operators that no ones uses. You can make all sorts of 
 weird operators like +*, *~, +++, ---, *--, /++, ~~, ~-, -~,
--, &++, ^^+, in++, |-, %~, ect...
+* is a specially bad idea, as I would read that as "a + (*b)", which is quite usual in C. But in general very cool. I love ~~ and |- the most :-)
Yeah |- does seem like an interesting one, not sure what it would mean though, I get the impression it's a wall or something. Also you can basicly combine any binOp and any number of unaryOps to create an arbitrary number of custom binOps. ~+*+*+*+ could be valid! You could probably make something like brainfuck in D's unary operators.
Jun 23 2015
prev sibling parent reply "Wyatt" <wyatt.epp gmail.com> writes:
On Wednesday, 17 June 2015 at 09:28:00 UTC, Tofu Ninja wrote:
 I actually thought about it more, and D does have a bunch of 
 binary operators that no ones uses. You can make all sorts of 
 weird operators like +*, *~, +++, ---, *--, /++, ~~, ~-, -~,
--, &++, ^^+, in++, |-, %~, ect...
void main(string[] args){ test a; test b; a +* b; } struct test{ private struct testAlpha{ test payload; } testAlpha opUnary(string s : "*")(){ return testAlpha(this); } void opBinary(string op : "+")(test rhs){ writeln("+"); } void opBinary(string op : "+")(testAlpha rhs){ writeln("+*"); } }
Oh right, meant to respond to this. I'll admit it took me a few to really get why that works-- it's fairly clever and moderately terrifying. (I showed it to a friend and he opined it may violate the grammar.) But playing with it a bit...well, it's very cumbersome having to do these overload gymnastics. It eats away at your opUnary space because of the need for private proxy types, and each one needs an opBinary defined to support it explicitly. It also means you can't make overloads for mismatched types or builtin types (at least, I couldn't figure out how in the few minutes I spent poking it over lunch). -Wyatt
Jun 24 2015
parent reply "Tofu Ninja" <emmons0 purdue.edu> writes:
On Wednesday, 24 June 2015 at 19:04:38 UTC, Wyatt wrote:
 On Wednesday, 17 June 2015 at 09:28:00 UTC, Tofu Ninja wrote:
 I actually thought about it more, and D does have a bunch of 
 binary operators that no ones uses. You can make all sorts of 
 weird operators like +*, *~, +++, ---, *--, /++, ~~, ~-, -~,
--, &++, ^^+, in++, |-, %~, ect...
void main(string[] args){ test a; test b; a +* b; } struct test{ private struct testAlpha{ test payload; } testAlpha opUnary(string s : "*")(){ return testAlpha(this); } void opBinary(string op : "+")(test rhs){ writeln("+"); } void opBinary(string op : "+")(testAlpha rhs){ writeln("+*"); } }
Oh right, meant to respond to this. I'll admit it took me a few to really get why that works-- it's fairly clever and moderately terrifying. (I showed it to a friend and he opined it may violate the grammar.) But playing with it a bit...well, it's very cumbersome having to do these overload gymnastics. It eats away at your opUnary space because of the need for private proxy types, and each one needs an opBinary defined to support it explicitly. It also means you can't make overloads for mismatched types or builtin types (at least, I couldn't figure out how in the few minutes I spent poking it over lunch). -Wyatt
I am thinking of writing a mixin that will set up the proxy for you so that you can just write. struct test { mixin binOpProxy("*"); void opBinary(string op : "+*", T)(T rhs){ writeln("+*"); } } The hard part will be to get it to work with arbitrarily long unary proxies. Eg: mixin binOpProxy("~-~"); void opBinary(string op : "+~-~", T)(T rhs){ writeln("+~-~"); }
Jun 24 2015
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 06/24/2015 11:41 PM, Tofu Ninja wrote:
 On Wednesday, 24 June 2015 at 19:04:38 UTC, Wyatt wrote:
 On Wednesday, 17 June 2015 at 09:28:00 UTC, Tofu Ninja wrote:
 I actually thought about it more, and D does have a bunch of binary
 operators that no ones uses. You can make all sorts of weird
 operators like +*, *~, +++, ---, *--, /++, ~~, ~-, -~,
--, &++, ^^+, in++, |-, %~, ect...
void main(string[] args){ test a; test b; a +* b; } struct test{ private struct testAlpha{ test payload; } testAlpha opUnary(string s : "*")(){ return testAlpha(this); } void opBinary(string op : "+")(test rhs){ writeln("+"); } void opBinary(string op : "+")(testAlpha rhs){ writeln("+*"); } }
Oh right, meant to respond to this. I'll admit it took me a few to really get why that works-- it's fairly clever and moderately terrifying. (I showed it to a friend and he opined it may violate the grammar.) But playing with it a bit...well, it's very cumbersome having to do these overload gymnastics. It eats away at your opUnary space because of the need for private proxy types, and each one needs an opBinary defined to support it explicitly. It also means you can't make overloads for mismatched types or builtin types (at least, I couldn't figure out how in the few minutes I spent poking it over lunch). -Wyatt
I am thinking of writing a mixin that will set up the proxy for you so that you can just write. struct test { mixin binOpProxy("*"); void opBinary(string op : "+*", T)(T rhs){ writeln("+*"); } } The hard part will be to get it to work with arbitrarily long unary proxies. Eg: mixin binOpProxy("~-~"); void opBinary(string op : "+~-~", T)(T rhs){ writeln("+~-~"); }
Obviously you will run into issues with precedence soon, but this should do it: import std.stdio; struct Test{ mixin(binOpProxy("+~+-~*--+++----*")); void opBinary(string op : "+~+-~*--+++----*", T)(T rhs){ writeln("+~+-~*--+++----*"); } } void main(){ Test a,b; a +~+-~*--+++----* b; } import std.string, std.algorithm, std.range; int operatorSuffixLength(string s){ int count(dchar c){ return 2-s.retro.countUntil!(d=>c!=d)%2; } if(s.endsWith("++")) return count('+'); if(s.endsWith("--")) return count('-'); return 1; } struct TheProxy(T,string s){ T unwrap; this(T unwrap){ this.unwrap=unwrap; } static if(s.length){ alias NextType=TheProxy!(T,s[0..$-operatorSuffixLength(s)]); alias FullType=NextType.FullType; mixin(` auto opUnary(string op : "`~s[$-operatorSuffixLength(s)..$]~`")(){ return NextType(unwrap); }`); }else{ alias FullType=typeof(this); } } string binOpProxy(string s)in{ assert(s.length>=1+operatorSuffixLength(s)); assert(!s.startsWith("++")); assert(!s.startsWith("--")); foreach(dchar c;s) assert("+-*~".canFind(c)); }body{ int len=operatorSuffixLength(s); return ` auto opUnary(string op:"`~s[$-len..$]~`")(){ return TheProxy!(typeof(this),"`~s[1..$-len]~`")(this); } auto opBinary(string op:"`~s[0]~`")(TheProxy!(typeof(this),"`~s[1..$-1]~`").FullType t){ return opBinary!"`~s~`"(t.unwrap); } `; }
Jun 24 2015
parent "Tofu Ninja" <emmons0 purdue.edu> writes:
On Thursday, 25 June 2015 at 01:32:22 UTC, Timon Gehr wrote:
 [...]
Heres what I came up with... I love D so much <3 module util.binOpProxy; import std.algorithm : joiner, map; import std.array : array; struct __typeproxy(T, string s) { enum op = s; T payload; auto opUnary(string newop)() { return __typeproxy!(T,newop~op)(payload); } } /** * Example: * struct test * { * mixin(binOpProxy!("~", "*")); * * void opBinary(string op : "+~~", T)(T rhs) * { * writeln("hello!"); * } * * void opBinary(string op : "+~+-~*--+++----*", T)(T rhs) * { * writeln("world"); * } * * void opBinary(string op, T)(T rhs) * { * writeln("default"); * } * } * */ enum binOpProxy(proxies ...) = ` import ` ~ __MODULE__ ~ ` : __typeproxy; auto opBinary(string op, D : __typeproxy!(T, T_op), T, string T_op) (D rhs) { return opBinary!(op~D.op)(rhs.payload); } ` ~ [proxies].map!((string a) => ` auto opUnary(string op : "` ~ a ~ `")() { return __typeproxy!(typeof(this),op)(this); } `).joiner.array;
Jun 25 2015
prev sibling parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 12/06/2015 9:30 a.m., jmh530 wrote:
 On Tuesday, 9 June 2015 at 03:26:25 UTC, Ilya Yaroshenko wrote:

 There are
 https://github.com/9il/simple_matrix and
 https://github.com/9il/cblas .
 I will try to rework them for Phobos.

 Any ideas and suggestions?
A well-supported matrix math library would definitely lead to me using D more. I would definitely applaud any work being done on this subject, but I still feel there are some enhancements (most seemingly minor) that would really make a matrix math library easy/fun to use. Most of what I discuss below is just syntactical sugar for some stuff that could be accomplished with loops or std.algorithm, but having it built-in would make practical use of a matrix math library much easier. I think Armadillo implements some of these as member functions, whereas other languages like R and Matlab have them more built-in. Disclaimer: I don't consider myself a D expert, so I could be horribly wrong on some of this stuff. 1) There is no support for assignment to arrays based on the values of another array. int[] A = [-1, 1, 5]; int[] B = [1, 2]; int[] C = A[B]; You would have to use int[] C = A[1..2];. In this simple example, it’s not really a big deal, but if I have a function that returns B, then I can’t just throw B in there. I would have to loop through B and assign it to C. So the type of assignment is possible, but if you’re frequently doing this type of array manipulation, then the number of loops you need starts increasing. 2) Along the same lines, there is no support for replacing the B above with an array of bools like bool[] B = [false, true, true]; or auto B = A.map!(a => a < 0); Again, it is doable with a loop, but this form of logical indexing is a pretty common idiom for people who use Matlab or R quite a bit. 3) In addition to being able to index by a range of values or bools, you would want to be able to make assignments based on this. So something like A[B] = c; This is a very common operation in R or Matlab. 4) Along the lines of #2, as an alternative to map, there is no support for array comparison operators. Something like int[3] B; B[] = A[] + 5; works, but bool[3] B; B[] = A[] > 0; doesn’t (I’m also not sure why I can’t just write auto B[] = A[] + 5;, but that’s neither here nor there). Moreover, it seems like only the mathematical operators work in this way. Mathematical functions from std.math, like exp, don’t seem to work. You have to use map (or a loop) with exp to get the result. I don’t have an issue with map, per se, but it seems inconsistent when some things work but not others. 5) You can only assign scalars to slices of arrays. There doesn’t seem to be an ability to assign an array to a slice. For instance, in #1, I couldn’t write A[0..1] = B; or A[0, 1] = B; instead of what I had written for C. 6) std.range and std.algorithm seem to have much better support for one dimensional containers than if you want to treat a container as two-dimensional. If you have a two-dimensional array and want to use map on every element, then there’s no issue. However, if you want to apply a function to each column or row, then you’d have to use a for loop (not even foreach). This seems to be a more difficult problem to solve than the others. I’m not sure what the best approach is, but it makes sense to look at other languages/libraries. In R, you have apply, which can operate on any dimensional array. Matlab has arrayfun. Numpy has apply_along_axis. Armadillo has .each_col and .each_row (one other thing about Armadillo is that you can switch between what underlying matrix math library is being used, like OpenBlas vs. Intel MKL).
Humm, work on getting gl3n into phobos or work on my ODBC driver manager. Tough choice.
Jun 11 2015
parent reply "jmh530" <john.michael.hall gmail.com> writes:
On Friday, 12 June 2015 at 03:35:31 UTC, Rikki Cattermole wrote:

 Humm, work on getting gl3n into phobos or work on my ODBC 
 driver manager. Tough choice.
I can only speak for myself. I'm sure there's a lot of value in solid ODBC support. I use SQL some, but I use matrix math more. I'm not that familiar with gl3n, but it looks like it's meant for the math used in OpenGL. My knowledge of OpenGL is limited. I had some cursory interest in the developments of Vulkan earlier in March, but without much of a background in OpenGL I didn't follow everything they were talking about. I don't think many other languages include OpenGL support in their standard libraries (though I imagine game developers would welcome it).
Jun 12 2015
parent reply "Tofu Ninja" <emmons0 purdue.edu> writes:
On Friday, 12 June 2015 at 17:10:08 UTC, jmh530 wrote:
 On Friday, 12 June 2015 at 03:35:31 UTC, Rikki Cattermole wrote:

 Humm, work on getting gl3n into phobos or work on my ODBC 
 driver manager. Tough choice.
I can only speak for myself. I'm sure there's a lot of value in solid ODBC support. I use SQL some, but I use matrix math more. I'm not that familiar with gl3n, but it looks like it's meant for the math used in OpenGL. My knowledge of OpenGL is limited. I had some cursory interest in the developments of Vulkan earlier in March, but without much of a background in OpenGL I didn't follow everything they were talking about. I don't think many other languages include OpenGL support in their standard libraries (though I imagine game developers would welcome it).
Matrix math is matrix math, it being for ogl makes no real difference. Also if you are waiting to learn vulkan but have not done any other graphics, don't, learn ogl now, vulkan will be harder.
Jun 12 2015
next sibling parent reply "jmh530" <john.michael.hall gmail.com> writes:
On Friday, 12 June 2015 at 17:56:53 UTC, Tofu Ninja wrote:

 Matrix math is matrix math, it being for ogl makes no real 
 difference.
I think it’s a little more complicated than that. BLAS and LAPACK (or variants on them) are low-level matrix math libraries that many higher-level libraries call. Few people actually use BLAS directly. So, clearly, not every matrix math library is the same. What differentiates BLAS from Armadillo is that you can be far more productive in Armadillo because the syntax is friendly (and quite similar to Matlab and others). There’s a reason why people use glm in C++. It’s probably the most productive way to do matrix math with OpenGL. However, it may not be the most productive way to do more general matrix math. That’s why I hear about people using Armadillo, Eigen, and Blaze, but I’ve never heard anyone recommend using glm. Syntax matters.
Jun 12 2015
parent Rikki Cattermole <alphaglosined gmail.com> writes:
On 13/06/2015 7:45 a.m., jmh530 wrote:
 On Friday, 12 June 2015 at 17:56:53 UTC, Tofu Ninja wrote:

 Matrix math is matrix math, it being for ogl makes no real difference.
I think it’s a little more complicated than that. BLAS and LAPACK (or variants on them) are low-level matrix math libraries that many higher-level libraries call. Few people actually use BLAS directly. So, clearly, not every matrix math library is the same. What differentiates BLAS from Armadillo is that you can be far more productive in Armadillo because the syntax is friendly (and quite similar to Matlab and others). There’s a reason why people use glm in C++. It’s probably the most productive way to do matrix math with OpenGL. However, it may not be the most productive way to do more general matrix math. That’s why I hear about people using Armadillo, Eigen, and Blaze, but I’ve never heard anyone recommend using glm. Syntax matters.
The reason I am considering gl3n is because it is old solid code. It's proven itself. It'll make the review process relatively easy. But hey, if we want to do it right, we'll never get any implementation in.
Jun 12 2015
prev sibling parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Friday, 12 June 2015 at 17:56:53 UTC, Tofu Ninja wrote:
 On Friday, 12 June 2015 at 17:10:08 UTC, jmh530 wrote:
 On Friday, 12 June 2015 at 03:35:31 UTC, Rikki Cattermole 
 wrote:

 Humm, work on getting gl3n into phobos or work on my ODBC 
 driver manager. Tough choice.
I can only speak for myself. I'm sure there's a lot of value in solid ODBC support. I use SQL some, but I use matrix math more. I'm not that familiar with gl3n, but it looks like it's meant for the math used in OpenGL. My knowledge of OpenGL is limited. I had some cursory interest in the developments of Vulkan earlier in March, but without much of a background in OpenGL I didn't follow everything they were talking about. I don't think many other languages include OpenGL support in their standard libraries (though I imagine game developers would welcome it).
Matrix math is matrix math, it being for ogl makes no real difference.
The tiny subset of numerical linear algebra that is relevant for graphics (mostly very basic operations, 2,3 or 4 dimensions) is not at all representative of the whole. The algorithms are different and the APIs are often necessarily different. Even just considering scale, no one sane calls in to BLAS to multiply a 3*3 matrix by a 3 element vector, simultaneously no one sane *doesn't* call in to BLAS or an equivalent to multiply two 500*500 matrices.
Jun 13 2015
parent reply "Tofu Ninja" <emmons0 purdue.edu> writes:
On Saturday, 13 June 2015 at 08:45:20 UTC, John Colvin wrote:
 The tiny subset of numerical linear algebra that is relevant 
 for graphics (mostly very basic operations, 2,3 or 4 
 dimensions) is not at all representative of the whole. The 
 algorithms are different and the APIs are often necessarily 
 different.

 Even just considering scale, no one sane calls in to BLAS to 
 multiply a 3*3 matrix by a 3 element vector, simultaneously no 
 one sane *doesn't* call in to BLAS or an equivalent to multiply 
 two 500*500 matrices.
I think there is a conflict of interest with what people want. There seem to be people like me who only want or need simple matrices like glm to do basic geometric/graphics related stuff. Then there is the group of people who want large 500x500 matrices to do weird crazy maths stuff. Maybe they should be kept separate? In which case then we are really talking about adding two different things. Maybe have a std.math.matrix and a std.blas?
Jun 13 2015
next sibling parent reply Rikki Cattermole <alphaglosined gmail.com> writes:
On 13/06/2015 10:35 p.m., Tofu Ninja wrote:
 On Saturday, 13 June 2015 at 08:45:20 UTC, John Colvin wrote:
 The tiny subset of numerical linear algebra that is relevant for
 graphics (mostly very basic operations, 2,3 or 4 dimensions) is not at
 all representative of the whole. The algorithms are different and the
 APIs are often necessarily different.

 Even just considering scale, no one sane calls in to BLAS to multiply
 a 3*3 matrix by a 3 element vector, simultaneously no one sane
 *doesn't* call in to BLAS or an equivalent to multiply two 500*500
 matrices.
I think there is a conflict of interest with what people want. There seem to be people like me who only want or need simple matrices like glm to do basic geometric/graphics related stuff. Then there is the group of people who want large 500x500 matrices to do weird crazy maths stuff. Maybe they should be kept separate? In which case then we are really talking about adding two different things. Maybe have a std.math.matrix and a std.blas?
IMO simple matrix is fine for a standard library. More complex highly specialized math library yeah no. Not enough gain for such a complex code. Where as matrix/vector support for e.g. OpenGL now that will have a high visibility to game devs.
Jun 13 2015
parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Saturday, 13 June 2015 at 10:37:39 UTC, Rikki Cattermole wrote:
 On 13/06/2015 10:35 p.m., Tofu Ninja wrote:
 On Saturday, 13 June 2015 at 08:45:20 UTC, John Colvin wrote:
 [...]
I think there is a conflict of interest with what people want. There seem to be people like me who only want or need simple matrices like glm to do basic geometric/graphics related stuff. Then there is the group of people who want large 500x500 matrices to do weird crazy maths stuff. Maybe they should be kept separate? In which case then we are really talking about adding two different things. Maybe have a std.math.matrix and a std.blas?
IMO simple matrix is fine for a standard library. More complex highly specialized math library yeah no. Not enough gain for such a complex code. Where as matrix/vector support for e.g. OpenGL now that will have a high visibility to game devs.
Linear algebra for graphics is the specialised case, not the other way around. As a possible name for something like gl3n in phobos, I like std.math.geometry
Jun 13 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Saturday, 13 June 2015 at 11:05:19 UTC, John Colvin wrote:
 Linear algebra for graphics is the specialised case, not the 
 other way around. As a possible name for something like gl3n in 
 phobos, I like std.math.geometry
A geometry library is different, it should be type safe when it comes to units, lengths, distances, areas... I think linear algebra should have the same syntax for small and large matrices and switch representation behind the scenes. The challenge is to figure out what kind of memory layouts you need to support in order to interact with existing frameworks/hardware with no conversion.
Jun 13 2015
parent reply "jmh530" <john.michael.hall gmail.com> writes:
On Saturday, 13 June 2015 at 11:18:54 UTC, Ola Fosheim Grøstad 
wrote:
 I think linear algebra should have the same syntax for small 
 and large matrices and switch representation behind the scenes.
Switching representations behind the scenes? Sounds complicated. I would think that if you were designing it from the ground up, you would have one general matrix math library. Then a graphics library could be built on top of that functionality. That way, as improvements are made to the matrix math functionality, the graphics library would benefit too. However, given that there already is a well developed math graphics library, I'm not sure what's optimal. I can see the argument for implementing gl3n in the standard library (as a specialized math graphics option) on its own if there is demand for it.
Jun 13 2015
parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Sunday, 14 June 2015 at 02:56:04 UTC, jmh530 wrote:
 On Saturday, 13 June 2015 at 11:18:54 UTC, Ola Fosheim Grøstad 
 wrote:
 I think linear algebra should have the same syntax for small 
 and large matrices and switch representation behind the scenes.
Switching representations behind the scenes? Sounds complicated.
You don't have much of a choice if you want it to perform. You have take take into consideration: 1. hardware factors such as SIMD and alignment 2. what is known at compile time and what is only known at runtime 3. common usage patterns (what elements are usually 0, 1 or a value) 4. when does it pay off to encode the matrix modifications and layout as meta information (like transpose and scalar multiplication or addition) And sometimes you might want to compute the inverse matrix when doing the transforms, rather than as a separate step for performance reasons.
 I would think that if you were designing it from the ground up, 
 you would have one general matrix math library. Then a graphics 
 library could be built on top of that functionality. That way, 
 as improvements are made to the matrix math functionality, the 
 graphics library would benefit too.
Yes, but nobody wants to use a matrix library that does not perform close to the hardware limitations, so the representation should be special cased to fit the hardware for common matrix layouts.
Jun 14 2015
prev sibling next sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Saturday, 13 June 2015 at 10:35:55 UTC, Tofu Ninja wrote:
 On Saturday, 13 June 2015 at 08:45:20 UTC, John Colvin wrote:
 The tiny subset of numerical linear algebra that is relevant 
 for graphics (mostly very basic operations, 2,3 or 4 
 dimensions) is not at all representative of the whole. The 
 algorithms are different and the APIs are often necessarily 
 different.

 Even just considering scale, no one sane calls in to BLAS to 
 multiply a 3*3 matrix by a 3 element vector, simultaneously no 
 one sane *doesn't* call in to BLAS or an equivalent to 
 multiply two 500*500 matrices.
I think there is a conflict of interest with what people want. There seem to be people like me who only want or need simple matrices like glm to do basic geometric/graphics related stuff. Then there is the group of people who want large 500x500 matrices to do weird crazy maths stuff. Maybe they should be kept separate? In which case then we are really talking about adding two different things. Maybe have a std.math.matrix and a std.blas?
Yes, that's what I was trying to point out. Anyway, gl3n or similar would be great to have in phobos, I've used it quite a bit and think it's great, but it should be very clear that it's not a general purpose matrix/linear algebra toolkit. It's a specialised set of types and operations specifically for low-dimensional geometry, with an emphasis on common graphics idioms.
Jun 13 2015
prev sibling next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 06/13/2015 12:35 PM, Tofu Ninja wrote:
 On Saturday, 13 June 2015 at 08:45:20 UTC, John Colvin wrote:
 The tiny subset of numerical linear algebra that is relevant for
 graphics (mostly very basic operations, 2,3 or 4 dimensions) is not at
 all representative of the whole. The algorithms are different and the
 APIs are often necessarily different.

 Even just considering scale, no one sane calls in to BLAS to multiply
 a 3*3 matrix by a 3 element vector, simultaneously no one sane
 *doesn't* call in to BLAS or an equivalent to multiply two 500*500
 matrices.
I think there is a conflict of interest with what people want. There seem to be people like me who only want or need simple matrices like glm to do basic geometric/graphics related stuff. Then there is the group of people who want large 500x500 matrices to do weird crazy maths stuff.
(It's neither weird nor crazy.)
 Maybe they should be kept separate?
I think there's no point to that. Just have dynamically sized and fixed sized versions. Why should they be incompatible? It's the same concept.
Jun 13 2015
prev sibling parent reply "weaselcat" <weaselcat gmail.com> writes:
On Saturday, 13 June 2015 at 10:35:55 UTC, Tofu Ninja wrote:
 On Saturday, 13 June 2015 at 08:45:20 UTC, John Colvin wrote:
 The tiny subset of numerical linear algebra that is relevant 
 for graphics (mostly very basic operations, 2,3 or 4 
 dimensions) is not at all representative of the whole. The 
 algorithms are different and the APIs are often necessarily 
 different.

 Even just considering scale, no one sane calls in to BLAS to 
 multiply a 3*3 matrix by a 3 element vector, simultaneously no 
 one sane *doesn't* call in to BLAS or an equivalent to 
 multiply two 500*500 matrices.
I think there is a conflict of interest with what people want. There seem to be people like me who only want or need simple matrices like glm to do basic geometric/graphics related stuff. Then there is the group of people who want large 500x500 matrices to do weird crazy maths stuff. Maybe they should be kept separate? In which case then we are really talking about adding two different things. Maybe have a std.math.matrix and a std.blas?
+1 nobody uses general purpose linear matrix libraries for games/graphics for a reason, many game math libraries take shortcuts everywhere and are extensively optimized(e.g, for cache lines) for the general purpose vec3/mat4 types. many performance benefits for massive matrices see performance detriments for tiny graphics-oriented matrices. This is just shoehorning, plain and simple.
Jun 14 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Sunday, 14 June 2015 at 08:14:21 UTC, weaselcat wrote:
 nobody uses general purpose linear matrix libraries for 
 games/graphics for a reason,
The reason is that C++ didn't provide anything. As a result each framework provide their own and you get N different libraries that are incompatible. There is no good reason for making small-matrix libraries incompatible with the rest of eco-system given the templating system you have in D. What you need is a library that supports multiple representations and can do the conversions. Of course, you'll do better if you also have term-rewriting/AST-macros.
Jun 14 2015
parent reply "Ilya Yaroshenko" <ilyayaroshenko gmail.com> writes:
On Sunday, 14 June 2015 at 09:07:19 UTC, Ola Fosheim Grøstad 
wrote:
 On Sunday, 14 June 2015 at 08:14:21 UTC, weaselcat wrote:
 nobody uses general purpose linear matrix libraries for 
 games/graphics for a reason,
The reason is that C++ didn't provide anything. As a result each framework provide their own and you get N different libraries that are incompatible. There is no good reason for making small-matrix libraries incompatible with the rest of eco-system given the templating system you have in D. What you need is a library that supports multiple representations and can do the conversions. Of course, you'll do better if you also have term-rewriting/AST-macros.
The reason is general purpose matrixes allocated at heap, but small graphic matrices are plain structs. `opCast(T)` should be enough.
Jun 14 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Sunday, 14 June 2015 at 09:19:19 UTC, Ilya Yaroshenko wrote:
 The reason is general purpose matrixes allocated at heap, but 
 small graphic matrices are plain structs.
No, the reason is that LA-libraries are C-libraries that also deal with variable sized matrices. A good generic API can support both. You cannot create a good generic API in C. You can in D.
Jun 14 2015
parent reply "Ilya Yaroshenko" <ilyayaroshenko gmail.com> writes:
On Sunday, 14 June 2015 at 09:25:25 UTC, Ola Fosheim Grøstad 
wrote:
 On Sunday, 14 June 2015 at 09:19:19 UTC, Ilya Yaroshenko wrote:
 The reason is general purpose matrixes allocated at heap, but 
 small graphic matrices are plain structs.
No, the reason is that LA-libraries are C-libraries that also deal with variable sized matrices. A good generic API can support both. You cannot create a good generic API in C. You can in D.
We need D own BLAS implementation to do it. Sight, DBLAS will be largest part of std.
Jun 14 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Sunday, 14 June 2015 at 09:59:22 UTC, Ilya Yaroshenko wrote:
 We need D own BLAS implementation to do it.
Why can't you use "version" for those that want to use a BLAS library for the implementation? Those who want replications of LAPACK/LINPACK APIs can use separate bindings? And those who want to use BLAS directly would not use phobos anyway, but a direct binding so they can switch implementation? I think a good generic higher level linear algebra library for D should aim to be equally useful for 2D Graphics, 3D/4D GPU graphics, CAD solid modelling, robotics, 3D raytracing, higher dimensional fractals, physics sims, image processing, signal processing, scientific computing (which is pretty wide) and more. The Phobos API should be user-level, not library-level like BLAS. IMO. You really want an API that look like this in Phobos? http://www.netlib.org/blas/ BLAS/LAPACK/LINPACK all originate in Fortran with a particular scientific tradition in mind, so I think one should rethink how D goes about this. Fortran has very primitive abstraction mechanisms. This stuff is stuck in the 80s…
Jun 14 2015
next sibling parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
I think there might be a disconnection in this thread. D only, or 
D frontend?

There are hardware vendor and commercial libraries that are 
heavily optimized for particular hardware configurations. There 
is no way a D-only solution can beat those. As an example Apple 
provides various implementations for their own machines, so an 
old program on a new machine can run faster than a static D-only 
library solution.

What D can provide is a unifying abstraction, but to get there 
one need to analyze what exists. Like Apple's Accelerate 
framework:

https://developer.apple.com/library/prerelease/ios/documentation/Accelerate/Reference/AccelerateFWRef/index.html#//apple_ref/doc/uid/TP40009465

That goes beyond BLAS. We also need to look at vDSP etc. You'll 
find similar things for Microsoft/Intel/AMD/ARM etc…
Jun 14 2015
parent "Ilya Yaroshenko" <ilyayaroshenko gmail.com> writes:
On Sunday, 14 June 2015 at 10:43:24 UTC, Ola Fosheim Grøstad 
wrote:
 I think there might be a disconnection in this thread. D only, 
 or D frontend?

 There are hardware vendor and commercial libraries that are 
 heavily optimized for particular hardware configurations. There 
 is no way a D-only solution can beat those. As an example Apple 
 provides various implementations for their own machines, so an 
 old program on a new machine can run faster than a static 
 D-only library solution.

 What D can provide is a unifying abstraction, but to get there 
 one need to analyze what exists. Like Apple's Accelerate 
 framework:

 https://developer.apple.com/library/prerelease/ios/documentation/Accelerate/Reference/AccelerateFWRef/index.html#//apple_ref/doc/uid/TP40009465

 That goes beyond BLAS. We also need to look at vDSP etc. You'll 
 find similar things for Microsoft/Intel/AMD/ARM etc…
+1
Jun 14 2015
prev sibling parent reply "Ilya Yaroshenko" <ilyayaroshenko gmail.com> writes:
On Sunday, 14 June 2015 at 10:15:08 UTC, Ola Fosheim Grøstad 
wrote:
 On Sunday, 14 June 2015 at 09:59:22 UTC, Ilya Yaroshenko wrote:
 We need D own BLAS implementation to do it.
Why can't you use "version" for those that want to use a BLAS library for the implementation? Those who want replications of LAPACK/LINPACK APIs can use separate bindings? And those who want to use BLAS directly would not use phobos anyway, but a direct binding so they can switch implementation? I think a good generic higher level linear algebra library for D should aim to be equally useful for 2D Graphics, 3D/4D GPU graphics, CAD solid modelling, robotics, 3D raytracing, higher dimensional fractals, physics sims, image processing, signal processing, scientific computing (which is pretty wide) and more. The Phobos API should be user-level, not library-level like BLAS. IMO. You really want an API that look like this in Phobos? http://www.netlib.org/blas/ BLAS/LAPACK/LINPACK all originate in Fortran with a particular scientific tradition in mind, so I think one should rethink how D goes about this. Fortran has very primitive abstraction mechanisms. This stuff is stuck in the 80s…
I am really don't understand what you mean with "generic" keyword. Do you want one matrix type that includes all cases??? I hope you does not. If not, yes it should be generic like all other Phobos. But we will have one module for 3D/4D geometric and 3D/4D matrix/vector multiplications, another module for general matrix (std.container.matrix) and another module with generic BLAS (std.numeric.blas) for general purpose matrixes. After all of that we can think about scripting like "m0 = m1*v*m2" features. I think LAPACK would not be implemented in Phobos, but we can use SciD instead.
Jun 14 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Sunday, 14 June 2015 at 11:43:46 UTC, Ilya Yaroshenko wrote:
 I am really don't understand what you mean with "generic" 
 keyword.

 Do you want one matrix type that includes all cases???
 I hope you does not.
Yes, that is what generic programming is about. The type should signify the semantics, not exact representation. Then you alias common types "float4x4" etc. It does take a lot of abstraction design work. I've done some of it in C++ for sliced views over memory and arrays and I'd say you need many iterations to get it right.
 If not, yes it should be generic like all other Phobos. But we 
 will have one module for 3D/4D geometric and 3D/4D 
 matrix/vector multiplications, another module for general 
 matrix (std.container.matrix) and another module with generic 
 BLAS (std.numeric.blas) for general purpose matrixes. After all 
 of that we can think about scripting like "m0 = m1*v*m2" 
 features.
All I can say is that I have a strong incentive to avoid using Phobos features if D does not automatically utilize the best OS/CPU vendor provided libraries in a portable manner and with easy-to-read high level abstractions. D's strength compared to C++/Rust is that D can evolve to be easier to use than those languages. C++/Rust are hard to use by nature. But usability takes a lot of API design effort, so it won't come easy. D's strength compared to Go is that it can better take advantage of hardware and provide better library abstractions, Go appears to deliberately avoid it. They probably want to stay nimble with very limited hardware-interfacing so that you can easily move it around in the cloud.
Jun 14 2015
parent reply "Ilya Yaroshenko" <ilyayaroshenko gmail.com> writes:
On Sunday, 14 June 2015 at 12:01:47 UTC, Ola Fosheim Grøstad 
wrote:
 On Sunday, 14 June 2015 at 11:43:46 UTC, Ilya Yaroshenko wrote:
 I am really don't understand what you mean with "generic" 
 keyword.

 Do you want one matrix type that includes all cases???
 I hope you does not.
Yes, that is what generic programming is about. The type should signify the semantics, not exact representation. Then you alias common types "float4x4" etc.
std.range has a lot of types + D arrays. The power in unified API (structural type system). For matrixes this API is very simple: operations like m1[] += m2, transposed, etc. Ilya
Jun 14 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Sunday, 14 June 2015 at 12:18:39 UTC, Ilya Yaroshenko wrote:
 std.range has a lot of types + D arrays.
 The power in unified API (structural type system).
Yeah, I agree that templates in C++/D more or less makes those type systems structural-like, even though C is using nominal typing. I've also found that although the combinatorial explosion is a possibility, most applications I write have a "types.h" file that define the subset I want to use for that application. So the combinatorial explosion is not such a big deal after all. But one need to be patient and add lots of static_asserts… since the template type system is weak.
 For matrixes this API is very simple: operations like m1[] += 
 m2, transposed, etc.
I think it is a bit more complicated than that. You also need to think about alignment, padding, strides, convolutions, identiy matrices, invertible matrices, windows on a stream, higher order matrices etc…
Jun 14 2015
parent reply "Ilya Yaroshenko" <ilyayaroshenko gmail.com> writes:
On Sunday, 14 June 2015 at 12:52:52 UTC, Ola Fosheim Grøstad 
wrote:
 On Sunday, 14 June 2015 at 12:18:39 UTC, Ilya Yaroshenko wrote:
 std.range has a lot of types + D arrays.
 The power in unified API (structural type system).
Yeah, I agree that templates in C++/D more or less makes those type systems structural-like, even though C is using nominal typing. I've also found that although the combinatorial explosion is a possibility, most applications I write have a "types.h" file that define the subset I want to use for that application. So the combinatorial explosion is not such a big deal after all. But one need to be patient and add lots of static_asserts… since the template type system is weak.
 For matrixes this API is very simple: operations like m1[] += 
 m2, transposed, etc.
I think it is a bit more complicated than that. You also need to think about alignment, padding, strides, convolutions, identiy matrices, invertible matrices, windows on a stream, higher order matrices etc…
Alignment, strides (windows on a stream - I understand it like Sliding Windows) are not a problem. Convolutions, identiy matrices, invertible matrices are stuff I don't want to see in Phobos. They are about "MathD" not about (big) standard library. For hight order slices see https://github.com/D-Programming-Language/phobos/pull/3397
Jun 14 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Sunday, 14 June 2015 at 13:48:23 UTC, Ilya Yaroshenko wrote:
 Alignment, strides (windows on a stream - I understand it like 
 Sliding Windows) are not a problem.
It isn't a problem if you use the best possible abstraction from the start. It is a problem if you don't focus on it from the start.
 Convolutions, identiy matrices, invertible matrices are stuff I 
 don't want to see in Phobos. They are about "MathD" not about 
 (big) standard library.
I don't see how you can get good performance without special casing identity matrices, transposed matrices and so on. You surely need to support matrix inversion, Gauss-Jordan elimination (or the equivalent) etc?
Jun 14 2015
parent reply "Ilya Yaroshenko" <ilyayaroshenko gmail.com> writes:
On Sunday, 14 June 2015 at 14:02:59 UTC, Ola Fosheim Grøstad 
wrote:
 On Sunday, 14 June 2015 at 13:48:23 UTC, Ilya Yaroshenko wrote:
 Alignment, strides (windows on a stream - I understand it like 
 Sliding Windows) are not a problem.
It isn't a problem if you use the best possible abstraction from the start. It is a problem if you don't focus on it from the start.
I am sorry for this trolling: Lisp is the best abstraction, thought. Sometimes I find very cool abstract libraries, with relatively small number of users. For example many programmers don't want to use Boost only because it's abstractions makes them crazy.
 Convolutions, identiy matrices, invertible matrices are stuff 
 I don't want to see in Phobos. They are about "MathD" not 
 about (big) standard library.
I don't see how you can get good performance without special casing identity matrices, transposed matrices and so on. You surely need to support matrix inversion, Gauss-Jordan elimination (or the equivalent) etc?
For daily scientific purposes - yes. For R/Matlab like mathematical library - yes. For real world application - no. Engineer can achieve best performance without special cases by lowering "abstraction" down. Simplicity and transparency ("how it works") is more important in this case.
Jun 14 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Sunday, 14 June 2015 at 14:25:11 UTC, Ilya Yaroshenko wrote:
 I am sorry for this trolling:
 Lisp is the best abstraction, thought.
Even it if was, it does not provide the meta info and alignment type constraints that makes it possible to hardware/SIMD optimize it behind the scenes.
 For example many programmers don't want to use Boost only 
 because it's abstractions makes them crazy.
Yes, C++ templates are a hard nut to crack, if D had added excellent pattern matching to its meta programming repertoire the I think this would be enough to put D in a different league. Application programmers should not have to deal with lots of type parameters, they can use the simplified version (aliases). That's what I do in my C++ libs, using templated aliasing to make a complicated type composition easy to use while still getting the benefits generic pattern matching and generic programming.
 Convolutions, identiy matrices, invertible matrices are stuff
For daily scientific purposes - yes. For R/Matlab like mathematical library - yes. For real world application - no. Engineer can achieve best performance without special cases by lowering "abstraction" down. Simplicity and transparency ("how it works") is more important in this case.
Getting platform optimized versions of frequently used heavy operations is the primary reason for why I would use a builtin library over rolling my own. Especially if the compiler has builtin high-level optimizations for the algebra. A naive basic matrix library is simple to write, I don't need standard library support for that + I get it to work the way I want by using SIMD registers directly... => I probably would not use it if I could implement it in less than 10 hours.
Jun 14 2015
next sibling parent "weaselcat" <weaselcat gmail.com> writes:
On Sunday, 14 June 2015 at 14:46:36 UTC, Ola Fosheim Grøstad 
wrote:
 Yes, C++ templates are a hard nut to crack, if D had added 
 excellent pattern matching to its meta programming repertoire 
 the I think this would be enough to put D in a different league.
https://github.com/solodon4/Mach7
Jun 14 2015
prev sibling parent reply "Ilya Yaroshenko" <ilyayaroshenko gmail.com> writes:
 A naive basic matrix library is simple to write, I don't need 
 standard library support for that + I get it to work the way I 
 want by using SIMD registers directly... => I probably would 
 not use it if I could implement it in less than 10 hours.
A naive std.algorithm and std.range is easy to write too.
Jun 14 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Sunday, 14 June 2015 at 15:15:38 UTC, Ilya Yaroshenko wrote:
 A naive basic matrix library is simple to write, I don't need 
 standard library support for that + I get it to work the way I 
 want by using SIMD registers directly... => I probably would 
 not use it if I could implement it in less than 10 hours.
A naive std.algorithm and std.range is easy to write too.
I wouldn't know. People have different needs. Builtin for-each-loops, threads and SIMD support are more important to me than iterators (ranges). But the problem with linear algebra is that you might want to do SIMD optimized versions where you calculate 4 equations at the time, do reshuffling etc. So a library solution has to provide substantial benefits.
Jun 14 2015
parent reply "Ilya Yaroshenko" <ilyayaroshenko gmail.com> writes:
On Sunday, 14 June 2015 at 18:05:33 UTC, Ola Fosheim Grøstad 
wrote:
 On Sunday, 14 June 2015 at 15:15:38 UTC, Ilya Yaroshenko wrote:
 A naive basic matrix library is simple to write, I don't need 
 standard library support for that + I get it to work the way 
 I want by using SIMD registers directly... => I probably 
 would not use it if I could implement it in less than 10 
 hours.
A naive std.algorithm and std.range is easy to write too.
I wouldn't know. People have different needs. Builtin for-each-loops, threads and SIMD support are more important to me than iterators (ranges). But the problem with linear algebra is that you might want to do SIMD optimized versions where you calculate 4 equations at the time, do reshuffling etc. So a library solution has to provide substantial benefits.
Yes, but it would be hard to create SIMD optimised version. What do you think about this chain of steps? 1. Create generalised (only type template and my be flags) BLAS algorithms (probably slow) with CBLAS like API. 2. Allow users to use existing CBLAS libraries inside generalised BLAS. 3. Start to improve generalised BLAS with SIMD instructions. 4. And then continue discussion about type of matrixes we want...
Jun 14 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Sunday, 14 June 2015 at 18:49:21 UTC, Ilya Yaroshenko wrote:
 Yes, but it would be hard to create SIMD optimised version.
Then again clang is getting better at this stuff.
 What do you think about this chain of steps?

 1. Create generalised (only type template and my be flags) BLAS 
 algorithms (probably  slow) with CBLAS like API.
 2. Allow users to use existing CBLAS libraries inside 
 generalised BLAS.
 3. Start to improve generalised BLAS with SIMD instructions.
 4. And then continue discussion about type of matrixes we 
 want...
Hmm… I don't know. In general I think the best thing to do is to develop libraries with a project and then turn it into something more abstract. If I had more time I think I would have made the assumption that we could make LDC produce whatever next version of clang can do with pragmas/GCC-extensions and used that assumption for building some prototypes. So I would: 1. protoype typical constructs in C, compile it with next version of llvm/clang (with e.g. 4xloop-unrolling and try different optimization/vectorizing options) the look at the output in LLVM IR and assembly mnemonic code. 2. Then write similar code with hardware optimized BLAS and benchmark where the overhead between pure C/LLVM and BLAS calls balance out to even. Then you have a rough idea of what the limitations of the current infrastructure looks like, and can start modelling the template types in D? I'm not sure that you should use SIMD directly, but align the memory for it. Like, on iOS you end up using LLVM subsets because of the new bitcode requirements. Ditto for PNACL. Just a thought, but that's what I would I do.
Jun 14 2015
next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
Another thing worth noting is that I believe Intel has put some 
effort into next gen (?) LLVM/Clang for autovectorizing into 
AVX2. It might be worth looking into as it uses a mask that 
allows the CPU to skip computations that would lead to no change, 
but I think it is only available on last gen Intel CPUs.

Also worth keeping in mind is that future versions of LLVM will 
have to deal with GCC extensions and perhaps also Clang pragmas. 
So maybe take a look at:

http://clang.llvm.org/docs/LanguageExtensions.html#vectors-and-extended-vectors

and

http://clang.llvm.org/docs/LanguageExtensions.html#extensions-for-loop-hint-optimizations

?
Jun 14 2015
prev sibling parent reply "anonymous" <a b.cd> writes:
 1. Create generalised (only type template and my be flags) 
 BLAS algorithms (probably  slow) with CBLAS like API.
See [1] (the Matmul benchmark) Julia Native is probably backed with Intel MKL or OpenBLAS. D version was optimized by Martin Nowak [2] and is still _much_ slower.
 2. Allow users to use existing CBLAS libraries inside 
 generalised BLAS.
I think a good interface is more important than speed of default implementation (at least for e.g large matrix multiplication). Just use existing code for speed... Goto's papers about his BLAS: [3][4] Having something a competitive in D would be great but probably a lot of work. Without a good D interface dstep + openBLAS/Atlas header will not look that bad. Note I am not talking about small matrices/graphics.
 3. Start to improve generalised BLAS with SIMD instructions.
nice, but not really important. Good interface to existing high quality BLAS seems more important to me than fast D linear algebra implementation + CBLAS like interface.
 4. And then continue discussion about type of matrixes we 
 want...
+1
 2. Then write similar code with hardware optimized BLAS and 
 benchmark where the overhead between pure C/LLVM and BLAS calls 
 balance out to even.
may there are more important / beneficial things to work on - assuming total time of contributors is fix and used for other D stuff:) [1] https://github.com/kostya/benchmarks [2] https://github.com/kostya/benchmarks/pull/6 [3] http://www.cs.utexas.edu/users/flame/pubs/GotoTOMS2.pdf [4] http://www.cs.utexas.edu/users/pingali/CS378/2008sp/papers/gotoPaper.pdf
Jun 14 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Sunday, 14 June 2015 at 21:31:53 UTC, anonymous wrote:
 2. Then write similar code with hardware optimized BLAS and 
 benchmark where the overhead between pure C/LLVM and BLAS 
 calls balance out to even.
may there are more important / beneficial things to work on - assuming total time of contributors is fix and used for other D stuff:)
Sure, but that is what I'd do if I had the time. Get a baseline for what kind of NxN sizes D can reasonably be expected to deal with in a "naive brute force" manner. Then consider pushing anything beyond that over to something more specialized. *shrugs*
Jun 14 2015
next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Sunday, 14 June 2015 at 21:50:02 UTC, Ola Fosheim Grøstad 
wrote:
 Sure, but that is what I'd do if I had the time. Get a baseline 
 for what kind of NxN sizes D can reasonably be expected to deal 
 with in a "naive brute force" manner.
In case it isn't obvious: a potential advantage of a simple algorithm that do "naive brute force" is that the backend might stand a better chance optimizing it, at least when you have a matrix that is known at compile time.
Jun 15 2015
prev sibling parent reply "anonymous" <a b.de> writes:
On Sunday, 14 June 2015 at 21:50:02 UTC, Ola Fosheim Grøstad 
wrote:
 On Sunday, 14 June 2015 at 21:31:53 UTC, anonymous wrote:
 2. Then write similar code with hardware optimized BLAS and 
 benchmark where the overhead between pure C/LLVM and BLAS 
 calls balance out to even.
may there are more important / beneficial things to work on - assuming total time of contributors is fix and used for other D stuff:)
Sure, but that is what I'd do if I had the time. Get a baseline for what kind of NxN sizes D can reasonably be expected to deal with in a "naive brute force" manner. Then consider pushing anything beyond that over to something more specialized. *shrugs*
On Sunday, 14 June 2015 at 21:50:02 UTC, Ola Fosheim Grøstad wrote:
 On Sunday, 14 June 2015 at 21:31:53 UTC, anonymous wrote:
 2. Then write similar code with hardware optimized BLAS and 
 benchmark where the overhead between pure C/LLVM and BLAS 
 calls balance out to even.
may there are more important / beneficial things to work on - assuming total time of contributors is fix and used for other D stuff:)
Sure, but that is what I'd do if I had the time. Get a baseline for what kind of NxN sizes D can reasonably be expected to deal with in a "naive brute force" manner. Then consider pushing anything beyond that over to something more specialized. *shrugs*
sorry, I should read more careful. I understand 'optimize default implementation to the speed of high quality BLAS for _any_/large matrix size'. Great if it is done but imo there is no real pressure to do it and probably needs lot of time of experts. To benchmark when existing BLAS is actually faster is than 'naive brute force' sounds very good and reasonable.
Jun 15 2015
next sibling parent "Ilya Yaroshenko" <ilyayaroshenko gmail.com> writes:
On Monday, 15 June 2015 at 08:12:17 UTC, anonymous wrote:
 I understand 'optimize default implementation to the speed of 
 high quality BLAS for _any_/large matrix size'. Great if it is 
 done but imo there is no real pressure to do it and probably 
 needs lot of time of experts.
+1
Jun 15 2015
prev sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Monday, 15 June 2015 at 08:12:17 UTC, anonymous wrote:
 sorry, I should read more careful. I understand 'optimize 
 default implementation to the speed of high quality BLAS for 
 _any_/large matrix size'. Great if it is done but imo there is 
 no real pressure to do it and probably needs lot of time of 
 experts.

 To benchmark when existing BLAS is actually faster is than 
 'naive brute force' sounds very good and reasonable.
Yes. Well, I think there are some different expectations to what a standard library should include. In my view BLAS is primarily an API that matters because people have existing code bases, therefore it is common to have good implementations for it. I don't really see any reason for why new programs should target it. I think it is a good idea to stay higher level. Provide simple implementations that the optimizer can deal with. Then have a benchmarking program that run on different configurations (os+hardware) to measure when the non-D libraries perform better and use those when they are faster. So I don't think phobos should provide BLAS as such. That's what I would do, anyway.
Jun 15 2015
prev sibling next sibling parent reply "John Chapman" <johnch_atms hotmail.com> writes:
It's a shame ucent/cent never got implemented. But couldn't they 
be added to Phobos? I often need a 128-bit type with better 
precision than float and double.
Jun 10 2015
next sibling parent reply "ponce" <contact gam3sfrommars.fr> writes:
On Wednesday, 10 June 2015 at 07:56:46 UTC, John Chapman wrote:
 It's a shame ucent/cent never got implemented. But couldn't 
 they be added to Phobos? I often need a 128-bit type with 
 better precision than float and double.
FWIW: https://github.com/d-gamedev-team/gfm/blob/master/math/gfm/math/wideint.d
Jun 10 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 6/10/15 1:53 AM, ponce wrote:
 On Wednesday, 10 June 2015 at 07:56:46 UTC, John Chapman wrote:
 It's a shame ucent/cent never got implemented. But couldn't they be
 added to Phobos? I often need a 128-bit type with better precision
 than float and double.
FWIW: https://github.com/d-gamedev-team/gfm/blob/master/math/gfm/math/wideint.d
Yes, arbitrary fixed-size integrals would be good to have in Phobos. Who's the author of that code? Can we get something going here? -- Andrei
Jun 10 2015
parent "ponce" <contact gam3sfrommars.fr> writes:
On Wednesday, 10 June 2015 at 15:44:40 UTC, Andrei Alexandrescu 
wrote:
 On 6/10/15 1:53 AM, ponce wrote:
 On Wednesday, 10 June 2015 at 07:56:46 UTC, John Chapman wrote:
 It's a shame ucent/cent never got implemented. But couldn't 
 they be
 added to Phobos? I often need a 128-bit type with better 
 precision
 than float and double.
FWIW: https://github.com/d-gamedev-team/gfm/blob/master/math/gfm/math/wideint.d
Yes, arbitrary fixed-size integrals would be good to have in Phobos. Who's the author of that code? Can we get something going here? -- Andrei
Sorry for the delay. I wrote this code a while earlier. I will relicense it anyway that is needed (if needed). Currently lack the time to polish it more (adding custom literals would be the one thing to do).
Jun 23 2015
prev sibling next sibling parent reply "John Chapman" <johnch_atms hotmail.com> writes:
On Wednesday, 10 June 2015 at 07:56:46 UTC, John Chapman wrote:
 It's a shame ucent/cent never got implemented. But couldn't 
 they be added to Phobos? I often need a 128-bit type with 
 better precision than float and double.
Other things I often have a need for: Weak references Queues, stacks, sets Logging Custom date/time formatting Locale-aware number/currency formatting HMAC (for OAuth) URI parsing Sending email (SMTP) Continuations for std.parallelism.Task Database connectivity (sounds like this is on the cards) HTTP listener
Jun 10 2015
next sibling parent ketmar <ketmar ketmar.no-ip.org> writes:
On Wed, 10 Jun 2015 09:12:15 +0000, John Chapman wrote:

 On Wednesday, 10 June 2015 at 07:56:46 UTC, John Chapman wrote:
 It's a shame ucent/cent never got implemented. But couldn't they be
 added to Phobos? I often need a 128-bit type with better precision than
 float and double.
=20 Other things I often have a need for: =20 Weak references
+inf for including that into Phobos. current implementations are hacks=20 that may stop working when internals will change, but if it will be in=20 Phobos, it will be always up-to-date.=
Jun 10 2015
prev sibling next sibling parent reply "Robert burner Schadek" <rburners gmail.com> writes:
On Wednesday, 10 June 2015 at 09:12:17 UTC, John Chapman wrote:

 Logging
std.experimental.logger!?
Jun 10 2015
parent "John Chapman" <johnch_atms hotmail.com> writes:
On Wednesday, 10 June 2015 at 09:30:37 UTC, Robert burner Schadek 
wrote:
 On Wednesday, 10 June 2015 at 09:12:17 UTC, John Chapman wrote:

 Logging
std.experimental.logger!?
Perfect, he said sheepishly.
Jun 10 2015
prev sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Wednesday, 10 June 2015 at 09:12:17 UTC, John Chapman wrote:
 HMAC (for OAuth)
https://github.com/D-Programming-Language/phobos/pull/3233 Unfortunately it triggers a module cycle bug on FreeBSD that I can't figure out, so it hasn't been merged yet.
Jun 10 2015
prev sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Wednesday, 10 June 2015 at 07:56:46 UTC, John Chapman wrote:
 It's a shame ucent/cent never got implemented. But couldn't 
 they be added to Phobos? I often need a 128-bit type with 
 better precision than float and double.
I think the next release of LDC will support it, at least on some platforms...
Jun 10 2015
prev sibling next sibling parent "rsw0x" <anonymous anonymous.com> writes:
On Sunday, 7 June 2015 at 18:27:16 UTC, Robert burner Schadek 
wrote:
 Phobos is awesome, the libs of go, python and rust only have 
 better marketing.
 As discussed on dconf, phobos needs to become big and blow the 
 rest out of the sky.

 http://wiki.dlang.org/DIP80

 lets get OT, please discuss
std.container.concurrent.*
Jun 13 2015
prev sibling next sibling parent reply Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On 06/07/2015 02:27 PM, Robert burner Schadek wrote:
 Phobos is awesome, the libs of go, python and rust only have better
 marketing.
 As discussed on dconf, phobos needs to become big and blow the rest out
 of the sky.

 http://wiki.dlang.org/DIP80

 lets get OT, please discuss
What are the problems with std.json?
Jun 13 2015
parent reply "weaselcat" <weaselcat gmail.com> writes:
On Saturday, 13 June 2015 at 16:53:22 UTC, Nick Sabalausky wrote:
 On 06/07/2015 02:27 PM, Robert burner Schadek wrote:
 Phobos is awesome, the libs of go, python and rust only have 
 better
 marketing.
 As discussed on dconf, phobos needs to become big and blow the 
 rest out
 of the sky.

 http://wiki.dlang.org/DIP80

 lets get OT, please discuss
What are the problems with std.json?
slow
Jun 13 2015
parent "Dennis Ritchie" <dennis.ritchie mail.ru> writes:
Good start:
http://code.dlang.org/packages/dip80-ndslice
https://github.com/9il/dip80-ndslice/blob/master/source/std/experimental/range/ndslice.d

I miss the function `sliced` in Phobos.
Jun 13 2015
prev sibling parent reply "Ilya Yaroshenko" <ilyayaroshenko gmail.com> writes:
On Sunday, 7 June 2015 at 18:27:16 UTC, Robert burner Schadek 
wrote:
 Phobos is awesome, the libs of go, python and rust only have 
 better marketing.
 As discussed on dconf, phobos needs to become big and blow the 
 rest out of the sky.

 http://wiki.dlang.org/DIP80

 lets get OT, please discuss
N-dimensional slices is ready for comments! Announce http://forum.dlang.org/thread/rilfmeaqkailgpxoziuo forum.dlang.org Ilya
Jun 15 2015
parent reply "Dennis Ritchie" <dennis.ritchie mail.ru> writes:
On Monday, 15 June 2015 at 10:00:43 UTC, Ilya Yaroshenko wrote:
 N-dimensional slices is ready for comments!
It seems to me that the properties of the matrix require `row` and `col` like this: import std.stdio, std.experimental.range.ndslice, std.range : iota; void main() { auto matrix = 100.iota.sliced(3, 4, 5); writeln(matrix[0]); // [[0, 1, 2, 3, 4], [5, 6, 7, 8, 9], [10, 11, 12, 13, 14], [15, 16, 17, 18, 19]] // writeln(matrix[0].row); // 4 // writeln(matrix[0].col); // 5 } P.S. I'm not exactly sure that these properties should work exactly as in my code :)
Jun 15 2015
next sibling parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Monday, 15 June 2015 at 13:44:53 UTC, Dennis Ritchie wrote:
 On Monday, 15 June 2015 at 10:00:43 UTC, Ilya Yaroshenko wrote:
 N-dimensional slices is ready for comments!
It seems to me that the properties of the matrix require `row` and `col` like this: import std.stdio, std.experimental.range.ndslice, std.range : iota; void main() { auto matrix = 100.iota.sliced(3, 4, 5); writeln(matrix[0]); // [[0, 1, 2, 3, 4], [5, 6, 7, 8, 9], [10, 11, 12, 13, 14], [15, 16, 17, 18, 19]] // writeln(matrix[0].row); // 4 // writeln(matrix[0].col); // 5 } P.S. I'm not exactly sure that these properties should work exactly as in my code :)
try .length!0 and .length!1 or .shape[0] and .shape[1]
Jun 15 2015
parent "Ilya Yaroshenko" <ilyayaroshenko gmail.com> writes:
On Monday, 15 June 2015 at 13:55:16 UTC, John Colvin wrote:
 On Monday, 15 June 2015 at 13:44:53 UTC, Dennis Ritchie wrote:
 On Monday, 15 June 2015 at 10:00:43 UTC, Ilya Yaroshenko wrote:
 N-dimensional slices is ready for comments!
It seems to me that the properties of the matrix require `row` and `col` like this: import std.stdio, std.experimental.range.ndslice, std.range : iota; void main() { auto matrix = 100.iota.sliced(3, 4, 5); writeln(matrix[0]); // [[0, 1, 2, 3, 4], [5, 6, 7, 8, 9], [10, 11, 12, 13, 14], [15, 16, 17, 18, 19]] // writeln(matrix[0].row); // 4 // writeln(matrix[0].col); // 5 } P.S. I'm not exactly sure that these properties should work exactly as in my code :)
try .length!0 and .length!1 or .shape[0] and .shape[1]
Nitpick: shape contains lengths and strides: .shape.lengths[0] and .shape.lengths[1]
Jun 15 2015
prev sibling parent reply "Ilya Yaroshenko" <ilyayaroshenko gmail.com> writes:
On Monday, 15 June 2015 at 13:44:53 UTC, Dennis Ritchie wrote:
 On Monday, 15 June 2015 at 10:00:43 UTC, Ilya Yaroshenko wrote:
 N-dimensional slices is ready for comments!
It seems to me that the properties of the matrix require `row` and `col` like this: import std.stdio, std.experimental.range.ndslice, std.range : iota; void main() { auto matrix = 100.iota.sliced(3, 4, 5); writeln(matrix[0]); // [[0, 1, 2, 3, 4], [5, 6, 7, 8, 9], [10, 11, 12, 13, 14], [15, 16, 17, 18, 19]] // writeln(matrix[0].row); // 4 // writeln(matrix[0].col); // 5 } P.S. I'm not exactly sure that these properties should work exactly as in my code :)
This works: unittest { import std.stdio, std.experimental.range.ndslice; import std.range : iota; auto matrix = 100.iota.sliced(3, 4, 5); writeln(matrix[0]); writeln(matrix[0].length); // 4 writeln(matrix[0].length!0); // 4 writeln(matrix[0].length!1); // 5 writeln(matrix.length!2); // 5 } Prints: //[[0, 1, 2, 3, 4], [5, 6, 7, 8, 9], [10, 11, 12, 13, 14], [15, 16, 17, 18, 19]] //4 //4 //5 I am note sure that we need something like `height`/row and `width`/col for nd-slices. This kind of names can be used after casting to the future `std.container.matrix`.
Jun 15 2015
parent "Dennis Ritchie" <dennis.ritchie mail.ru> writes:
On Monday, 15 June 2015 at 14:32:20 UTC, Ilya Yaroshenko wrote:
 I am note sure that we need something like `height`/row and 
 `width`/col for nd-slices. This kind of names can be used after 
 casting to the future `std.container.matrix`.
Here something similar implemented: https://github.com/k3kaimu/carbon/blob/master/source/carbon/linear.d#L52-L56 Want in the future something like `rows' and `cols`: https://github.com/k3kaimu/carbon/blob/master/source/carbon/linear.d#L156-L157 Waiting for `static foreach`. This design really helps a lot to implement multidimensional slices.
Jun 15 2015