www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - colour lib needs reviewers

reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
I think I'm about as happy with my colour lib as I'm going to be.
It really needs reviews.

I added packed-RGB support, including weird micro-float and
shared-exponent formats.
They're important for interacting with any real-time rendering libraries.
There is only one texture format I'm aware of that isn't supported,
and that is u7e3 floats, only available on Xbox360 ;)

Chromatic adaptation functions are in.
I've done a tidy/reorg pass.

I'm not sure what else should be in the scope of this lib.

PR: https://github.com/dlang/phobos/pull/2845
dub: https://code.dlang.org/packages/color
docs: http://dtest.thecybershadow.net/artifact/website-04a64e024cf75be39700bebd3a50d26f6c7bd163-7185c8ec7b15c9e785880cab6d512e6f/web/library-prerelease/std/experimental/color.html

- Manu
Sep 11 2016
next sibling parent reply Guillaume Chatelet <chatelet.guillaume gmail.com> writes:
On Monday, 12 September 2016 at 04:14:27 UTC, Manu wrote:

 I think I'm about as happy with my colour lib as I'm going to 
 be. It really needs reviews.

 I added packed-RGB support, including weird micro-float and
 shared-exponent formats.
 They're important for interacting with any real-time rendering 
 libraries.
 There is only one texture format I'm aware of that isn't 
 supported,
 and that is u7e3 floats, only available on Xbox360 ;)

 Chromatic adaptation functions are in.
 I've done a tidy/reorg pass.

 I'm not sure what else should be in the scope of this lib.

 PR: https://github.com/dlang/phobos/pull/2845
 dub: https://code.dlang.org/packages/color
 docs: 
 http://dtest.thecybershadow.net/artifact/website-04a64e024cf75be39700bebd3a50d26f6c7bd163-7185c8ec7b15c9e785880cab6d512e6f/web/library-prerelease/std/experimental/color.html

 - Manu


Looks pretty good to me.

Quick question though. Do you handle endianness?
DPX might be a dead file format but it's main usage relies on 
10bits packed RGB into 32 bits (last two bits are discarded or 
alpha) but it can be either little or big endian.

Good job on the library though!
Sep 12 2016
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
I thought about it, but it added (even more) complexity, and I'm not
really sure it's the MO of a color lib to worry about the endian-ness
of storage.

I could support "_le"/"_be" suffix on the PackedRGB format string, but
then you have to also specify the word size, ie, is the packed color
32bit words (eg, 11,11,10) or 16bit words (16f,16f,16f,16f)?
It's only really practically applicable to PackedRGB, so I wonder why
just one color type should worry about it, but not all the others...

Consider an unpacked RGB with 16bit elements... that may also suffer
endian problems, but RGB is a runtime type, not a storage type, which
means it doesn't have a pack/unpack process... so there's no
meaningful palace to do it.

On 12 September 2016 at 18:09, Guillaume Chatelet via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Monday, 12 September 2016 at 04:14:27 UTC, Manu wrote:

 I think I'm about as happy with my colour lib as I'm going to be. It
 really needs reviews.

 I added packed-RGB support, including weird micro-float and
 shared-exponent formats.
 They're important for interacting with any real-time rendering libraries.
 There is only one texture format I'm aware of that isn't supported,
 and that is u7e3 floats, only available on Xbox360 ;)

 Chromatic adaptation functions are in.
 I've done a tidy/reorg pass.

 I'm not sure what else should be in the scope of this lib.

 PR: https://github.com/dlang/phobos/pull/2845
 dub: https://code.dlang.org/packages/color
 docs:
 http://dtest.thecybershadow.net/artifact/website-04a64e024cf75be39700bebd3a50d26f6c7bd163-7185c8ec7b15c9e785880cab6d512e6f/web/library-prerelease/std/experimental/color.html

 - Manu



 Looks pretty good to me.

 Quick question though. Do you handle endianness?
 DPX might be a dead file format but it's main usage relies on 10bits packed
 RGB into 32 bits (last two bits are discarded or alpha) but it can be either
 little or big endian.

 Good job on the library though!
Sep 12 2016
parent reply Guillaume Chatelet <chatelet.guillaume gmail.com> writes:
On Monday, 12 September 2016 at 09:09:29 UTC, Manu wrote:

 I thought about it, but it added (even more) complexity, and 
 I'm not really sure it's the MO of a color lib to worry about 
 the endian-ness of storage.

 I could support "_le"/"_be" suffix on the PackedRGB format 
 string, but
 then you have to also specify the word size, ie, is the packed 
 color
 32bit words (eg, 11,11,10) or 16bit words (16f,16f,16f,16f)?
 It's only really practically applicable to PackedRGB, so I 
 wonder why
 just one color type should worry about it, but not all the 
 others...

 Consider an unpacked RGB with 16bit elements... that may also 
 suffer endian problems, but RGB is a runtime type, not a 
 storage type, which means it doesn't have a pack/unpack 
 process... so there's no meaningful palace to do it.


Yep I totally understand the rationale - I've been there too :)
I don't think it belong to Color but maybe Buffer(1) could be 
templated on endianness? Buffer is indeed the place where you 
deal with storage.
If you don't want to add it to the library I think you should add 
a paragraph in the documentation explaining why it's not in there 
and how people can overcome it (copy + change endianness or use 
an adapter?).

Also, it would be nice to have some code snippets in the 
documentation.

Last thing, for the white points you didn't specify which CIE 
standard observer(2) version you used: CIE 1931 2° Standard 
Observer or CIE 1964 10° Standard Observer? IIRC the standard 
illuminant are not the same depending on the observer.



1. 
http://dtest.thecybershadow.net/artifact/website-04a64e024cf75be39700bebd3a50d26f6c7bd163-7185c8ec7b15c9e785880cab6d512e6f/web/library-prerelease/std/experimental/color/packedrgb/buffer.html
2. 
https://en.wikipedia.org/wiki/CIE_1931_color_space#CIE_standard_observer
Sep 12 2016
parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 12 September 2016 at 21:13, Guillaume Chatelet via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Monday, 12 September 2016 at 09:09:29 UTC, Manu wrote:

 I thought about it, but it added (even more) complexity, and I'm not
 really sure it's the MO of a color lib to worry about the endian-ness of
 storage.

 I could support "_le"/"_be" suffix on the PackedRGB format string, but
 then you have to also specify the word size, ie, is the packed color
 32bit words (eg, 11,11,10) or 16bit words (16f,16f,16f,16f)?
 It's only really practically applicable to PackedRGB, so I wonder why
 just one color type should worry about it, but not all the others...

 Consider an unpacked RGB with 16bit elements... that may also suffer
 endian problems, but RGB is a runtime type, not a storage type, which means
 it doesn't have a pack/unpack process... so there's no meaningful palace to
 do it.



 Yep I totally understand the rationale - I've been there too :)
 I don't think it belong to Color but maybe Buffer(1) could be templated on
 endianness? Buffer is indeed the place where you deal with storage.
 If you don't want to add it to the library I think you should add a
 paragraph in the documentation explaining why it's not in there and how
 people can overcome it (copy + change endianness or use an adapter?).

 Also, it would be nice to have some code snippets in the documentation.

 Last thing, for the white points you didn't specify which CIE standard
 observer(2) version you used: CIE 1931 2° Standard Observer or CIE 1964 10°
 Standard Observer? IIRC the standard illuminant are not the same depending
 on the observer.


That's very true. The data is all CIE 1931 reference. I haven't seen
deployed use of the updated standard... I don't know why.
Sep 12 2016
prev sibling next sibling parent Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Monday, 12 September 2016 at 04:14:27 UTC, Manu wrote:

 I think I'm about as happy with my colour lib as I'm going to 
 be. It really needs reviews.

 [...]


Looks great!

1. packedrgb has not enough docs to understand what it is.
2. colorFromString should have scope argument.

- Ilya
Sep 12 2016
prev sibling next sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Monday, 12 September 2016 at 04:14:27 UTC, Manu wrote:

 I think I'm about as happy with my colour lib as I'm going to 
 be. It really needs reviews.

 I'm not sure what else should be in the scope of this lib.

 PR: https://github.com/dlang/phobos/pull/2845
 dub: https://code.dlang.org/packages/color
 docs: 
 http://dtest.thecybershadow.net/artifact/website-04a64e024cf75be39700bebd3a50d26f6c7bd163-7185c8ec7b15c9e785880cab6d512e6f/web/library-prerelease/std/experimental/color.html

 [...]
 - Manu


1. Maybe that the each package ddoc comment should mention that 
everything is always  safe nothrow  nogc...For me it wasn't 
directly clear:
- seen that there's opCast: Q: safe ?
- seen in the sources that opCast call a templatized function
- seen that the unittest for this function are not annotated
- then only seen that the leading " safe nothrow  nogc:" at the 
beg of the package.

2. Q: is there anything to convert a color to grey scale ?
Sep 12 2016
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 12 September 2016 at 19:30, Basile B. via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Monday, 12 September 2016 at 04:14:27 UTC, Manu wrote:

 I think I'm about as happy with my colour lib as I'm going to be. It
 really needs reviews.

 I'm not sure what else should be in the scope of this lib.

 PR: https://github.com/dlang/phobos/pull/2845
 dub: https://code.dlang.org/packages/color
 docs:
 http://dtest.thecybershadow.net/artifact/website-04a64e024cf75be39700bebd3a50d26f6c7bd163-7185c8ec7b15c9e785880cab6d512e6f/web/library-prerelease/std/experimental/color.html

 [...]
 - Manu



 1. Maybe that the each package ddoc comment should mention that everything
 is always  safe nothrow  nogc...For me it wasn't directly clear:
 - seen that there's opCast: Q: safe ?
 - seen in the sources that opCast call a templatized function
 - seen that the unittest for this function are not annotated
 - then only seen that the leading " safe nothrow  nogc:" at the beg of the
 package.


How do you think this should this look? (it's best to leave these
comments on the PR)



 2. Q: is there anything to convert a color to grey scale ?


This isn't a short-answer question :) .. There are many ways depending
on the job.

A simple way might be:
  L8 grey = cast(L8)RGB8(r, g, b); // L8 is luminance only, casting
will do the right thing

This is the 'normal' way, but there are better ways...

A superior way if you want a top-quality result would be to use the
Lab luminance value, since Lab is perceptually uniform, when you
desaturate, it maintains the colors relative luminance correctly,
something like:

  Lab!float lab = cast(Lab!float)RGB8(r, g, b); // cast your color to Lab
  lab.a = 0; // eliminate saturation on a axis
  lab.b = 0; // eliminate saturation on b axis
  // lab.L is now pure luminance
  RGB8 c = cast(RGB8)lab; // cast it back to RGB, and your RGB will be
greyscale, but with more perceptually correct conversion.

Lots of pow's in that conversion, so it's slower.

This link demonstrates some competing techniques:
https://en.wikipedia.org/wiki/HSL_and_HSV#/media/File:Fire-breather_CIELAB_L*.jpg
Press left/right to cycle through them.
The first method (cast to L8) is: Rec. 601 luma Y′.  (or more
accurately, my code above would be: sRGB luma Y')
The second method (Lab.L) is: CIELAB L*.

L* performs much better in low light.
Sep 12 2016
parent reply John Colvin <john.loughran.colvin gmail.com> writes:
On Monday, 12 September 2016 at 13:04:49 UTC, Manu wrote:

 2. Q: is there anything to convert a color to grey scale ?


 This isn't a short-answer question :) .. There are many ways 
 depending on the job.

 A simple way might be:
   L8 grey = cast(L8)RGB8(r, g, b); // L8 is luminance only, 
 casting
 will do the right thing

 This is the 'normal' way, but there are better ways...

 A superior way if you want a top-quality result would be to use 
 the Lab luminance value, since Lab is perceptually uniform, 
 when you desaturate, it maintains the colors relative luminance 
 correctly, something like:

   Lab!float lab = cast(Lab!float)RGB8(r, g, b); // cast your 
 color to Lab
   lab.a = 0; // eliminate saturation on a axis
   lab.b = 0; // eliminate saturation on b axis
   // lab.L is now pure luminance
   RGB8 c = cast(RGB8)lab; // cast it back to RGB, and your RGB 
 will be
 greyscale, but with more perceptually correct conversion.

 Lots of pow's in that conversion, so it's slower.

 This link demonstrates some competing techniques:
 https://en.wikipedia.org/wiki/HSL_and_HSV#/media/File:Fire-breather_CIELAB_L*.jpg
 Press left/right to cycle through them.
 The first method (cast to L8) is: Rec. 601 luma Y′.  (or more
 accurately, my code above would be: sRGB luma Y')
 The second method (Lab.L) is: CIELAB L*.

 L* performs much better in low light.


Is there some uniform policy w.r.t. casts here? If I take one 
colour type and cast it to another will it always do something 
unambiguously right or not compile? Seems to me that a function 
that takes a target type would be more obvious for the user.

I think the multiple different meanings of cast are a mistake in 
D. We can't go back now, but I wish there was only 
reinterpet_cast and everything else was a function.
Sep 12 2016
parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 13 September 2016 at 01:30, John Colvin via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Monday, 12 September 2016 at 13:04:49 UTC, Manu wrote:

 2. Q: is there anything to convert a color to grey scale ?



 This isn't a short-answer question :) .. There are many ways depending on
 the job.

 A simple way might be:
   L8 grey = cast(L8)RGB8(r, g, b); // L8 is luminance only, casting
 will do the right thing

 This is the 'normal' way, but there are better ways...

 A superior way if you want a top-quality result would be to use the Lab
 luminance value, since Lab is perceptually uniform, when you desaturate, it
 maintains the colors relative luminance correctly, something like:

   Lab!float lab = cast(Lab!float)RGB8(r, g, b); // cast your color to Lab
   lab.a = 0; // eliminate saturation on a axis
   lab.b = 0; // eliminate saturation on b axis
   // lab.L is now pure luminance
   RGB8 c = cast(RGB8)lab; // cast it back to RGB, and your RGB will be
 greyscale, but with more perceptually correct conversion.

 Lots of pow's in that conversion, so it's slower.

 This link demonstrates some competing techniques:

 https://en.wikipedia.org/wiki/HSL_and_HSV#/media/File:Fire-breather_CIELAB_L*.jpg
 Press left/right to cycle through them.
 The first method (cast to L8) is: Rec. 601 luma Y′.  (or more
 accurately, my code above would be: sRGB luma Y')
 The second method (Lab.L) is: CIELAB L*.

 L* performs much better in low light.



 Is there some uniform policy w.r.t. casts here? If I take one colour type
 and cast it to another will it always do something unambiguously right or
 not compile?


I think, yes?
So, the logic here is, if you cast, and then cast back, you will have
the same thing barring loss of precision.
Casts in this lib just change format (+precision), they don't change
value at all. I feel it's compatible with casting an int to a float...
it's the same sort of conversion.



 Seems to me that a function that takes a target type would be
 more obvious for the user.


Really? I like the cast this way. There is an explicit function in
package.d though...
Sep 12 2016
prev sibling next sibling parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Mon, 12 Sep 2016 14:14:27 +1000
schrieb Manu via Digitalmars-d <digitalmars-d puremagic.com>:


 I think I'm about as happy with my colour lib as I'm going to be.
 It really needs reviews.
 
 I added packed-RGB support, including weird micro-float and
 shared-exponent formats.
 They're important for interacting with any real-time rendering libraries.
 There is only one texture format I'm aware of that isn't supported,
 and that is u7e3 floats, only available on Xbox360 ;)
 
 Chromatic adaptation functions are in.
 I've done a tidy/reorg pass.
 
 I'm not sure what else should be in the scope of this lib.
 
 PR: https://github.com/dlang/phobos/pull/2845
 dub: https://code.dlang.org/packages/color
 docs: http://dtest.thecybershadow.net/artifact/website-04a64e024cf75be39700bebd3a50d26f6c7bd163-7185c8ec7b15c9e785880cab6d512e6f/web/library-prerelease/std/experimental/color.html
 
 - Manu


Nice work. What's u7e3? 3*10-bit-floats + 2-bit padding?

In the example for convert color you show that converting from
a color space without alpha (XYZ) to one with alpha (RGBA)
turns the pixel transparent. I believe you meant this as
something to lookout for when converting colors. What's the
reasoning for not setting maximum alpha as a more useful and
natural default?

Cool stuff there in the colorspace module, to get started with
getting photos or DV/MPEG into a linear color space. Should
BT.709 also have prefab conversion methods? It would seem HD
video is more wide spread nowadays.

There is a typo in "Function that converts a linear luminance
to gamme space." (colorspace.d)

I like how RGB color spaces are described as primaries in xyY
+ gamma functions.

It seems inconsistent that you can specify all kinds of
exponents when the exponent is shared but are forced to 5 bit
exponents when mixed into each component, while the mantissa
can still be adjusted. Would m7e3 have worked? (With f* being
the IEEE standard formats.)

The final package should contain a fair amount of
documentation, in my opinion also on the topics of sRGB vs.
linear and what "hybrid gamma" is.

What is your stance on pixel formats that are arranged in
planes? I assume that e.g. YUV is handled by a single channel
"sRGB luminance" + 2 signed 8-bit color planes? It may be
worth adding some hints in that direction.

All in all good work with a scientific feel to it.

-- 
Marco
Sep 12 2016
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 12 September 2016 at 21:28, Marco Leise via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 Am Mon, 12 Sep 2016 14:14:27 +1000
 schrieb Manu via Digitalmars-d <digitalmars-d puremagic.com>:


 I think I'm about as happy with my colour lib as I'm going to be.
 It really needs reviews.

 I added packed-RGB support, including weird micro-float and
 shared-exponent formats.
 They're important for interacting with any real-time rendering libraries.
 There is only one texture format I'm aware of that isn't supported,
 and that is u7e3 floats, only available on Xbox360 ;)

 Chromatic adaptation functions are in.
 I've done a tidy/reorg pass.

 I'm not sure what else should be in the scope of this lib.

 PR: https://github.com/dlang/phobos/pull/2845
 dub: https://code.dlang.org/packages/color
 docs: http://dtest.thecybershadow.net/artifact/website-04a64e024cf75be39700bebd3a50d26f6c7bd163-7185c8ec7b15c9e785880cab6d512e6f/web/library-prerelease/std/experimental/color.html

 - Manu


 Nice work. What's u7e3? 3*10-bit-floats + 2-bit padding?


Yup.



 In the example for convert color you show that converting from
 a color space without alpha (XYZ) to one with alpha (RGBA)
 turns the pixel transparent. I believe you meant this as
 something to lookout for when converting colors. What's the
 reasoning for not setting maximum alpha as a more useful and
 natural default?


I flip-flopped on this multiple times.
It's not so simple.
1. Alpha doesn't necessarily mean transparently, that's just one
(extremely common) use
2. 1 is the multiplication identity, but 0 is the additive identity. I
cant find either addition or multiplication to take precedence over
eachother.
3. With those 2 points above already making me feel a bit worried
about it, I think that appearing numbers out of nowhere is just
generally bad, so I went with a=0.

If alpha defaults to 1, then a*b works as expected (object does not
become transparent by this operation).
If alpha defaults to 0, then b-a works as expected (object does not
become transparent by this operation).
With that in mind, you realise alpha will *always* require explicit
handling... so I think 0 is a better default at that stage. It may be
possible to convince me otherwise ;)



 Cool stuff there in the colorspace module, to get started with
 getting photos or DV/MPEG into a linear color space. Should
 BT.709 also have prefab conversion methods? It would seem HD
 video is more wide spread nowadays.


BT.709's gamma func isn't changed from 601. Rec.2020 introduced a
higher precision version.



 There is a typo in "Function that converts a linear luminance
 to gamme space." (colorspace.d)


Thx. But please log comments like this on the PR so I don't lose track
of them? :)



 I like how RGB color spaces are described as primaries in xyY
 + gamma functions.


Seems natural :)

So, I was initially planning to make that struct the RGB template
argument, but then I realised that would make the symbols gigantic
(heaps of CS data in the symbol names), so I introduced the enum...
sadly though, this means you can't define a custom colour without
introducing a new custom-colour type to do what I originally intended.
I'm upset about this either way...
I think I should leave it how it is... custom colours are probably
more rare than gigantic symbol names ;)



 It seems inconsistent that you can specify all kinds of
 exponents when the exponent is shared but are forced to 5 bit
 exponents when mixed into each component, while the mantissa
 can still be adjusted. Would m7e3 have worked? (With f* being
 the IEEE standard formats.)


I was thinking that, and mentioned as such in the comments near those lines ;)
Thing is, there are no realtime small floats with exponent != 5bits
except the unique 7e3 format on xbox 360.
I can extend this in an unbreaking way.

It's just pretty verbose is all: "rgb_11e5_11e5_10e5". I mean, it's
fine... but maybe people wouldn't find this intuitive? Many
programmers don't know what an 'exponent' or 'mantissa' is... :/



 The final package should contain a fair amount of
 documentation, in my opinion also on the topics of sRGB vs.
 linear and what "hybrid gamma" is.


Yeah, but is it really my job to teach someone about colour space?
Wikipedia can do that much better than I can... :/
I agree, but I don't really know where to draw the line, or how to organise it.

Perhaps people with strong opinions on how this should be presented
can create a PR?



 What is your stance on pixel formats that are arranged in
 planes? I assume that e.g. YUV is handled by a single channel
 "sRGB luminance" + 2 signed 8-bit color planes? It may be
 worth adding some hints in that direction.


I haven't supported YUV, but it is as you say. Nitpick: most of those
YUV sinals are probably broadcast signals, with rec.601 luminance ;)
YUV is a pretty big family, I'll do it if it's requested... There are
a lot of micro-permutations. It's easy to get carried away (ie; as you
can see in HSx).

I don't think 'types' can really express bitplanes well. We need to
express array operations to support them.
I want to extend this to support efficient array operations, but I
don't  really know how to work this into D's ranges efficiently, and
my other topic on the matter died before anything worthwhile emerged.



 All in all good work with a scientific feel to it.


Thanks, I struggled with this balance a lot. I'm fairly happy with it now.
Sep 12 2016
parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Tue, 13 Sep 2016 00:37:22 +1000
schrieb Manu via Digitalmars-d <digitalmars-d puremagic.com>:

 I flip-flopped on this multiple times.
 It's not so simple.
 1. Alpha doesn't necessarily mean transparently, that's just one
 (extremely common) use
 2. 1 is the multiplication identity, but 0 is the additive identity. I
 cant find either addition or multiplication to take precedence over
 eachother.
 3. With those 2 points above already making me feel a bit worried
 about it, I think that appearing numbers out of nowhere is just
 generally bad, so I went with a=0.



 If alpha defaults to 1, then a*b works as expected (object does not
 become transparent by this operation).
 If alpha defaults to 0, then b-a works as expected (object does not
 become transparent by this operation).


Additive color components can work like that, but the alpha
component just doesn't represent an amount of light. Don't try
too hard to make the arithmetics make sense, when the
practical use lies in 100% alpha by default. Converting every
color to full transparent is not helpful when you want to save
a 24-bit BMP as 32-bit PNG or get an RGBA color from an RGB
texture.
I don't know who declared that 100% alpha means opaque. For
what it's worth it could have been the other way around and
all the (1-a) and (a) in blending functions swapped.
And a blending function is what is needed here to give any
meaning at all to arithmetics on alpha. An additive one may be:

  color = color_dst + color_src * alpha_src
  alpha = alpha_dst


 With that in mind, you realise alpha will *always* require explicit
 handling... so I think 0 is a better default at that stage. It may be
 possible to convince me otherwise ;)


I fully agree with the first line, I just think that for
practical uses alpha is considered to be 100% in pixel formats
without alpha channel. Otherwise the most popular blending
functions would do nothing on RGB colors.
SDL for example also uses 100% alpha by default: "If the
specified pixel format has an alpha component it will be
returned as all 1 bits (fully opaque)."


 Cool stuff there in the colorspace module, to get started with
 getting photos or DV/MPEG into a linear color space. Should
 BT.709 also have prefab conversion methods? It would seem HD
 video is more wide spread nowadays.  


 
 BT.709's gamma func isn't changed from 601. Rec.2020 introduced a
 higher precision version.



 There is a typo in "Function that converts a linear luminance
 to gamme space." (colorspace.d)  


 
 Thx. But please log comments like this on the PR so I don't lose track
 of them? :)


Will do!


 It seems inconsistent that you can specify all kinds of
 exponents when the exponent is shared but are forced to 5 bit
 exponents when mixed into each component, while the mantissa
 can still be adjusted. Would m7e3 have worked? (With f* being
 the IEEE standard formats.)  


 
 I was thinking that, and mentioned as such in the comments near those lines ;)
 Thing is, there are no realtime small floats with exponent != 5bits
 except the unique 7e3 format on xbox 360.
 I can extend this in an unbreaking way.

 It's just pretty verbose is all: "rgb_11e5_11e5_10e5". I mean, it's
 fine... but maybe people wouldn't find this intuitive? Many
 programmers don't know what an 'exponent' or 'mantissa' is... :/


Yeah I thought that the common formats would be the IEEE
ones, especially f16 and that anything less are very niche
custom formats that come and go and need verbose descriptions.
Well, either way works.


 The final package should contain a fair amount of
 documentation, in my opinion also on the topics of sRGB vs.
 linear and what "hybrid gamma" is.  


 
 Yeah, but is it really my job to teach someone about colour space?
 Wikipedia can do that much better than I can... :/
 I agree, but I don't really know where to draw the line, or how to organise it.

 Perhaps people with strong opinions on how this should be presented
 can create a PR?


Alright, but hybrid gamma is really not something that can be
googled. Or rather I end up at Toyota's Gamma Hybrid product
page. :D

And as always, don't let the turkeys get you down.

-- 
Marco
Sep 12 2016
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 13 September 2016 at 07:00, Marco Leise via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 Am Tue, 13 Sep 2016 00:37:22 +1000
 schrieb Manu via Digitalmars-d <digitalmars-d puremagic.com>:

 I flip-flopped on this multiple times.
 It's not so simple.
 1. Alpha doesn't necessarily mean transparently, that's just one
 (extremely common) use
 2. 1 is the multiplication identity, but 0 is the additive identity. I
 cant find either addition or multiplication to take precedence over
 eachother.
 3. With those 2 points above already making me feel a bit worried
 about it, I think that appearing numbers out of nowhere is just
 generally bad, so I went with a=0.



 If alpha defaults to 1, then a*b works as expected (object does not
 become transparent by this operation).
 If alpha defaults to 0, then b-a works as expected (object does not
 become transparent by this operation).


 Additive color components can work like that, but the alpha
 component just doesn't represent an amount of light. Don't try
 too hard to make the arithmetics make sense, when the
 practical use lies in 100% alpha by default.


I understand the argument, I have stressed over this to no end.
I still think it's just not 'right'. :/



 Converting every
 color to full transparent is not helpful when you want to save
 a 24-bit BMP as 32-bit PNG or get an RGBA color from an RGB
 texture.


What is the worth of storing alpha data if it's uniform 0xFF anyway?
It sounds like you mean rgbx, not rgba (ie, 32bit, but no alpha).
There should only be an alpha channel if there's actually alpha data... right?



 I don't know who declared that 100% alpha means opaque. For
 what it's worth it could have been the other way around and
 all the (1-a) and (a) in blending functions swapped.
 And a blending function is what is needed here to give any
 meaning at all to arithmetics on alpha. An additive one may be:

   color = color_dst + color_src * alpha_src
   alpha = alpha_dst


I thought about adding blend's, but I stopped short on that. I think
that's firmly entering image processing territory, and I felt that was
one step beyond the MO of this particular lib... no?
Blending opens up a whole world.



 With that in mind, you realise alpha will *always* require explicit
 handling... so I think 0 is a better default at that stage. It may be
 possible to convince me otherwise ;)


 I fully agree with the first line, I just think that for
 practical uses alpha is considered to be 100% in pixel formats
 without alpha channel.


'rgbx' is for image formats without alpha channels...



 Otherwise the most popular blending
 functions would do nothing on RGB colors.


'rgbx' colors fed through a blend function would always properly imply alpha=1.



 SDL for example also uses 100% alpha by default: "If the
 specified pixel format has an alpha component it will be
 returned as all 1 bits (fully opaque)."



From which functions? Link me?


I'd love to see more precedents.



 It seems inconsistent that you can specify all kinds of
 exponents when the exponent is shared but are forced to 5 bit
 exponents when mixed into each component, while the mantissa
 can still be adjusted. Would m7e3 have worked? (With f* being
 the IEEE standard formats.)


 I was thinking that, and mentioned as such in the comments near those lines ;)
 Thing is, there are no realtime small floats with exponent != 5bits
 except the unique 7e3 format on xbox 360.
 I can extend this in an unbreaking way.

 It's just pretty verbose is all: "rgb_11e5_11e5_10e5". I mean, it's
 fine... but maybe people wouldn't find this intuitive? Many
 programmers don't know what an 'exponent' or 'mantissa' is... :/


 Yeah I thought that the common formats would be the IEEE
 ones, especially f16 and that anything less are very niche
 custom formats that come and go and need verbose descriptions.
 Well, either way works.


The verbose idea I said above is my best idea. I'll roll with that if
enough people agree.
There was the other fellow who posted before who feels like it's
already too complicated for him though.

I've been very carefully trying to put the complicated bits just one
level back from arms reach, so a casual user isn't confronted with
those details.
Putting that complexity in the format string makes it very
user-facing. That's the part I'm not wild about.



 The final package should contain a fair amount of
 documentation, in my opinion also on the topics of sRGB vs.
 linear and what "hybrid gamma" is.


 Yeah, but is it really my job to teach someone about colour space?
 Wikipedia can do that much better than I can... :/
 I agree, but I don't really know where to draw the line, or how to organise it.

 Perhaps people with strong opinions on how this should be presented
 can create a PR?


 Alright, but hybrid gamma is really not something that can be
 googled. Or rather I end up at Toyota's Gamma Hybrid product
 page. :D


True. I'm not even sure what the technical term for this sort of gamma
function is... I just made that up! :/
As Walter and others have asked, I'll have to start adding links to
reference material I guess, although that still feels really weird to
me for some reason.
Sep 12 2016
next sibling parent reply Random D user <no email.com> writes:
On Tuesday, 13 September 2016 at 02:00:44 UTC, Manu wrote:

 On 13 September 2016 at 07:00, Marco Leise via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:

 Am Tue, 13 Sep 2016 00:37:22 +1000
 schrieb Manu via Digitalmars-d <digitalmars-d puremagic.com>:
 Alright, but hybrid gamma is really not something that can be 
 googled. Or rather I end up at Toyota's Gamma Hybrid product 
 page. :D


 True. I'm not even sure what the technical term for this sort 
 of gamma
 function is... I just made that up! :/
 As Walter and others have asked, I'll have to start adding 
 links to
 reference material I guess, although that still feels really 
 weird to
 me for some reason.


FWIW I stumbled into this while double-checking HDR standards for 
my previous post. (I'm not a HDR expert, only somewhat interested 
because it's the future of displays/graphics)

https://en.wikipedia.org/wiki/Hybrid_Log-Gamma
Sep 13 2016
parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 14 September 2016 at 04:25, Random D user via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Tuesday, 13 September 2016 at 02:00:44 UTC, Manu wrote:

 On 13 September 2016 at 07:00, Marco Leise via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 Am Tue, 13 Sep 2016 00:37:22 +1000
 schrieb Manu via Digitalmars-d <digitalmars-d puremagic.com>:
 Alright, but hybrid gamma is really not something that can be googled. Or
 rather I end up at Toyota's Gamma Hybrid product page. :D



 True. I'm not even sure what the technical term for this sort of gamma
 function is... I just made that up! :/
 As Walter and others have asked, I'll have to start adding links to
 reference material I guess, although that still feels really weird to
 me for some reason.



 FWIW I stumbled into this while double-checking HDR standards for my
 previous post. (I'm not a HDR expert, only somewhat interested because it's
 the future of displays/graphics)

 https://en.wikipedia.org/wiki/Hybrid_Log-Gamma


Perfect, thanks!
Sep 13 2016
prev sibling next sibling parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Tue, 13 Sep 2016 12:00:44 +1000
schrieb Manu via Digitalmars-d <digitalmars-d puremagic.com>:


 What is the worth of storing alpha data if it's uniform 0xFF anyway?
 It sounds like you mean rgbx, not rgba (ie, 32bit, but no alpha).
 There should only be an alpha channel if there's actually alpha data... r=


ight?

I don't mean RGBX. JavaScript's canvas works that way for
example. I.e. the only pixel format is RGBA for simplicity's
sake and I'm not surprised it actually draws something if I
load it with a 24-bit graphic. ;)


 [=E2=80=A6] An additive one may be:

   color =3D color_dst + color_src * alpha_src
   alpha =3D alpha_dst =20


=20
 I thought about adding blend's, but I stopped short on that. I think
 that's firmly entering image processing territory, and I felt that was
 one step beyond the MO of this particular lib... no?
 Blending opens up a whole world.


I agree with that decision, and that it entails that
arithmetics are undefined for alpha channels. :-( Yeah bummer.
The idea that basic (saturating) arithmetics work on colors is
a great simplification that works for the most part, but let's
be fair, multiplying two HSV colors isn't exactly going to
yield a well defined hue either, just as multiplying two
angles doesn't give you a new angle. I.e.:
http://math.stackexchange.com/a/47880


 [=E2=80=A6]


 [=E2=80=A6]
 From which functions? Link me? =20
 I'd love to see more precedents.


Yep, that's better than arguing :)
So here are all graphics APIs I know and what they do when
they encounter colors without alpha and need a default value:

SDL:
https://wiki.libsdl.org/SDL_MapRGB
"If the specified pixel format has an alpha component it will be returned a=
s all 1 bits (fully opaque)."

Allegro:
https://github.com/liballeg/allegro5/blob/master/include/allegro5/internal/=
aintern_pixels.h#L59
(No docs, just source code that defaults to 255 for alpha when
converting a color from a bitmap with non-alpha pixel format
to an ALLEGRO_COLOR.)

Cairo:
https://www.cairographics.org/manual/cairo-cairo-t.html#cairo-set-source-rgb
"Sets the source pattern within cr to an opaque color."

Microsoft GDI+:
https://msdn.microsoft.com/de-de/library/windows/desktop/ms536255%28v=3Dvs.=
85%29.aspx
"The default value of the alpha component for this Color
object is 255."

Gnome GDK:
https://developer.gnome.org/gdk-pixbuf/2.33/gdk-pixbuf-Utilities.html#gdk-p=
ixbuf-add-alpha
"[=E2=80=A6] the alpha channel is initialized to 255 (full opacity)."

Qt:
http://doc.qt.io/qt-4.8/qcolor.html#alpha-blended-drawing
"By default, the alpha-channel is set to 255 (opaque)."

OpenGL:
https://www.opengl.org/sdk/docs/man2/xhtml/glColor.xml
"glColor3 variants specify new red, green, and blue values
explicitly and set the current alpha value to 1.0 (full
intensity) implicitly."
(Note: The color can be queried and shows a=3D1.0 without
blending operations setting it internally if needed.)

Java (AWT):
https://docs.oracle.com/javase/7/docs/api/java/awt/Color.html#Color%28int,%=
20boolean%29
"If the hasalpha argument is false, alpha is defaulted to 255."

Apple's Quartz does not seem to provide color space
conversions and always requires the user to give the alpha
value for new colors, so there is no default:
https://developer.apple.com/library/tvos/documentation/GraphicsImaging/Refe=



One thing I noticed is that many of those strictly separate
color spaces from alpha as concepts. For example in Quartz
*all* color spaces have alpha. In Allegro color space
conversions are ignorant of alpha. That begs the question
what should happen when you convert RGBA to a HLx
color space and back to RGBA. Can you retain the alpha value?
CSS3 for example has HSLA colors that raise the bar a bit.

--=20
Marco
Sep 13 2016
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 14 September 2016 at 04:34, Marco Leise via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 Am Tue, 13 Sep 2016 12:00:44 +1000
 schrieb Manu via Digitalmars-d <digitalmars-d puremagic.com>:


 What is the worth of storing alpha data if it's uniform 0xFF anyway?
 It sounds like you mean rgbx, not rgba (ie, 32bit, but no alpha).
 There should only be an alpha channel if there's actually alpha data... right?


 I don't mean RGBX. JavaScript's canvas works that way for
 example. I.e. the only pixel format is RGBA for simplicity's
 sake and I'm not surprised it actually draws something if I
 load it with a 24-bit graphic. ;)


Given this example, isn't it the job of the image loader to populate
the image with data?
If you ask for an RGBA image loaded from a 24bit BMP or something,
sure, it should put 0xFF in there, but I don't know if it's right that
that logic belongs here...?


 […] An additive one may be:

   color = color_dst + color_src * alpha_src
   alpha = alpha_dst


 I thought about adding blend's, but I stopped short on that. I think
 that's firmly entering image processing territory, and I felt that was
 one step beyond the MO of this particular lib... no?
 Blending opens up a whole world.


 I agree with that decision, and that it entails that
 arithmetics are undefined for alpha channels. :-( Yeah bummer.
 The idea that basic (saturating) arithmetics work on colors is
 a great simplification that works for the most part, but let's
 be fair, multiplying two HSV colors isn't exactly going to
 yield a well defined hue either,


You'll notice I didn't add arithmetic operators to the HSx type ;)
If you have HSx colors,  and want to do arithmetic, cast it to RGB first.



 just as multiplying two
 angles doesn't give you a new angle. I.e.:
 http://math.stackexchange.com/a/47880


I went through a brief phase where I thought about adding an angle
type (beside normint), but I felt it was overkill.
I still wonder if it's the right thing to do though... some type that
understands a circle, and making angle arithmetic work as expected.
I like my solution of not providing arithmetic operators for HSx and
LCh for now though ;)



 […]


 […]
 From which functions? Link me?
 I'd love to see more precedents.


 Yep, that's better than arguing :)
 So here are all graphics APIs I know and what they do when
 they encounter colors without alpha and need a default value:

 SDL:
 https://wiki.libsdl.org/SDL_MapRGB
 "If the specified pixel format has an alpha component it will be returned as
all 1 bits (fully opaque)."

 Allegro:
 https://github.com/liballeg/allegro5/blob/master/include/allegro5/internal/aintern_pixels.h#L59
 (No docs, just source code that defaults to 255 for alpha when
 converting a color from a bitmap with non-alpha pixel format
 to an ALLEGRO_COLOR.)

 Cairo:
 https://www.cairographics.org/manual/cairo-cairo-t.html#cairo-set-source-rgb
 "Sets the source pattern within cr to an opaque color."

 Microsoft GDI+:
 https://msdn.microsoft.com/de-de/library/windows/desktop/ms536255%28v=vs.85%29.aspx
 "The default value of the alpha component for this Color
 object is 255."

 Gnome GDK:
 https://developer.gnome.org/gdk-pixbuf/2.33/gdk-pixbuf-Utilities.html#gdk-pixbuf-add-alpha
 "[…] the alpha channel is initialized to 255 (full opacity)."

 Qt:
 http://doc.qt.io/qt-4.8/qcolor.html#alpha-blended-drawing
 "By default, the alpha-channel is set to 255 (opaque)."

 OpenGL:
 https://www.opengl.org/sdk/docs/man2/xhtml/glColor.xml
 "glColor3 variants specify new red, green, and blue values
 explicitly and set the current alpha value to 1.0 (full
 intensity) implicitly."
 (Note: The color can be queried and shows a=1.0 without
 blending operations setting it internally if needed.)

 Java (AWT):
 https://docs.oracle.com/javase/7/docs/api/java/awt/Color.html#Color%28int,%20boolean%29
 "If the hasalpha argument is false, alpha is defaulted to 255."

 Apple's Quartz does not seem to provide color space
 conversions and always requires the user to give the alpha
 value for new colors, so there is no default:
 https://developer.apple.com/library/tvos/documentation/GraphicsImaging/Reference/CGColor/index.html#//apple_ref/c/func/CGColorCreate


Touche! ;)



 One thing I noticed is that many of those strictly separate
 color spaces from alpha as concepts. For example in Quartz
 *all* color spaces have alpha. In Allegro color space
 conversions are ignorant of alpha. That begs the question
 what should happen when you convert RGBA to a HLx
 color space and back to RGBA. Can you retain the alpha value?
 CSS3 for example has HSLA colors that raise the bar a bit.


I've actually had the thought that I should remove alpha from RGB, and
instead provide an aggregate colour type where you could make RGBA by
RGB +  alpha, and each element in the aggregate would follow its own
natural arithmetic rules... again, it feels like massive overkill
though, and RGBA 8 is just way simpler for the 99% case.

That aggregate thing can appear, or be added in the future, or by
users. The library and conversions are all extensible.
Sep 13 2016
parent Marco Leise <Marco.Leise gmx.de> writes:
Am Wed, 14 Sep 2016 11:36:12 +1000
schrieb Manu via Digitalmars-d <digitalmars-d puremagic.com>:


 On 14 September 2016 at 04:34, Marco Leise via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 JavaScript's canvas works that way for
 example. I.e. the only pixel format is RGBA for simplicity's
 sake and I'm not surprised it actually draws something if I
 load it with a 24-bit graphic. ;) =20


=20
 Given this example, isn't it the job of the image loader to populate
 the image with data?2
 [=E2=80=A6]


I admit is was a constructed example. You can't load an image
directly into a HTML5 canvas. Shame on me.


 You'll notice I didn't add arithmetic operators to the HSx type ;)
 If you have HSx colors,  and want to do arithmetic, cast it to RGB first.


Now that you say it, yes I was wondering how the arithmetics
work since I couldn't find any opBinary.
=20

 I went through a brief phase where I thought about adding an angle
 type (beside normint), but I felt it was overkill.
 I still wonder if it's the right thing to do though... some type that
 understands a circle, and making angle arithmetic work as expected.


I see. I'm also wondering what equality means for a color with
floating point hue 10=C2=B0 and another with hue 730=C2=B0. I guess it
is ok the way it is now, because wrapping the values doesn't
guarantee they map to the same value either. Floating point
equality is flawed either way.

--=20
Marco
Sep 14 2016
prev sibling parent Basile B. <b2.temp gmx.com> writes:
On Tuesday, 13 September 2016 at 02:00:44 UTC, Manu wrote:

 What is the worth of storing alpha data if it's uniform 0xFF 
 anyway?


He he, that's the big problem with classic image formats. When a 
graphic is described in term of vector, fill, stroke, gradient, 
layer, etc (not to name SVG) there's no more waste.
Sep 13 2016
prev sibling next sibling parent reply Guillaume Chatelet <chatelet.guillaume gmail.com> writes:
On Monday, 12 September 2016 at 04:14:27 UTC, Manu wrote:

 I think I'm about as happy with my colour lib as I'm going to 
 be. It really needs reviews.

 I added packed-RGB support, including weird micro-float and
 shared-exponent formats.
 They're important for interacting with any real-time rendering 
 libraries.
 There is only one texture format I'm aware of that isn't 
 supported,
 and that is u7e3 floats, only available on Xbox360 ;)

 Chromatic adaptation functions are in.
 I've done a tidy/reorg pass.

 I'm not sure what else should be in the scope of this lib.

 PR: https://github.com/dlang/phobos/pull/2845
 dub: https://code.dlang.org/packages/color
 docs: 
 http://dtest.thecybershadow.net/artifact/website-04a64e024cf75be39700bebd3a50d26f6c7bd163-7185c8ec7b15c9e785880cab6d512e6f/web/library-prerelease/std/experimental/color.html

 - Manu


And also, do you handle limited (16-235) to full dynamic range 
(0-255) conversions?

It's a setting some acquisition/graphic card allow.
https://wiki.videolan.org/VSG:Video:Color_washed_out/

 From this page as well 
http://www.digitalpreservation.gov/formats/fdd/fdd000352.shtml

"Video and wide range. The ITU-R specifications support values 
for Y, Cb, and Cr that conform to what is sometimes called "video 
range," "legal range," or "studio swing" levels. Expressed in 
terms of 8-bit tonal range values, video range has a 16-235 
levels for Y and 16-240 levels for Cr and Cb. The term video 
range is used to contrast with "wide range" or "super white " 
values from 0 to 255, sometimes used when video signals are 
created using computer-based graphics applications."
Sep 12 2016
next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Mon, 12 Sep 2016 11:31:13 +0000
schrieb Guillaume Chatelet <chatelet.guillaume gmail.com>:


 On Monday, 12 September 2016 at 04:14:27 UTC, Manu wrote:

 I think I'm about as happy with my colour lib as I'm going to 
 be. It really needs reviews.

 I added packed-RGB support, including weird micro-float and
 shared-exponent formats.
 They're important for interacting with any real-time rendering 
 libraries.
 There is only one texture format I'm aware of that isn't 
 supported,
 and that is u7e3 floats, only available on Xbox360 ;)

 Chromatic adaptation functions are in.
 I've done a tidy/reorg pass.

 I'm not sure what else should be in the scope of this lib.

 PR: https://github.com/dlang/phobos/pull/2845
 dub: https://code.dlang.org/packages/color
 docs: 
 http://dtest.thecybershadow.net/artifact/website-04a64e024cf75be39700bebd3a50d26f6c7bd163-7185c8ec7b15c9e785880cab6d512e6f/web/library-prerelease/std/experimental/color.html

 - Manu  


 
 And also, do you handle limited (16-235) to full dynamic range 
 (0-255) conversions?
 
 It's a setting some acquisition/graphic card allow.
 https://wiki.videolan.org/VSG:Video:Color_washed_out/
 
  [...]


Practically all video content is encoded with the limited
range, from VHS to HDTV, IIRC and HiFi equipment uses it as
well, only computers are not so familiar with it and when
connected to a TV over HDMI one or the other may think that
the other part thinks ... well colors are often washed out,
in effect.
On the other hand this luminance compression can apply to any
color format, so it is probably best to leave it to the user
to compress colors if the output is a video file.

-- 
Marco
Sep 12 2016
prev sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 12 September 2016 at 21:31, Guillaume Chatelet via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Monday, 12 September 2016 at 04:14:27 UTC, Manu wrote:

 I think I'm about as happy with my colour lib as I'm going to be. It
 really needs reviews.

 I added packed-RGB support, including weird micro-float and
 shared-exponent formats.
 They're important for interacting with any real-time rendering libraries.
 There is only one texture format I'm aware of that isn't supported,
 and that is u7e3 floats, only available on Xbox360 ;)

 Chromatic adaptation functions are in.
 I've done a tidy/reorg pass.

 I'm not sure what else should be in the scope of this lib.

 PR: https://github.com/dlang/phobos/pull/2845
 dub: https://code.dlang.org/packages/color
 docs:
 http://dtest.thecybershadow.net/artifact/website-04a64e024cf75be39700bebd3a50d26f6c7bd163-7185c8ec7b15c9e785880cab6d512e6f/web/library-prerelease/std/experimental/color.html

 - Manu



 And also, do you handle limited (16-235) to full dynamic range (0-255)
 conversions?


Actually, I spent quite some time stressing about dynamic range.
Trouble is, there's no real format spec's relating to dynamic range,
it tends to be by convention.
I'm not sure what the API would look like, so I haven't approached that problem.
It's very easy to do the transform yourself, I don't really think it
needs to be part of the type, particularly since there's no
conventional way to express it.

Note also that I didn't do any YUV formats (this case you discuss is
usually related to older broadcast formats)... didn't know if they'd
be particularly useful to people. At least, it's not in my initial
offering.
Sep 12 2016
next sibling parent Guillaume Chatelet <chatelet.guillaume gmail.com> writes:
On Monday, 12 September 2016 at 14:12:35 UTC, Manu wrote:

 On 12 September 2016 at 21:31, Guillaume Chatelet via 
 Digitalmars-d <digitalmars-d puremagic.com> wrote:

 [...]


 Actually, I spent quite some time stressing about dynamic range.
 Trouble is, there's no real format spec's relating to dynamic 
 range,
 it tends to be by convention.
 I'm not sure what the API would look like, so I haven't 
 approached that problem.
 It's very easy to do the transform yourself, I don't really 
 think it
 needs to be part of the type, particularly since there's no
 conventional way to express it.

 Note also that I didn't do any YUV formats (this case you 
 discuss is usually related to older broadcast formats)... 
 didn't know if they'd be particularly useful to people. At 
 least, it's not in my initial offering.


Yes I agree, as you and Marco said: I think it's best to leave it 
to clients of the API.
Sep 12 2016
prev sibling parent Guillaume Piolat <first.last gmail.com> writes:
On Monday, 12 September 2016 at 14:12:35 UTC, Manu wrote:

 Note also that I didn't do any YUV formats (this case you 
 discuss is usually related to older broadcast formats)... 
 didn't know if they'd be particularly useful to people. At 
 least, it's not in my initial offering.


As someone who had to write lots of code involving such video 
color spaces, I'd say to don't bother.
- Video engineers will write their own highly-optimized version 
anyway,
- Dynamic range is indeed very often 16-235 / 16-240 but often 
with no meta-date allowing to know if this is the case,
- you often don't want clamping on each intermediate results
Sep 12 2016
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/12/2016 12:14 AM, Manu via Digitalmars-d wrote:

 I think I'm about as happy with my colour lib as I'm going to be.
 It really needs reviews.


Upon a quick look it seems to have the right amount of "notes", as 
Emperor Joseph II would say. Well balanced, nothing jumping at you. Nice 
work. Regarding Phobos suitability: One thing I'd like to hear from the 
community is the applicability of this work. I know very little about 
colors (only the RGB was familiar from HTML etc, but then there was no 
YUL), but that doesn't mean much. The library is quite isolated from the 
rest of Phobos, e.g. a stronger case would be made if there were some 
std.experimental.canvas package which uses the color package. Or at 
least a std.experimental.terminal package or something. A good case for 
inclusion would motivate the presence of the library as a catalyst for 
many user-level abstractions. -- Andrei
Sep 12 2016
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 13 September 2016 at 00:37, Andrei Alexandrescu via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 09/12/2016 12:14 AM, Manu via Digitalmars-d wrote:

 I think I'm about as happy with my colour lib as I'm going to be.
 It really needs reviews.



 Upon a quick look it seems to have the right amount of "notes", as Emperor
 Joseph II would say. Well balanced, nothing jumping at you. Nice work.


Thanks. It has taken a lot more work than I actually expected to make
this not suck!


 Regarding Phobos suitability: One thing I'd like to hear from the community
 is the applicability of this work. I know very little about colors (only the
 RGB was familiar from HTML etc, but then there was no YUL), but that doesn't
 mean much. The library is quite isolated from the rest of Phobos, e.g. a
 stronger case would be made if there were some std.experimental.canvas
 package which uses the color package. Or at least a
 std.experimental.terminal package or something. A good case for inclusion
 would motivate the presence of the library as a catalyst for many user-level
 abstractions. -- Andrei


So, the main reason I wanted to push for this in phobos is because
it's an enabler.
There is no multimedia in phobos at all. I think that's a shame, and
this is a starting point. Any conceivable image based work depends on
this as foundation. If pixel format types are phobos-supplied, then D
image libraries will naturally be compatible, as opposed to naturally
incompatible since every image library will otherwise re-implement
their own colour types (and almost always incorrectly).
The thing about colours is, they're HARD to get right, but more
importantly, most programmers have no idea that it's actually a hard
problem and won't tend to go looking for a lib for a colour type (Ie,
struct { ubyte r, g, b;}... simple! *cough*). They probably will use
it if it's in phobos though for compatibility reasons.
Image processing is super easy by contrast.

One of my first port-of-calls after this work would be a graph/plot
library... I'd use the hell out of that. I constantly want to output
data visualisations!

It also enables seamless interaction with realtime rendering api's.
Ie, if OpenGL/D3D/Vulkan bindings support phobos supplied colour
types, then image decoding libraries, and other sources of image data
become easy to render without any requirement for adaptation.
Rendering libs and image/video data source libs are almost always
separate, independent libraries. We need to maximise probability that
they are compatible out of the gate.

I didn't want to touch 'images' in this work, because there are
countless subjective designs to present those API's, but I don't think
there's many competing designs for a colour library, so it felt it was
the best and most bang-for-buck starting place overall.

Also, I think this has very high synergy with ndslice. I think ndslice
+  color is the foundation for image processing, or at least image
data transport.
Sep 12 2016
next sibling parent Adam D. Ruppe <destructionator gmail.com> writes:
On Monday, 12 September 2016 at 15:06:29 UTC, Manu wrote:

 So, the main reason I wanted to push for this in phobos is 
 because it's an enabler.


Indeed. I know there's a window library that was waiting on this 
work to try for Phobos.

Also, worth noting that the one module I import more than any 
other in my libs is.... color.d. Image load/save depends on it. 
Windowing display depends on it. Graphing depends on one or the 
other, GUI depends on windowing, terminal I managed to keep 
independent... but it is REALLY tempting to make it depend on 
color too.

It is a very useful base building block, and moreover, having it 
in Phobos means User X's graphing lib and my image libs could 
conceivably be able to interchange data with ease since we both 
use the same underlying data structure!
Sep 12 2016
prev sibling next sibling parent Edwin van Leeuwen <edder tkwsping.nl> writes:
On Monday, 12 September 2016 at 15:06:29 UTC, Manu wrote:

 One of my first port-of-calls after this work would be a 
 graph/plot library... I'd use the hell out of that. I 
 constantly want to output data visualisations!


Just for your information. ggplotd currently uses an old version 
of your library on the backend. I'm planning on porting it soon 
to the latest (dub) version.
Sep 12 2016
prev sibling next sibling parent Guillaume Chatelet <chatelet.guillaume gmail.com> writes:
On Monday, 12 September 2016 at 15:06:29 UTC, Manu wrote:

 On 13 September 2016 at 00:37, Andrei Alexandrescu via

 Regarding Phobos suitability: One thing I'd like to hear from 
 the community is the applicability of this work. I know very 
 little about colors (only the RGB was familiar from HTML etc, 
 but then there was no YUL), but that doesn't mean much. The 
 library is quite isolated from the rest of Phobos, e.g. a 
 stronger case would be made if there were some 
 std.experimental.canvas package which uses the color package. 
 Or at least a std.experimental.terminal package or something. 
 A good case for inclusion would motivate the presence of the 
 library as a catalyst for many user-level abstractions. -- 
 Andrei


 So, the main reason I wanted to push for this in phobos is 
 because
 it's an enabler.
 There is no multimedia in phobos at all. I think that's a 
 shame, and
 this is a starting point. Any conceivable image based work 
 depends on
 this as foundation. If pixel format types are phobos-supplied, 
 then D
 image libraries will naturally be compatible, as opposed to 
 naturally
 incompatible since every image library will otherwise 
 re-implement
 their own colour types (and almost always incorrectly).
 The thing about colours is, they're HARD to get right, but more
 importantly, most programmers have no idea that it's actually a 
 hard
 problem and won't tend to go looking for a lib for a colour 
 type (Ie,
 struct { ubyte r, g, b;}... simple! *cough*). They probably 
 will use
 it if it's in phobos though for compatibility reasons.
 Image processing is super easy by contrast.

 One of my first port-of-calls after this work would be a 
 graph/plot library... I'd use the hell out of that. I 
 constantly want to output data visualisations!

 It also enables seamless interaction with realtime rendering 
 api's. Ie, if OpenGL/D3D/Vulkan bindings support phobos 
 supplied colour types, then image decoding libraries, and other 
 sources of image data become easy to render without any 
 requirement for adaptation. Rendering libs and image/video data 
 source libs are almost always separate, independent libraries. 
 We need to maximise probability that they are compatible out of 
 the gate.

 I didn't want to touch 'images' in this work, because there are 
 countless subjective designs to present those API's, but I 
 don't think there's many competing designs for a colour 
 library, so it felt it was the best and most bang-for-buck 
 starting place overall.

 Also, I think this has very high synergy with ndslice. I think 
 ndslice +  color is the foundation for image processing, or at 
 least image data transport.


+1

It's simply amazing how the concept of color is well understood 
by everyone and yet utterly hard to understand in the details.

Color is a Universe in itself! Just have a look at Metamerism(1) 
(colors that look the same but aren't), Human Tetrachromacy(2) 
(some people - mostly women - perceive more colors than others). 
I'm not even scratching the surface: it connects with everything: 
Physics, Physiology, Art, Entertainement, Evolution(3)...

Having a high quality Color API in Phobos is a differentiator and 
an enabler: think Raytracer, image batch processing, scientific 
analysis, interactions with GUI or rendering APIs.
I think it's an important addition to Phobos - I might be biased 
though :-P

1. https://en.wikipedia.org/wiki/Metamerism_(color)
2. https://en.wikipedia.org/wiki/Tetrachromacy#Humans
3. https://www.youtube.com/watch?v=qrKZBh8BL_U
Sep 12 2016
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/12/2016 8:06 AM, Manu via Digitalmars-d wrote:

 So, the main reason I wanted to push for this in phobos is because
 it's an enabler.
 There is no multimedia in phobos at all. I think that's a shame, and
 this is a starting point. Any conceivable image based work depends on
 this as foundation. If pixel format types are phobos-supplied, then D
 image libraries will naturally be compatible, as opposed to naturally
 incompatible since every image library will otherwise re-implement
 their own colour types (and almost always incorrectly).
 The thing about colours is, they're HARD to get right, but more
 importantly, most programmers have no idea that it's actually a hard
 problem and won't tend to go looking for a lib for a colour type (Ie,
 struct { ubyte r, g, b;}... simple! *cough*). They probably will use
 it if it's in phobos though for compatibility reasons.
 Image processing is super easy by contrast.

 One of my first port-of-calls after this work would be a graph/plot
 library... I'd use the hell out of that. I constantly want to output
 data visualisations!

 It also enables seamless interaction with realtime rendering api's.
 Ie, if OpenGL/D3D/Vulkan bindings support phobos supplied colour
 types, then image decoding libraries, and other sources of image data
 become easy to render without any requirement for adaptation.
 Rendering libs and image/video data source libs are almost always
 separate, independent libraries. We need to maximise probability that
 they are compatible out of the gate.

 I didn't want to touch 'images' in this work, because there are
 countless subjective designs to present those API's, but I don't think
 there's many competing designs for a colour library, so it felt it was
 the best and most bang-for-buck starting place overall.

 Also, I think this has very high synergy with ndslice. I think ndslice
 +  color is the foundation for image processing, or at least image
 data transport.


Thanks for the rationale. I'm glad to see you're planning follow-on libraries! 
I'd use a graph/plot library myself. Some of the rationale above should go into 
the documentation for the library.
Sep 12 2016
parent jmh530 <john.michael.hall gmail.com> writes:
On Monday, 12 September 2016 at 22:55:01 UTC, Walter Bright wrote:

 Thanks for the rationale. I'm glad to see you're planning 
 follow-on libraries! I'd use a graph/plot library myself. Some 
 of the rationale above should go into the documentation for the 
 library.


You might try
https://github.com/BlackEdder/ggplotd
Sep 12 2016
prev sibling next sibling parent reply Guillaume Piolat <first.last gmail.com> writes:
On Monday, 12 September 2016 at 04:14:27 UTC, Manu wrote:

 I think I'm about as happy with my colour lib as I'm going to 
 be. It really needs reviews.


 - Manu


My thoughts:
- I've wanted a function like colorFromString many times. It's 
especially nice with the added #RGBA and #RRGGBBAA syntax that 
eg. SVG lacks.
- The knowledge encoded in this library is great in itself.
- This could form the basis of the next-gen DbI image library so 
it is much needed.
=> I would certainly vote for inclusion.
Sep 12 2016
parent reply Edwin van Leeuwen <edder tkwsping.nl> writes:
On Monday, 12 September 2016 at 19:55:57 UTC, Guillaume Piolat 
wrote:

 - I've wanted a function like colorFromString many times. It's 
 especially nice with the added #RGBA and #RRGGBBAA syntax that 
 eg. SVG lacks.


What happens when the string is invalid? Does it throw an error?
Sep 12 2016
next sibling parent Guillaume Piolat <first.last gmail.com> writes:
On Monday, 12 September 2016 at 19:59:58 UTC, Edwin van Leeuwen 
wrote:

 On Monday, 12 September 2016 at 19:55:57 UTC, Guillaume Piolat 
 wrote:

 - I've wanted a function like colorFromString many times. It's 
 especially nice with the added #RGBA and #RRGGBBAA syntax that 
 eg. SVG lacks.


 What happens when the string is invalid? Does it throw an error?


It throws an Exception, like it should.
Sep 12 2016
prev sibling parent reply John Colvin <john.loughran.colvin gmail.com> writes:
On Monday, 12 September 2016 at 19:59:58 UTC, Edwin van Leeuwen 
wrote:

 On Monday, 12 September 2016 at 19:55:57 UTC, Guillaume Piolat 
 wrote:

 - I've wanted a function like colorFromString many times. It's 
 especially nice with the added #RGBA and #RRGGBBAA syntax that 
 eg. SVG lacks.


 What happens when the string is invalid? Does it throw an error?


It should be integrated with std.conv.parse and std.conv.to and 
follow their conventions.
Sep 12 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/12/2016 2:08 PM, John Colvin wrote:

 On Monday, 12 September 2016 at 19:59:58 UTC, Edwin van Leeuwen wrote:

 On Monday, 12 September 2016 at 19:55:57 UTC, Guillaume Piolat wrote:

 - I've wanted a function like colorFromString many times. It's especially
 nice with the added #RGBA and #RRGGBBAA syntax that eg. SVG lacks.


 What happens when the string is invalid? Does it throw an error?


 It should be integrated with std.conv.parse and std.conv.to and follow their
 conventions.


I'm not so sure. Follow the conventions, ok, but integrating them in - they are 
already large and complex.
Sep 12 2016
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 13 September 2016 at 12:05, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 9/12/2016 2:08 PM, John Colvin wrote:

 On Monday, 12 September 2016 at 19:59:58 UTC, Edwin van Leeuwen wrote:

 On Monday, 12 September 2016 at 19:55:57 UTC, Guillaume Piolat wrote:

 - I've wanted a function like colorFromString many times. It's
 especially
 nice with the added #RGBA and #RRGGBBAA syntax that eg. SVG lacks.



 What happens when the string is invalid? Does it throw an error?



 It should be integrated with std.conv.parse and std.conv.to and follow
 their
 conventions.



 I'm not so sure. Follow the conventions, ok, but integrating them in - they
 are already large and complex.


std.conv.to  seems to 'just work' for color type conversion.
I'd like to support string -> color conversion via std.conv.to; ie:
auto c = "#FFFF00".to!RGB8;
Is there a simple way to do that?
Sep 12 2016
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/12/16 10:50 PM, Manu via Digitalmars-d wrote:

 std.conv.to  seems to 'just work' for color type conversion.
 I'd like to support string -> color conversion via std.conv.to; ie:
 auto c = "#FFFF00".to!RGB8;
 Is there a simple way to do that?


That was the purpose of toImpl, which I recall has been recently 
removed... -- Andrei
Sep 13 2016
prev sibling parent reply ketmar <ketmar ketmar.no-ip.org> writes:
On Tuesday, 13 September 2016 at 02:50:02 UTC, Manu wrote:

 std.conv.to  seems to 'just work' for color type conversion.
 I'd like to support string -> color conversion via std.conv.to; 
 ie:
 auto c = "#FFFF00".to!RGB8;
 Is there a simple way to do that?


yes. just define a ctor that accepts string, and you're there. it 
works both for classes and structs.
Sep 13 2016
parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 13 September 2016 at 20:19, ketmar via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Tuesday, 13 September 2016 at 02:50:02 UTC, Manu wrote:

 std.conv.to  seems to 'just work' for color type conversion.
 I'd like to support string -> color conversion via std.conv.to; ie:
 auto c = "#FFFF00".to!RGB8;
 Is there a simple way to do that?



 yes. just define a ctor that accepts string, and you're there. it works both
 for classes and structs.


Oh... it already works! :P
Sep 13 2016
prev sibling next sibling parent reply Random D user <no email.com> writes:
On Monday, 12 September 2016 at 04:14:27 UTC, Manu wrote:

 I think I'm about as happy with my colour lib as I'm going to 
 be. It really needs reviews.

 - Manu


Hi. I'm just a random forum lurker, but here's my feedback.

It needs more docs/examples for the basic usage cases (i.e. how 
to skip docs and start working).

Now at first glance it looks like working with RGBA (most common 
case (with srgb)) is complicated and while reading the docs I 
kind of felt like I'd just write my own.

Few basic things I'd be interested in (examples):
1. How to pack/unpack color into uint/rgba channels. Also can I 
swizzle channels directly? How would I extract a single channel 
from color?
2. Can I just cast an array of bytes into a RGBA array and 
operate on that?
3. How to scale color channels e.g. 5 bits to 8 bits and vice 
versa. How to convert color formats? Can I combine 
scale/conversion with single channel or channel swizzling?
4. Which way are RGBA format names defined ("byte array order" r 
is first byte or "uint order" r is in high bits)?
5. Is it the same for packed rgba formats?
I'm not sure if packedrgb is a necessary distinction. Aren't they 
all kind of packed (small structs)? There's no guarantee that 1 
byte is not just a part of a bigger field.

Btw. Your RGBA8 example is a bit weird:
static assert(RGBA8("0x908000FF") == RGBA8(0x80,0x00,0xFF,0x90));
I'd assume RGBA8(r,g,b,a) would be either "uint order" r(0x90), 
g(0x80), b(0x00), a(0xFF) or "byte array order" (le) r(0xFF), 
g(0x00), b(0x80), a(0x90)

6. Perhaps you should define all the usual color formats, since 
everybody and their cat has their own preference for rgba channel 
orders.
In my experience all these 4 variants for RGBA8888 are pretty 
common:
RGBA8888
BGRA8888
ARGB8888
ABGR8888
and their X and 24bit variants.

For 16 bits fairly common are:
RGB565 and RGBA5551, also sometimes you see one of RGBA4444 
permutations (like RGBA8 above).
Also L16 is pretty common for heightfields or other data 
images/textures.

I guess there are also planar formats, but I think those are more 
of a Image problem space.

Just as a completely random idea - How about aliases for HDR 
formats like HDR10 and Dolby Vision? Kind of looks like they're 
just combinations what you already have.
Sep 12 2016
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 13 September 2016 at 06:29, Random D user via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Monday, 12 September 2016 at 04:14:27 UTC, Manu wrote:

 I think I'm about as happy with my colour lib as I'm going to be. It
 really needs reviews.

 - Manu



 Hi. I'm just a random forum lurker, but here's my feedback.

 It needs more docs/examples for the basic usage cases (i.e. how to skip docs
 and start working).

 Now at first glance it looks like working with RGBA (most common case (with
 srgb)) is complicated and while reading the docs I kind of felt like I'd
 just write my own.


You'll get it wrong though!

Can you describe what  you  perceive to be hard?

  import std.experimental.color;

  RGBA8 a(0xFF808000);
  RGBA8 b("GoldenRodYellow");
  auto r = a + b; // do normal operations

Or:
  auto image = new RGBA8[height][width];
  image[x][y] = RGBA8("Aquamarine");



 Few basic things I'd be interested in (examples):
 1. How to pack/unpack color into uint/rgba channels.


auto c  = RGBA8(r, g, b, a);
uint r = c.r.value; // 'r' is a 'NormalizedInt", so you need '.value'
to get the raw integer value
float g = cast(float)c.g; // cast to float knows about the normalized 0-1 range


 Also can I swizzle channels directly?


I could add something like:
  auto brg =  c.swizzle!"brg";

The result would be strongly typed to the new arrangement.


 How would I extract a single channel from color?


uint r = c.r.value; // like this?


 2. Can I just cast an array of bytes into a RGBA array and operate on that?


Yes. Of course, if the image library returned an array of colors, you
wouldn't have an array of raw bytes in the first place.


 3. How to scale color channels e.g. 5 bits to 8 bits and vice versa. How to
 convert color formats? Can I combine scale/conversion with single channel or
 channel swizzling?


alias CustomPackedColor = PackedRGB!("BGR_5_6_5", ubyte); // <- ordered bgr
CustomPackedColor[] image = loadPackedImage();

auto unpacked = cast(RGB8)image[0]; // unpack and swizzle

I'm not sure what you mean by "with single channel"?


 4. Which way are RGBA format names defined ("byte array order" r is first
 byte or "uint order" r is in high bits)?


I describe formats in byte (bit) order. It's the only reasonable way
that works outside of a uint.


 5. Is it the same for packed rgba formats?


Yes.


 I'm not sure if packedrgb is a necessary distinction. Aren't they all kind
 of packed (small structs)? There's no guarantee that 1 byte is not just a
 part of a bigger field.


RGBA8 has 4 ubytes, and you can operate on them.
  ubyte a, b; a += b; // <- is valid D code
Unpacked types can do maths. Packed types can't do maths, they need to
be unpacked before you can do anything, and you don't generally want
the result to be immediately packed again.
If I allowed maths on PackedRGB for instance, it would be expected
that the result is re-packed at the end of every operation, and that
would be useless.

PackedRGB is for storage, not for doing work. This is why they're distinct.


 Btw. Your RGBA8 example is a bit weird:
 static assert(RGBA8("0x908000FF") == RGBA8(0x80,0x00,0xFF,0x90));
 I'd assume RGBA8(r,g,b,a) would be either "uint order" r(0x90), g(0x80),
 b(0x00), a(0xFF) or "byte array order" (le) r(0xFF), g(0x00), b(0x80),
 a(0x90)


I'm showing that parsing colors from string's follow the web
convention. I can't redefine how colours are expressed on the web for
decades.
I think you've just highlighted that I need to remove the '0x'

be clear.


 6. Perhaps you should define all the usual color formats, since everybody
 and their cat has their own preference for rgba channel orders.
 In my experience all these 4 variants for RGBA8888 are pretty common:
 RGBA8888
 BGRA8888
 ARGB8888
 ABGR8888
 and their X and 24bit variants.


I don't think they're common at all. Only 2 are common (one of which
is deprecated), the other 2 are very niche.
I was trying to balance a tasteful amount of instantiations. I don't
really want to provide an instantiation of every single realtime
format I've ever encountered out of the box.



 For 16 bits fairly common are:
 RGB565 and RGBA5551, also sometimes you see one of RGBA4444 permutations
 (like RGBA8 above).


Nobody uses 16bit textures anymore. Compressed textures are both MUCH
smaller, and generally higher quality.


 Also L16 is pretty common for heightfields or other data images/textures.


I could probably put that one.


 I guess there are also planar formats, but I think those are more of a Image
 problem space.


I think if people want to provide instantiations for convenience, it
should be in the libs that interact with those formats.

For instance, I imagine in OpenGL binding there might be:
  alias C = ColorFor!GL_R11F_G11F_B10F; // <- instantiation of the
proper type for the GL enum.

Most people really just want RGB8, or RGBA8. If you're interacting
with PackedRGB, you're already in specialist territory.



 Just as a completely random idea - How about aliases for HDR formats like
 HDR10 and Dolby Vision? Kind of looks like they're just combinations what
 you already have.


This is very specialist. Instantiations are expensive. If they're not
used, it's a waste of compiler effort.
Sep 12 2016
next sibling parent reply John Colvin <john.loughran.colvin gmail.com> writes:
On Tuesday, 13 September 2016 at 01:05:56 UTC, Manu wrote:

 Also can I swizzle channels directly?


 I could add something like:
   auto brg =  c.swizzle!"brg";

 The result would be strongly typed to the new arrangement.


Perfect use-case for opDispatch like in gl3n.
Sep 13 2016
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 13 September 2016 at 17:47, John Colvin via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Tuesday, 13 September 2016 at 01:05:56 UTC, Manu wrote:

 Also can I swizzle channels directly?



 I could add something like:
   auto brg =  c.swizzle!"brg";

 The result would be strongly typed to the new arrangement.



 Perfect use-case for opDispatch like in gl3n.


One trouble with arbitrary swizzling is that people often want to
write expressions like "c.xyzy", or whatever, but repeating
components... and that's not something my colours can do. It's useful
in realtime code, but it doesn't mean anything, and you can't
interpret that value as a colour anymore after you do that.
This sort of thing is used when you're not actually storing colours in
textures, but instead just some arbitrary data. Don't use a colour
type for that, use a vector type instead.
What that sort of swizzling really is, are vector operations, not
colour operations. Maybe I could add an API to populate a vector from
the components of colours, in-order... but then we don't actually have
linear algebra functions in phobos either! So colour->vectors and
arbitrary swizzling is probably not something we need immediately.

In my lib, colours are colours. If you have `BGR8 x` and `RGB8 y`, and
add them, you don't get x.b+y.r, x.g+y.g, x.r+y.b... that's not a
colour operation, that's an element-wise vector operation.
Sep 13 2016
next sibling parent Guillaume Piolat <first.last gmail.com> writes:
On Tuesday, 13 September 2016 at 09:31:53 UTC, Manu wrote:

 In my lib, colours are colours. If you have `BGR8 x` and `RGB8 
 y`, and add them, you don't get x.b+y.r, x.g+y.g, x.r+y.b... 
 that's not a colour operation, that's an element-wise vector 
 operation.


I think swizzling is way overrated, especially swizzling 
assignment.


Which is the most readable?

     color.rb = color.br;     // swizzling
     swap(color.r, color.b);  // no swizzling

     auto color = RGBA8(c.rgb, 255);         // swizzling
     auto color = RGBA8(c.r, c.g, c.b, 255); // no swizzling

And finally:

     auto rb = color.rgrb; // When do you ever need that?
Sep 13 2016
prev sibling parent reply John Colvin <john.loughran.colvin gmail.com> writes:
On Tuesday, 13 September 2016 at 09:31:53 UTC, Manu wrote:

 On 13 September 2016 at 17:47, John Colvin via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:

 On Tuesday, 13 September 2016 at 01:05:56 UTC, Manu wrote:

 Also can I swizzle channels directly?



 I could add something like:
   auto brg =  c.swizzle!"brg";

 The result would be strongly typed to the new arrangement.



 Perfect use-case for opDispatch like in gl3n.


 One trouble with arbitrary swizzling is that people often want 
 to
 write expressions like "c.xyzy", or whatever, but repeating
 components... and that's not something my colours can do. It's 
 useful
 in realtime code, but it doesn't mean anything, and you can't
 interpret that value as a colour anymore after you do that.
 This sort of thing is used when you're not actually storing 
 colours in
 textures, but instead just some arbitrary data. Don't use a 
 colour
 type for that, use a vector type instead.
 What that sort of swizzling really is, are vector operations, 
 not
 colour operations. Maybe I could add an API to populate a 
 vector from
 the components of colours, in-order... but then we don't 
 actually have
 linear algebra functions in phobos either! So colour->vectors 
 and
 arbitrary swizzling is probably not something we need 
 immediately.

 In my lib, colours are colours. If you have `BGR8 x` and `RGB8 
 y`, and add them, you don't get x.b+y.r, x.g+y.g, x.r+y.b... 
 that's not a colour operation, that's an element-wise vector 
 operation.


Fair enough, you know much better than me how useful it would be. 
I was just suggesting that if you do support some sorts of 
swizzling then opDispatch would allow you to avoid users having 
to use strings. It would be as simple to implement as `alias 
opDispatch = swizzle;` given the swizzle function you were using 
before.
Sep 13 2016
parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 14 September 2016 at 00:56, John Colvin via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Tuesday, 13 September 2016 at 09:31:53 UTC, Manu wrote:

 On 13 September 2016 at 17:47, John Colvin via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:

 On Tuesday, 13 September 2016 at 01:05:56 UTC, Manu wrote:

 Also can I swizzle channels directly?




 I could add something like:
   auto brg =  c.swizzle!"brg";

 The result would be strongly typed to the new arrangement.




 Perfect use-case for opDispatch like in gl3n.



 One trouble with arbitrary swizzling is that people often want to
 write expressions like "c.xyzy", or whatever, but repeating
 components... and that's not something my colours can do. It's useful
 in realtime code, but it doesn't mean anything, and you can't
 interpret that value as a colour anymore after you do that.
 This sort of thing is used when you're not actually storing colours in
 textures, but instead just some arbitrary data. Don't use a colour
 type for that, use a vector type instead.
 What that sort of swizzling really is, are vector operations, not
 colour operations. Maybe I could add an API to populate a vector from
 the components of colours, in-order... but then we don't actually have
 linear algebra functions in phobos either! So colour->vectors and
 arbitrary swizzling is probably not something we need immediately.

 In my lib, colours are colours. If you have `BGR8 x` and `RGB8 y`, and add
 them, you don't get x.b+y.r, x.g+y.g, x.r+y.b... that's not a colour
 operation, that's an element-wise vector operation.



 Fair enough, you know much better than me how useful it would be. I was just
 suggesting that if you do support some sorts of swizzling then opDispatch
 would allow you to avoid users having to use strings. It would be as simple
 to implement as `alias opDispatch = swizzle;` given the swizzle function you
 were using before.


Oh, it would be super-useful! Just that it's not a colour operation,
its liner algebra... I think it's a job for another lib. Getting a
colour as a vector is all that's needed.
Sep 13 2016
prev sibling parent reply Random D user <no email.com> writes:
On Tuesday, 13 September 2016 at 01:05:56 UTC, Manu wrote:

 Can you describe what  you  perceive to be hard?


Well, I just skimmed through the docs and I didn't look at the 
code, so that sense it was an "honest" view for phobos proposal. 
Also I was trying to convey that based on the docs it "looks 
hard", although I was suspecting that it isn't really.

So the list of things was a list of examples that I would've 
wanted to see in the docs. Maybe in an overview or "here's how 
you use it" page. I can read the details if I really need them, 
but I think it's important to have a cook book examples for quick 
start.

In general, I think basics should be dead simple (even over 
simplified for the very basic case), but the API and the docs 
should provide me layers of increasing detail and sophistication, 
which I can study if (or when) I really need control and care 
about the details.


 Few basic things I'd be interested in (examples):
 1. How to pack/unpack color into uint/rgba channels.


 auto c  = RGBA8(r, g, b, a);
 uint r = c.r.value; // 'r' is a 'NormalizedInt", so you need 
 '.value'
 to get the raw integer value
 float g = cast(float)c.g; // cast to float knows about the 
 normalized 0-1 range


Let's put these in the docs.


 Also can I swizzle channels directly?


 I could add something like:
   auto brg =  c.swizzle!"brg";


It's just a nicety. I guess I could just define exact color 
formats and convert (let the lib do the rest), or just unpack and 
repack.

Also the docs might want to mention which way is the 
preferred/designed way.


 2. Can I just cast an array of bytes into a RGBA array and 
 operate on that?


 Yes. Of course, if the image library returned an array of 
 colors, you wouldn't have an array of raw bytes in the first 
 place.


Cool. I just wanted to make sure something like this possible.

Also should be in the docs as an example, so that people don't 
have to try this out themselves.
Or search through github for details.


 3. How to scale color channels e.g. 5 bits to 8 bits and vice 
 versa. How to convert color formats? Can I combine 
 scale/conversion with single channel or channel swizzling?


 alias CustomPackedColor = PackedRGB!("BGR_5_6_5", ubyte); // <- 
 ordered bgr
 CustomPackedColor[] image = loadPackedImage();

 auto unpacked = cast(RGB8)image[0]; // unpack and swizzle

 I'm not sure what you mean by "with single channel"?


For example if you first unpack RGBA_uint32 into R,G,B,A then you 
want to take G (or just swizzle color.g) and scale that to 5 bits 
or 16 bits for what ever use case you might have.

Looks to me like this can be handled with something like 
PackedRGB!("G_5") and then do color conversion into that.


 4. Which way are RGBA format names defined ("byte array order" 
 r is first byte or "uint order" r is in high bits)?


 I describe formats in byte (bit) order. It's the only 
 reasonable way
 that works outside of a uint.


Agreed. This should be in the docs, so that it's clear to users. 
(especially, since RGBA8888 is the common case and it does fit 
into uint)


 I'm showing that parsing colors from string's follow the web
 convention. I can't redefine how colours are expressed on the 
 web for
 decades.
 I think you've just highlighted that I need to remove the '0x'

 should
 be clear.


Right. I was interpreting 0x........ as c hex literal. You should 
document that it's following the web convention.


 6. Perhaps you should define all the usual color formats, 
 since everybody
 and their cat has their own preference for rgba channel orders.
 In my experience all these 4 variants for RGBA8888 are pretty 
 common:
 RGBA8888
 BGRA8888
 ARGB8888
 ABGR8888
 and their X and 24bit variants.


 I don't think they're common at all. Only 2 are common (one of 
 which
 is deprecated), the other 2 are very niche.


Deprecated by who? Shouldn't phobos grade lib include all 
reasonable platforms?

I agree that you probably don't see too much variation within 
windows APIs, images (BMP etc.) or with D3D GPU textures (they're 
compressed anyway and asset pipelines to do the conversions 
beforehand), but I was more of thinking image libraries and the 
numerous, sometimes old and quirky, image formats. Or perhaps 
someone wants to implement a software blitter for their favorite 
toy device/embedded project.


 I was trying to balance a tasteful amount of instantiations. I 
 don't
 really want to provide an instantiation of every single realtime
 format I've ever encountered out of the box.


Seems reasonable.


 For 16 bits fairly common are:
 RGB565 and RGBA5551, also sometimes you see one of RGBA4444 
 permutations
 (like RGBA8 above).


 Nobody uses 16bit textures anymore. Compressed textures are 
 both MUCH smaller, and generally higher quality.


Sure, agreed. These are not too useful as GPU textures these days 
(even on mobile), but if you do software 2d, load older image 
formats (image viewer etc.) or something even more specialized, 
these are pretty useful.

I guess I was going for comprehensive (within reason), where as 
you were going for "90% of 2016 colors/image data", which is fine.

Anyways, it would be nice to see color (and hopefully image + 
simple blitter) in phobos.
There's no need to write the same convert/copy/stretch/blend loop 
over and over again. And cpu-side code is nice to have for quick 
prototyping and small-scale work.


 Just as a completely random idea - How about aliases for HDR 
 formats like HDR10 and Dolby Vision? Kind of looks like 
 they're just combinations what you already have.


 This is very specialist. Instantiations are expensive. If 
 they're not used, it's a waste of compiler effort.


HDR TVs and displays are coming fast and with that I'm sure HDR 
editing and photography will be common, which of course means HDR 
color formats and apps. But these can be added later.
Sep 13 2016
parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 14 September 2016 at 04:10, Random D user via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 In general, I think basics should be dead simple (even over simplified for
 the very basic case)


I think they are:

  import std.experimental.color;
  RGB8 color("red");

I don't think i's possible to make it simpler than that?



 but the API and the docs should provide me layers of
 increasing detail and sophistication, which I can study if (or when) I
 really need control and care about the details.


I'll work on the docs some more.



 Deprecated by who? Shouldn't phobos grade lib include all reasonable
 platforms?


They're all comprehensively supported, I just didn't pre-instantiate
every possible format imaginable. That would be devastating to compile
times, and have absolutely no value.

BGR really just exists because WINAPI. Microsoft haven't recommended
people write GDI based software for decades.
Look at DX10(,11,12), BGR isn't even supported in modern DirectX.
Microsoft left BGR to die years ago (and good riddance).
That said, there is an instantiation for BGR in package.d... the
reversed versions (ie, alpha first) are completely unusual; just some
big-endian games consoles.

Modern formats have the channels in order; RGB, this is true for all
data types, ubyte, short, float, etc.



 I agree that you probably don't see too much variation within windows APIs,
 images (BMP etc.) or with D3D GPU textures (they're compressed anyway and
 asset pipelines to do the conversions beforehand), but I was more of
 thinking image libraries and the numerous, sometimes old and quirky, image
 formats. Or perhaps someone wants to implement a software blitter for their
 favorite toy device/embedded project.


"favorite toy device/embedded project" <- this is the definition of
specialist task. I don't mind if that person has to type 'alias' at
the top of their app.



 For 16 bits fairly common are:
 RGB565 and RGBA5551, also sometimes you see one of RGBA4444 permutations
 (like RGBA8 above).



 Nobody uses 16bit textures anymore. Compressed textures are both MUCH
 smaller, and generally higher quality.



 Sure, agreed. These are not too useful as GPU textures these days (even on
 mobile), but if you do software 2d, load older image formats (image viewer
 etc.) or something even more specialized, these are pretty useful.

 I guess I was going for comprehensive (within reason), where as you were
 going for "90% of 2016 colors/image data", which is fine.


There is comprehensive support for all these formats... I just didn't
pre-instantiate the templates.
It's not hard to do at the top of your program:
  alias MyRGBType = RGB!(...details...);



 Anyways, it would be nice to see color (and hopefully image + simple
 blitter) in phobos.


I think that'll come. This is a pre-requisite.



 There's no need to write the same convert/copy/stretch/blend loop over and
 over again. And cpu-side code is nice to have for quick prototyping and
 small-scale work.


Sure.



 Just as a completely random idea - How about aliases for HDR formats like
 HDR10 and Dolby Vision? Kind of looks like they're just combinations what
 you already have.



 This is very specialist. Instantiations are expensive. If they're not
 used, it's a waste of compiler effort.



 HDR TVs and displays are coming fast and with that I'm sure HDR editing and
 photography will be common, which of course means HDR color formats and
 apps. But these can be added later.


There is already support for HDR.
Use a floating point RGB with UHDTV colour space: alias HDR =
RGB!("rgb", float, linear(?), RGBColorSpace.UHDTV);
CompressedRGB also has comprehensive support for all the HDR formats
I've ever encountered.

My point was, I don't want to provide instantiations for these
types... most people don't need it. People writing HDR rendering
software will instantiate the types they need themselves. It's very
specialist work.
Sep 13 2016
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/11/2016 9:14 PM, Manu via Digitalmars-d wrote:

 I think I'm about as happy with my colour lib as I'm going to be.
 It really needs reviews.


I have zero expertise in color programming, and have no idea what a color 
library might do or what it would be used for.

I looked at the front page of the documentation. I would like to see some sort 
of explanation of who, what, where, why and how of a color library. Like what
is 
a color space, etc. Maybe add some links to explanatory Wikipedia pages?

"Set of colors defined by X11, adopted by the W3C, SVG, and other popular 
libraries."

Would be a great place to add clickable links for the relevant X11, W3C, SVG, 
etc. specification pages.

It really, really needs a "References:" section, with links like:

     https://en.wikipedia.org/wiki/RGB_color_model

Are there any definitive books on the topic?

What is a BGR color type? What is a UV color type? What is a UVW color type? 
What is an alpha color type? What is luminance?

"This module implements HSV, HSL, HSI, HCY, HWB, HCG color types."
"angular color spaces"

Some references to what these are would be nice.

Contrast with:

     http://dlang.org/phobos/std_regex.html

where general information is given and links for more are provided (although 
that is weak and could be improved).

-------

My last comment is what can I do with a color library that would not require 
some additional library?
Sep 12 2016
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/12/16 6:51 PM, Walter Bright wrote:

 My last comment is what can I do with a color library that would not
 require some additional library?


Great question and well asked. It was mine as well but I could not put 
it as succinctly.  -- Andrei
Sep 12 2016
prev sibling next sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 13 September 2016 at 08:51, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 9/11/2016 9:14 PM, Manu via Digitalmars-d wrote:

 I think I'm about as happy with my colour lib as I'm going to be.
 It really needs reviews.



 I have zero expertise in color programming, and have no idea what a color
 library might do or what it would be used for.

 I looked at the front page of the documentation. I would like to see some
 sort of explanation of who, what, where, why and how of a color library.
 Like what is a color space, etc. Maybe add some links to explanatory
 Wikipedia pages?

 "Set of colors defined by X11, adopted by the W3C, SVG, and other popular
 libraries."

 Would be a great place to add clickable links for the relevant X11, W3C,
 SVG, etc. specification pages.

 It really, really needs a "References:" section, with links like:

     https://en.wikipedia.org/wiki/RGB_color_model

 Are there any definitive books on the topic?

 What is a BGR color type? What is a UV color type? What is a UVW color type?
 What is an alpha color type? What is luminance?

 "This module implements HSV, HSL, HSI, HCY, HWB, HCG color types."
 "angular color spaces"

 Some references to what these are would be nice.

 Contrast with:

     http://dlang.org/phobos/std_regex.html

 where general information is given and links for more are provided (although
 that is weak and could be improved).


Okay. I'll work on this feedback.



 My last comment is what can I do with a color library that would not require
 some additional library?


With color, ndslice, and std.algorithm, you can do quite a lot of
useful image processing already.
But my point before was that I see this primarily as an enabler. We
really need a graph/plot library for instance! Also, multimedia
libraries will all become naturally compatible when the primitive
working type is defined in phobos.
Sep 12 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/12/2016 6:11 PM, Manu via Digitalmars-d wrote:

 Okay. I'll work on this feedback.


Wonderful!



 My last comment is what can I do with a color library that would not require
 some additional library?


 With color, ndslice, and std.algorithm, you can do quite a lot of
 useful image processing already.


Ok, but I have no idea how. Perhaps include something as an example? Phobos in 
general lacks short, pithy, and *useful* examples. Such can go a long way 
towards selling the color library to users.



 But my point before was that I see this primarily as an enabler. We
 really need a graph/plot library for instance! Also, multimedia
 libraries will all become naturally compatible when the primitive
 working type is defined in phobos.


I had read your post on this topic after making my post. I think it's a strong 
answer.
Sep 12 2016
parent Walter Bright <newshound2 digitalmars.com> writes:
On 9/12/2016 7:03 PM, Walter Bright wrote:

 On 9/12/2016 6:11 PM, Manu via Digitalmars-d wrote:

 Okay. I'll work on this feedback.


 Wonderful!


BTW, linking to authoritative reference material has the effect of reassuring 
the user that the library is designed by someone who knows his stuff. It has
the 
same effect as notes and references appendices have in any good technical book.
Sep 12 2016
prev sibling next sibling parent Andrea Fontana <nospam example.com> writes:
On Monday, 12 September 2016 at 22:51:14 UTC, Walter Bright wrote:

 My last comment is what can I do with a color library that 
 would not require some additional library?


I think it's easy to do some image processing using netbpm [1] as 
format.
Reading/Writing images doesn't require any other library.

And using ndslices a lot of things can be done :)

Andrea

[1] https://en.wikipedia.org/wiki/Netpbm_format
Sep 13 2016
prev sibling parent Guillaume Piolat <first.last gmail.com> writes:
On Monday, 12 September 2016 at 22:51:14 UTC, Walter Bright wrote:

 My last comment is what can I do with a color library that 
 would not require some additional library?


Having correct conversions.

Converting sRGB to linear RGB is typically not done correctly, 
pow isn't enough.
It also dissipates most misunderstanding about color space (yes 
Lab also are dependent on the color point, it was something I 
wasn't sure about and this library says it clearly).

Right now I rely on the OS for Apple RGB to sRGB conversion, this 
library would allow me to do it without using a system call.

Other uses I can imagine:
- Downsampling an image preserving colors (compressed RGB isn't 
necessarily the right sep to do those)
- Perceptually correct color gradients. In font rendering,
- HSV is very useful for colouring stuff in video games, it's 
what the color selection does in painter-programs like GIMP. So 
lots of library out there have a HSV <-> RGB conversions
- converting to packed RGB is useful for programs using GPUs. 
Same reason than half-float support is useful.
Sep 13 2016
prev sibling next sibling parent reply Meta <jared771 gmail.com> writes:
On Monday, 12 September 2016 at 04:14:27 UTC, Manu wrote:

 I think I'm about as happy with my colour lib as I'm going to 
 be. It really needs reviews.

 I added packed-RGB support, including weird micro-float and
 shared-exponent formats.
 They're important for interacting with any real-time rendering 
 libraries.
 There is only one texture format I'm aware of that isn't 
 supported,
 and that is u7e3 floats, only available on Xbox360 ;)

 Chromatic adaptation functions are in.
 I've done a tidy/reorg pass.

 I'm not sure what else should be in the scope of this lib.

 PR: https://github.com/dlang/phobos/pull/2845
 dub: https://code.dlang.org/packages/color
 docs: 
 http://dtest.thecybershadow.net/artifact/website-04a64e024cf75be39700bebd3a50d26f6c7bd163-7185c8ec7b15c9e785880cab6d512e6f/web/library-prerelease/std/experimental/color.html

 - Manu


Looks like a nice and simple module. My only complaint is that 
DDOC sections such as Returns, Args, etc. don't seem to be used 
at all. Also maybe a bit more detailed of an overview in 
package.d, which I assume is the entry point for the 
documentation.
Sep 14 2016
next sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 14 September 2016 at 22:47, Meta via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On Monday, 12 September 2016 at 04:14:27 UTC, Manu wrote:

 I think I'm about as happy with my colour lib as I'm going to be. It
 really needs reviews.

 I added packed-RGB support, including weird micro-float and
 shared-exponent formats.
 They're important for interacting with any real-time rendering libraries.
 There is only one texture format I'm aware of that isn't supported,
 and that is u7e3 floats, only available on Xbox360 ;)

 Chromatic adaptation functions are in.
 I've done a tidy/reorg pass.

 I'm not sure what else should be in the scope of this lib.

 PR: https://github.com/dlang/phobos/pull/2845
 dub: https://code.dlang.org/packages/color
 docs:
 http://dtest.thecybershadow.net/artifact/website-04a64e024cf75be39700bebd3a50d26f6c7bd163-7185c8ec7b15c9e785880cab6d512e6f/web/library-prerelease/std/experimental/color.html

 - Manu



 Looks like a nice and simple module. My only complaint is that DDOC sections
 such as Returns, Args, etc. don't seem to be used at all. Also maybe a bit
 more detailed of an overview in package.d, which I assume is the entry point
 for the documentation.


Cheers.
Yeah, I need to do better with ddoc.

... I'm just gonna go on the record and say that I am really, really
not enjoying ddoc ;)

If I could get people starting to review the code, I'll polish up the
docs over the next few days.
Sep 14 2016
next sibling parent reply Meta <jared771 gmail.com> writes:
On Wednesday, 14 September 2016 at 13:28:23 UTC, Manu wrote:

 Cheers.
 Yeah, I need to do better with ddoc.

 ... I'm just gonna go on the record and say that I am really, 
 really not enjoying ddoc ;)


I can't say I'm a fan either.
Sep 14 2016
next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Wed, 14 Sep 2016 13:47:43 +0000
schrieb Meta <jared771 gmail.com>:


 On Wednesday, 14 September 2016 at 13:28:23 UTC, Manu wrote:

 Cheers.
 Yeah, I need to do better with ddoc.

 ... I'm just gonna go on the record and say that I am really,=20
 really not enjoying ddoc ;) =20


=20
 I can't say I'm a fan either.


You can replace $(D =E2=80=A6) with `=E2=80=A6` nowadays. It made me a bit =
more
of a fan. :)

--=20
Marco
Sep 14 2016
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/14/2016 09:47 AM, Meta wrote:

 On Wednesday, 14 September 2016 at 13:28:23 UTC, Manu wrote:

 Cheers.
 Yeah, I need to do better with ddoc.

 ... I'm just gonna go on the record and say that I am really, really
 not enjoying ddoc ;)


 I can't say I'm a fan either.


Why? -- Andrei
Sep 14 2016
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2016-09-14 22:14, Andrei Alexandrescu wrote:


 Why? -- Andrei


I've recently worked on a theme for Ddoc that will be used in TextMate. 
For convince I started by adding the CSS inline in the head tag. Turns 
out if you have CSS looking like this:

.foo {
   font-weight: bold;
   background-color: white;
}

Everything works perfectly fine. Then I added text color as well:

color: black;

Guess what happens? It cuts, it cuts everything after "color". The 
solution is to use a colon macro:

Macros:
   COLON = :

color$(COLON) black;

It took me quite a while and a lot of frustration until I figured that 
out. But I guess that's an inherit problem with a macro system.

That is what I'm missing from Ddoc:

* Inline source code for each symbol [1]. The [1] example both has a 
button to show the source code inline and a direct link to GitHub

* Link to GitHub for each symbol [1] and for the whole module. The way 
it's done now with $(PHOBOSSRC std/net/_isemail.d) is not standardized 
and feels like a hack. Why slashes and why the underscore? If anything 
it should be the fully qualified name. Ideally it should be automatic, 
or a compiler recognized macro that inserts the link

* Ddoc should organize symbols by visibility [2]
* Ddoc should generate index table [3]
* Ddoc should organize a module by symbol category [4]
* Add a way to inherit documentation from the parent class [5]

* Ddoc should give a list of inherited members [6]. JavaDoc does this by 
default [7]

* Consistent way of cross-referencing. Ideally one would just pass the 
fully qualified name (or local name for symbols in the same module) to a 
compiler recognized macro and it would generate the appropriate links

* An index page with all symbols, including a filter [8]. This is super 
convenient when you know the symbol to use but you need to double check 
how it's used or the behavior. I use it almost every day with Ruby

All this should be done within Ddoc without the need for any post 
processing.

In general I feel that Ddoc is too much focused on a generic macro 
system instead of focusing on being a good tool for writing 
documentation for code.

[1] 
http://api.rubyonrails.org/classes/ActiveRecord/FinderMethods.html#method-i-first

[2] https://issues.dlang.org/show_bug.cgi?id=11893
[3] https://issues.dlang.org/show_bug.cgi?id=11891
[4] https://issues.dlang.org/show_bug.cgi?id=11892
[5] https://issues.dlang.org/show_bug.cgi?id=10399
[6] https://issues.dlang.org/show_bug.cgi?id=7688

[7] 
http://help.eclipse.org/luna/index.jsp?topic=%2Forg.eclipse.jdt.doc.isv%2Freference%2Fapi%2Forg%2Feclipse%2Fjdt%2Fcore%2Fdom%2FSimpleName.html

[8] http://ruby-doc.org/core-2.3.1

-- 
/Jacob Carlborg
Sep 14 2016
next sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
Do we mind moving this discussion to the other topic I started
actually on this topic?

It's slightly upsetting that this discussion about ddoc is more
popular than the colour lib I spent months workinng on :(


On 15 September 2016 at 16:55, Jacob Carlborg via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 2016-09-14 22:14, Andrei Alexandrescu wrote:


 Why? -- Andrei



 I've recently worked on a theme for Ddoc that will be used in TextMate. For
 convince I started by adding the CSS inline in the head tag. Turns out if
 you have CSS looking like this:

 .foo {
   font-weight: bold;
   background-color: white;
 }

 Everything works perfectly fine. Then I added text color as well:

 color: black;

 Guess what happens? It cuts, it cuts everything after "color". The solution
 is to use a colon macro:

 Macros:
   COLON = :

 color$(COLON) black;

 It took me quite a while and a lot of frustration until I figured that out.
 But I guess that's an inherit problem with a macro system.

 That is what I'm missing from Ddoc:

 * Inline source code for each symbol [1]. The [1] example both has a button
 to show the source code inline and a direct link to GitHub

 * Link to GitHub for each symbol [1] and for the whole module. The way it's
 done now with $(PHOBOSSRC std/net/_isemail.d) is not standardized and feels
 like a hack. Why slashes and why the underscore? If anything it should be
 the fully qualified name. Ideally it should be automatic, or a compiler
 recognized macro that inserts the link

 * Ddoc should organize symbols by visibility [2]
 * Ddoc should generate index table [3]
 * Ddoc should organize a module by symbol category [4]
 * Add a way to inherit documentation from the parent class [5]

 * Ddoc should give a list of inherited members [6]. JavaDoc does this by
 default [7]

 * Consistent way of cross-referencing. Ideally one would just pass the fully
 qualified name (or local name for symbols in the same module) to a compiler
 recognized macro and it would generate the appropriate links

 * An index page with all symbols, including a filter [8]. This is super
 convenient when you know the symbol to use but you need to double check how
 it's used or the behavior. I use it almost every day with Ruby

 All this should be done within Ddoc without the need for any post
 processing.

 In general I feel that Ddoc is too much focused on a generic macro system
 instead of focusing on being a good tool for writing documentation for code.

 [1]
 http://api.rubyonrails.org/classes/ActiveRecord/FinderMethods.html#method-i-first

 [2] https://issues.dlang.org/show_bug.cgi?id=11893
 [3] https://issues.dlang.org/show_bug.cgi?id=11891
 [4] https://issues.dlang.org/show_bug.cgi?id=11892
 [5] https://issues.dlang.org/show_bug.cgi?id=10399
 [6] https://issues.dlang.org/show_bug.cgi?id=7688

 [7]
 http://help.eclipse.org/luna/index.jsp?topic=%2Forg.eclipse.jdt.doc.isv%2Freference%2Fapi%2Forg%2Feclipse%2Fjdt%2Fcore%2Fdom%2FSimpleName.html

 [8] http://ruby-doc.org/core-2.3.1

 --
 /Jacob Carlborg
Sep 15 2016
parent reply Jacob Carlborg <doob me.com> writes:
On 2016-09-15 09:11, Manu via Digitalmars-d wrote:

 Do we mind moving this discussion to the other topic I started
 actually on this topic?

 It's slightly upsetting that this discussion about ddoc is more
 popular than the colour lib I spent months workinng on :(


Sorry.

Is it the "ddoc latex/formulas?" topic? That's the only recent topic by 
you related to ddoc. But I have no interest in either latex or formulas.

-- 
/Jacob Carlborg
Sep 15 2016
parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 15 September 2016 at 19:23, Jacob Carlborg via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 2016-09-15 09:11, Manu via Digitalmars-d wrote:

 Do we mind moving this discussion to the other topic I started
 actually on this topic?

 It's slightly upsetting that this discussion about ddoc is more
 popular than the colour lib I spent months workinng on :(



 Sorry.

 Is it the "ddoc latex/formulas?" topic? That's the only recent topic by you
 related to ddoc. But I have no interest in either latex or formulas.


Yeah, sorry I forgot that detail. That thread meandered in the same
direction though.
Sep 15 2016
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/15/2016 02:55 AM, Jacob Carlborg wrote:

 On 2016-09-14 22:14, Andrei Alexandrescu wrote:


 Why? -- Andrei


 I've recently worked on a theme for Ddoc that will be used in TextMate.
 For convince I started by adding the CSS inline in the head tag. Turns
 out if you have CSS looking like this:

 .foo {
   font-weight: bold;
   background-color: white;
 }

 Everything works perfectly fine. Then I added text color as well:

 color: black;

 Guess what happens? It cuts, it cuts everything after "color". The
 solution is to use a colon macro:

 Macros:
   COLON = :

 color$(COLON) black;

 It took me quite a while and a lot of frustration until I figured that
 out. But I guess that's an inherit problem with a macro system.


Yah, this is terrible (it happens with http: all the time). We need to 
suppress the special meaning of ":" to only the cases when it's preceded 
by only one word and followed by a newline, e,g, "\ncolor:\n".


 That is what I'm missing from Ddoc:

 * Inline source code for each symbol [1]. The [1] example both has a
 button to show the source code inline and a direct link to GitHub


Can this be done by defining the expansion macros appropriately?


 * Link to GitHub for each symbol [1] and for the whole module. The way
 it's done now with $(PHOBOSSRC std/net/_isemail.d) is not standardized
 and feels like a hack. Why slashes and why the underscore?


Slashes are tolerable. The underscore I agree it's a quirk.


 If anything
 it should be the fully qualified name.


That'd be nice.


 Ideally it should be automatic,
 or a compiler recognized macro that inserts the link


Ironically the underscore is necessary because ddoc does too much 
automatically.


 * Ddoc should organize symbols by visibility [2]
 * Ddoc should generate index table [3]
 * Ddoc should organize a module by symbol category [4]
 * Add a way to inherit documentation from the parent class [5]

 * Ddoc should give a list of inherited members [6]. JavaDoc does this by
 default [7]

 * Consistent way of cross-referencing. Ideally one would just pass the
 fully qualified name (or local name for symbols in the same module) to a
 compiler recognized macro and it would generate the appropriate links

 * An index page with all symbols, including a filter [8]. This is super
 convenient when you know the symbol to use but you need to double check
 how it's used or the behavior. I use it almost every day with Ruby

 All this should be done within Ddoc without the need for any post
 processing.


I think we should start with a postprocessing-based approach and see 
what capabilities need to be added to obviate the need for postprocessing.


 In general I feel that Ddoc is too much focused on a generic macro
 system instead of focusing on being a good tool for writing
 documentation for code.


That's a fair assessment, and an actionable one.


Andrei
Sep 15 2016
parent reply Jacob Carlborg <doob me.com> writes:
On 2016-09-15 14:54, Andrei Alexandrescu wrote:


 Yah, this is terrible (it happens with http: all the time). We need to
 suppress the special meaning of ":" to only the cases when it's preceded
 by only one word and followed by a newline, e,g, "\ncolor:\n".


It's useful for many of the sections to allow the section name on the 
same line as the content:

/// Returns: some useful value
int foo();

See the Ddoc documentation for more examples [1].


 Can this be done by defining the expansion macros appropriately?


What are "the expansion macros" ?


 Slashes are tolerable. The underscore I agree it's a quirk.


There are several (other) problems:

* How do I know I should use slashes? There's no documentation
* It's not a standard macro. If it's not here [2] nobody will find it or 
use it


 That's a fair assessment, and an actionable one.


Perhaps the current implementation of Ddoc could be moved to a separate 
tool that would focus on being a macro system. That can still be used 
for the site and other documents. Ddoc in the compiler could be improved 
to be a better tool for documenting code. I think these are two separate 
use cases that might not be best accomplished with the same tool.

[1] http://dlang.org/spec/ddoc.html#standard_srction
[2] http://dlang.org/spec/ddoc.html#predefined_macros

-- 
/Jacob Carlborg
Sep 15 2016
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/15/2016 02:34 PM, Jacob Carlborg wrote:

 On 2016-09-15 14:54, Andrei Alexandrescu wrote:


 Yah, this is terrible (it happens with http: all the time). We need to
 suppress the special meaning of ":" to only the cases when it's preceded
 by only one word and followed by a newline, e,g, "\ncolor:\n".


 It's useful for many of the sections to allow the section name on the
 same line as the content:

 /// Returns: some useful value
 int foo();

 See the Ddoc documentation for more examples [1].


That's fine because "Returns" is special-cased. Random words at the 
beginning of a line followed by ":" should not receive special treatment.


 Can this be done by defining the expansion macros appropriately?


 What are "the expansion macros" ?


Those DDOC_XXX things that are used for formatting declarations.


 Slashes are tolerable. The underscore I agree it's a quirk.


 There are several (other) problems:

 * How do I know I should use slashes? There's no documentation


"Monkey see, monkey do." :o)


 * It's not a standard macro. If it's not here [2] nobody will find it or
 use it


Guess we can improve the (d)docs.


 That's a fair assessment, and an actionable one.


 Perhaps the current implementation of Ddoc could be moved to a separate
 tool that would focus on being a macro system. That can still be used
 for the site and other documents. Ddoc in the compiler could be improved
 to be a better tool for documenting code. I think these are two separate
 use cases that might not be best accomplished with the same tool.


That'd be neat. As always, somebody's got to do it...


Andrei
Sep 15 2016
parent Jacob Carlborg <doob me.com> writes:
On 2016-09-15 20:55, Andrei Alexandrescu wrote:


 That's fine because "Returns" is special-cased. Random words at the
 beginning of a line followed by ":" should not receive special treatment.


Ah, so you mean all non-standard non-special sections? Basically any 
section that is not listed in the ddoc documentation.


 Those DDOC_XXX things that are used for formatting declarations.


Hmm, I don't think there's a DDOC_ macro that evaluates to the full 
source code a given symbol. For a function that would include the source 
code of the body.


 Guess we can improve the (d)docs.


But then we need to have a standardized macro for that.


 That'd be neat. As always, somebody's got to do it...


As always.

-- 
/Jacob Carlborg
Sep 16 2016
prev sibling parent Meta <jared771 gmail.com> writes:
On Wednesday, 14 September 2016 at 20:14:48 UTC, Andrei 
Alexandrescu wrote:

 On 09/14/2016 09:47 AM, Meta wrote:

 On Wednesday, 14 September 2016 at 13:28:23 UTC, Manu wrote:

 Cheers.
 Yeah, I need to do better with ddoc.

 ... I'm just gonna go on the record and say that I am really, 
 really
 not enjoying ddoc ;)


 I can't say I'm a fan either.


 Why? -- Andrei


The syntax is clunky and it's poorly documented. I've been 
lurking this mailing list since late 2012 and sporadically 
contributing since late 2013, and I had no idea you could do 
something like your example for Manu using DDOC. It seems like a 
real case of NiH syndrome; there are plenty of more widely used 
systems that would probably fit D just fine, but instead we have 
our own special macro documentation language. Really the only 
thing DDOC has going for it in comparison to something like 
Doxygen, IMO, is momentum. It's just too much of a pain to switch 
now.
Sep 15 2016
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/14/2016 09:28 AM, Manu via Digitalmars-d wrote:

 ... I'm just gonna go on the record and say that I am really, really
 not enjoying ddoc ;)


I was using ddoc today for the Greeks in DIP1000 and was musing about 
what makes it annoying. I think a major part is the macro invocation 
syntax: $(MACRO a1, a2, a3). After writing it for a few dozen times 
today, I figured it's objectively awkward to type (at least on the US 
keyboard): the dollar sign followed by the left paren is just unpleasant 
for mechanical reasons.

The resulting look is alien too, as opposed to the more familiar 
$MACRO(a1, a2, a3) or the CPP-style MACRO(a1, a2, a3).

There are a couple of un(der)documented tricks without which life is 
very difficult. For example, people commonly believe that in order to 
embed a literal comma they need to go like this:

COMMA = ,

and then $(MACRO Hello$(COMMA) world, a2). That scales poorly with the 
size of the argument. In fact, there's a much better solution:

ARGS = $0

and then $(MACRO $(ARGS Hello, world), a2) allows any amount of text 
with commas. Sadly no similar solution for unpaired parens.

Also, the fact that the doc opens with the text and then has an obscure 
"Macros:" line at a point is unpleasant. The macro definitions should 
come first, followed by a separator and then the content. That would 
make for a much nicer flow.

There are a few other macros I never leave home without:

ARGS = $0
COMMA = ,
DOLLAR = $
LPAREN = (
RPAREN = )
TAIL = $+
COLON = :
LF = $(LF)
$(LF)
COMMENT =
SP = $(COMMENT space coming) $(COMMENT that was a space)

TAIL sadly doesn't work properly.

Overall: I'm trying to build a list of ddoc grievances that can be 
fixed. It's the de jure and de facto standard for D documentation (with 
the ddoc processor or ddox), so we'd do good to improve it.

Please add your own ideas.


Andrei
Sep 14 2016
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/14/2016 05:31 PM, Andrei Alexandrescu wrote:

 After writing it for a few dozen times today, I figured it's objectively
 awkward to type (at least on the US keyboard): the dollar sign followed
 by the left paren is just unpleasant for mechanical reasons.

 The resulting look is alien too, as opposed to the more familiar
 $MACRO(a1, a2, a3) or the CPP-style MACRO(a1, a2, a3).


I meant to add here: we can improve this in a backward-compatible way by 
allowing users to choose syntax by means of some notation, e.g. this 
thing as the first macro:

DD0C_SYNTAX = $(X)

for the existing syntax, vs.

DD0C_SYNTAX = $X()

for the $MACRO(a1, a2) syntax. We may also allow users to define the 
argument separator, also on a per-document basis. By default:

DDOC_ARGUMENT_SEPARATOR = ,

We may also allow custom parens and so on.


Andrei
Sep 14 2016
prev sibling next sibling parent reply bachmeier <no spam.net> writes:
On Wednesday, 14 September 2016 at 21:31:51 UTC, Andrei 
Alexandrescu wrote:


 Please add your own ideas.


 Andrei


One thing that I didn't like when I used DDOC was that I couldn't 
find a way to add a block of text directly to the output file. I 
know that can be done with html, but it's not the same. When I am 
programming I want to make notes (the way I make comments) but 
not get bogged down with all the macros/html stuff. Unformatted 
text is way better than unwritten text. Blocks of markdown would 
be better, but blocks of text would have helped.

[Maybe this is possible but I was a new D user at that time and 
haven't had a reason to look into it.]
Sep 14 2016
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/14/16 7:04 PM, bachmeier wrote:

 On Wednesday, 14 September 2016 at 21:31:51 UTC, Andrei Alexandrescu wrote:


 Please add your own ideas.


 Andrei


 One thing that I didn't like when I used DDOC was that I couldn't find a
 way to add a block of text directly to the output file. I know that can
 be done with html, but it's not the same. When I am programming I want
 to make notes (the way I make comments) but not get bogged down with all
 the macros/html stuff. Unformatted text is way better than unwritten
 text. Blocks of markdown would be better, but blocks of text would have
 helped.

 [Maybe this is possible but I was a new D user at that time and haven't
 had a reason to look into it.]


Do you mean like a macro $(VERBATIM ... text with balanced parens ...)? 
-- Andrei
Sep 14 2016
parent reply bachmeier <no spam.net> writes:
On Thursday, 15 September 2016 at 00:42:57 UTC, Andrei 
Alexandrescu wrote:


 Do you mean like a macro $(VERBATIM ... text with balanced 
 parens ...)? -- Andrei


Yes. Unfortunately I cannot find any mention of it on this page:
http://dlang.org/spec/ddoc.html
Sep 15 2016
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/15/2016 11:25 AM, bachmeier wrote:

 On Thursday, 15 September 2016 at 00:42:57 UTC, Andrei Alexandrescu wrote:


 Do you mean like a macro $(VERBATIM ... text with balanced parens
 ...)? -- Andrei


 Yes. Unfortunately I cannot find any mention of it on this page:
 http://dlang.org/spec/ddoc.html


It doesn't exist :o). I agree it's needed, I've hit the matter a few 
times. See https://issues.dlang.org/show_bug.cgi?id=16498. -- Andrei
Sep 15 2016
prev sibling next sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 15 September 2016 at 07:31, Andrei Alexandrescu via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 09/14/2016 09:28 AM, Manu via Digitalmars-d wrote:

 ... I'm just gonna go on the record and say that I am really, really
 not enjoying ddoc ;)



 I was using ddoc today for the Greeks in DIP1000 and was musing about what
 makes it annoying. I think a major part is the macro invocation syntax:
 $(MACRO a1, a2, a3). After writing it for a few dozen times today, I figured
 it's objectively awkward to type (at least on the US keyboard): the dollar
 sign followed by the left paren is just unpleasant for mechanical reasons.

 The resulting look is alien too, as opposed to the more familiar $MACRO(a1,
 a2, a3) or the CPP-style MACRO(a1, a2, a3).

 There are a couple of un(der)documented tricks without which life is very
 difficult. For example, people commonly believe that in order to embed a
 literal comma they need to go like this:

 COMMA = ,

 and then $(MACRO Hello$(COMMA) world, a2). That scales poorly with the size
 of the argument. In fact, there's a much better solution:

 ARGS = $0

 and then $(MACRO $(ARGS Hello, world), a2) allows any amount of text with
 commas. Sadly no similar solution for unpaired parens.

 Also, the fact that the doc opens with the text and then has an obscure
 "Macros:" line at a point is unpleasant. The macro definitions should come
 first, followed by a separator and then the content. That would make for a
 much nicer flow.

 There are a few other macros I never leave home without:

 ARGS = $0
 COMMA = ,
 DOLLAR = $
 LPAREN = (
 RPAREN = )
 TAIL = $+
 COLON = :
 LF = $(LF)
 $(LF)
 COMMENT =
 SP = $(COMMENT space coming) $(COMMENT that was a space)

 TAIL sadly doesn't work properly.

 Overall: I'm trying to build a list of ddoc grievances that can be fixed.
 It's the de jure and de facto standard for D documentation (with the ddoc
 processor or ddox), so we'd do good to improve it.

 Please add your own ideas.


I haven't had enough experience for meaningful input. I have
experienced the comma and parens things myself, and I generally just
hate that everything is a macro, like you said yourself above.
It didn't occur to me to attempt the sort of workarounds you
illustrated here, I rather just factored the commas and parens out of
my docs and moved on ;)

Like, I really just don't care enough to try and understand ddoc
sufficiently to have a bag of tricks like those you demonstrated above
to workaround these issues. It's not a skill I *want* to possess,
rather, in this case, it's a skill I'm being forced into.
You're welcome to call me lazy, I'd suggest I'm being realistic.
Perhaps a phobos contributor will be required to work it out (as seems
to be a requirement for me right now), but normal programmers
wouldn't. In 7 years, I've never been motivated to find workarounds to
ddoc shortcomings before now, and now, it's only because people are
hassling me. There's no intrinsic motivation here... one reason I use
D is because I hate the C preprocessor ;)

If I had to suggest, I'd introduce doxygen style \tags alongside the
macros, then when people try and type docs in the way they've been
doing for decades, it'll just work, and they can get on with their
code. Nobody likes writing documentation, it needs to have the minimum
possible friction or people just won't.

Added bonus; when you cut&paste C code to adapt bindings, the existing
docs on the C code would remain valid documentation without additional
amendment.
I bind C code regularly, and my goto 'solution' is to just remove the
doxygen doco comments from the translated code and point them to the
C/C++ docs instead.
Again, call me what you like, I think my behaviour is probably typical
though. I'm just being honest.
Sep 14 2016
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/14/16 9:28 PM, Manu via Digitalmars-d wrote:

 Like, I really just don't care enough to try and understand ddoc
 sufficiently to have a bag of tricks like those you demonstrated above
 to workaround these issues. It's not a skill I *want* to possess,
 rather, in this case, it's a skill I'm being forced into.
 You're welcome to call me lazy, I'd suggest I'm being realistic.
 Perhaps a phobos contributor will be required to work it out (as seems
 to be a requirement for me right now), but normal programmers
 wouldn't. In 7 years, I've never been motivated to find workarounds to
 ddoc shortcomings before now, and now, it's only because people are
 hassling me. There's no intrinsic motivation here... one reason I use
 D is because I hate the C preprocessor ;)


I don't see those workarounds to ddoc shortcomings. m4 and all macro 
systems I know of have similar idioms. It's the nature of macro 
processing. Are you simply saying you're familiar with other 
documentation tools and are not motivated to get into another one? 
That's entirely fair.


 If I had to suggest, I'd introduce doxygen style \tags alongside the
 macros, then when people try and type docs in the way they've been
 doing for decades, it'll just work, and they can get on with their
 code. Nobody likes writing documentation, it needs to have the minimum
 possible friction or people just won't.


I have difficulty understanding this. I haven't looked at Doxygen in a 
long time and never really used it, but from what I see at 
https://www.stack.nl/~dimitri/doxygen/manual/commands.html it seems the 
\tags you refer to are just a form of macros. The syntax is different, 
i.e. you'd write \a hello whereas in html you'd write <i>hello</i>, in 
latex \textit{hello}, in ddoc $(I hello). It's a matter of syntax and 
though I agree syntax matters (and it would be nice to make ddoc's more 
configurable), is it right to assume you simply want a syntax you're 
more familiar with?

How do you mean people "type docs in the way they've been doing for 
decades"? Are you implying doxygen not only has been around for decades, 
but it's been some sort of ubiquitous standard? Honest question, I'm 
definitely not getting that.

Does this boil down to - if we replace the macro syntax with one closer 
to doxygen things will just click? (That may as well be the case.)


Andrei
Sep 14 2016
next sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 15 September 2016 at 13:49, Andrei Alexandrescu via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 9/14/16 9:28 PM, Manu via Digitalmars-d wrote:

 Like, I really just don't care enough to try and understand ddoc
 sufficiently to have a bag of tricks like those you demonstrated above
 to workaround these issues. It's not a skill I *want* to possess,
 rather, in this case, it's a skill I'm being forced into.
 You're welcome to call me lazy, I'd suggest I'm being realistic.
 Perhaps a phobos contributor will be required to work it out (as seems
 to be a requirement for me right now), but normal programmers
 wouldn't. In 7 years, I've never been motivated to find workarounds to
 ddoc shortcomings before now, and now, it's only because people are
 hassling me. There's no intrinsic motivation here... one reason I use
 D is because I hate the C preprocessor ;)



 I don't see those workarounds to ddoc shortcomings. m4 and all macro systems
 I know of have similar idioms. It's the nature of macro processing. Are you
 simply saying you're familiar with other documentation tools and are not
 motivated to get into another one? That's entirely fair.


That's absolutely what I'm saying, but not only that, given that I'm
being forced into it now, I'm actively not enjoying it.
I understand it's a macro system, and I'm saying that's the problem as
I perceive it :)

I clearly recall when I learned doxygen. I generally liked it, and I
was motivated to populate my code with docs.
Perhaps there's an obvious counter-pressure in this case since I
already know doxygen, but I think that's insignificant compared to the
fact that I just don't like ddoc.
This macro system is not what I would have done, and it's just not
compelling. I don't want documentation-realted macro definitions in my
code. I also hate the paren spam.



 If I had to suggest, I'd introduce doxygen style \tags alongside the
 macros, then when people try and type docs in the way they've been
 doing for decades, it'll just work, and they can get on with their
 code. Nobody likes writing documentation, it needs to have the minimum
 possible friction or people just won't.



 I have difficulty understanding this. I haven't looked at Doxygen in a long
 time and never really used it, but from what I see at
 https://www.stack.nl/~dimitri/doxygen/manual/commands.html it seems the
 \tags you refer to are just a form of macros. The syntax is different, i.e.
 you'd write \a hello whereas in html you'd write <i>hello</i>, in latex
 \textit{hello}, in ddoc $(I hello). It's a matter of syntax and though I
 agree syntax matters (and it would be nice to make ddoc's more
 configurable), is it right to assume you simply want a syntax you're more
 familiar with?


I think of those examples, doxygen is objectively the least annoying
syntax! It's 2 bytes, and doesn't introduce delimiters or parens of
any kind.

Sure, that's definitely an influence, I hate paren spam, but it's not
all. If in practise I was able to make my docs be how I want as easily
as I was able to the first time I encountered doxygen, I wouldn't be
bothered. But I've had a number of contacts with ddoc now, and it's
just not intuitive how to make my docs how I want, whereas with
doxygen, I was walking within minutes, seconds even.

I think I have a sense of uneasy-ness that comes from my intuition
that macros are macros, and I don't know which ones are 'special', or
which are just macros.
Ie, some macros are obviously not actually macros, but keywords in disguise.
Doxygen says things like "\a word" (style this as an argument), ddoc
seems to be $(I word) (italic this), which says nothing about the
semantic meaning of the thing allowing styling to be deferred.
I refer to this page: https://dlang.org/spec/ddoc.html
There's nothing there that says how to style a template argument for
instance, it's all 'italic, big, small, bold' etc, and skimming other
ddoc code, I see people using raw styling like $(I ..) (as you just
demonstrated above).
At face value, it feels clumsy and poorly defined. I'm sure I'm
incorrect, but I just didn't have this same first-impression with

before).

The only reference I've ever looked at for doxygen is this page:
https://www.stack.nl/~dimitri/doxygen/manual/commands.html
I never read another line from their manuals, everything else emerged
via osmosis, and I think osmosis is the correct way a programmer
should learn this stuff, which means it needs to be as obvious and
intuitive as possible.



 How do you mean people "type docs in the way they've been doing for
 decades"? Are you implying doxygen not only has been around for decades, but
 it's been some sort of ubiquitous standard? Honest question, I'm definitely
 not getting that.


Do you feel that doxygen is not a ubiquitous standard? It's been
around for like 20 years.
I can't think of any non-doxygen docs I interact with regularly other
than Microsoft docs... Maybe this is just a C/C++ ecosystem thing?



 Does this boil down to - if we replace the macro syntax with one closer to
 doxygen things will just click? (That may as well be the case.)


Well, I think the core of my uneasy-ness is that the macros have no
clear semantic meaning.
Obviously I can design my own set of macros that express my semantic
meaning... to hell with standards! ;)
Your prior post said "here's a bunch of macros I never leave home
without", that's exactly the problem as I see it.
I don't want too see ddoc macro definition blocks appearing at random
locations in my source code.

And I'll also remind you of my past criticism that my solution to
porting C code to D, is to block delete the doxygen comment blocks
rather than bothering to port them to D. If I had a reason to say what
you said above "if we replace the macro syntax with one closer to
doxygen...", this is that reason, but it's the fact that they're
macros rather than keywords that makes me upset, so replacing the
macros with a different syntax probably won't make me feel less
uneasy.

Perhaps my uneasy-ness would subside if the list of 'builtins'
(https://dlang.org/spec/ddoc.html) was more comprehensive (comparable
to doxygen commands), and I never saw a ddoc macro definition in code
anywhere. My gripe with the syntax would probably have passed without
comment if that were the case and I would have just gotten used to it.

And don't get me wrong, I think it's absolutely fantastic that D has a
built-in documentation system. I just don't really like it.
Sep 15 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/15/2016 12:06 AM, Manu via Digitalmars-d wrote:

 I think I have a sense of uneasy-ness that comes from my intuition
 that macros are macros, and I don't know which ones are 'special', or
 which are just macros.


That's easy, they are all just macros, and can all be redefined.
Sep 15 2016
parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 15 September 2016 at 17:38, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 9/15/2016 12:06 AM, Manu via Digitalmars-d wrote:

 I think I have a sense of uneasy-ness that comes from my intuition
 that macros are macros, and I don't know which ones are 'special', or
 which are just macros.



 That's easy, they are all just macros, and can all be redefined.


Yeah... I suspected this was the case ;)
Sep 15 2016
prev sibling parent reply Patrick Schluter <Patrick.Schluter bbox.fr> writes:
On Thursday, 15 September 2016 at 03:49:55 UTC, Andrei 
Alexandrescu wrote:

 On 9/14/16 9:28 PM, Manu via Digitalmars-d wrote:

 Like, I really just don't care enough to try and understand 
 ddoc
 sufficiently to have a bag of tricks like those you 
 demonstrated above
 to workaround these issues. It's not a skill I *want* to 
 possess,
 rather, in this case, it's a skill I'm being forced into.
 You're welcome to call me lazy, I'd suggest I'm being 
 realistic.
 Perhaps a phobos contributor will be required to work it out 
 (as seems
 to be a requirement for me right now), but normal programmers
 wouldn't. In 7 years, I've never been motivated to find 
 workarounds to
 ddoc shortcomings before now, and now, it's only because 
 people are
 hassling me. There's no intrinsic motivation here... one 
 reason I use
 D is because I hate the C preprocessor ;)


 I don't see those workarounds to ddoc shortcomings. m4 and all 
 macro systems I know of have similar idioms. It's the nature of 
 macro processing. Are you simply saying you're familiar with 
 other documentation tools and are not motivated to get into 
 another one? That's entirely fair.


 If I had to suggest, I'd introduce doxygen style \tags 
 alongside the
 macros, then when people try and type docs in the way they've 
 been
 doing for decades, it'll just work, and they can get on with 
 their
 code. Nobody likes writing documentation, it needs to have the 
 minimum
 possible friction or people just won't.


 I have difficulty understanding this. I haven't looked at 
 Doxygen in a long time and never really used it, but from what 
 I see at 
 https://www.stack.nl/~dimitri/doxygen/manual/commands.html it 
 seems the \tags you refer to are just a form of macros. The 
 syntax is different, i.e. you'd write \a hello whereas in html 
 you'd write <i>hello</i>, in latex \textit{hello}, in ddoc $(I 
 hello). It's a matter of syntax and though I agree syntax 
 matters (and it would be nice to make ddoc's more 
 configurable), is it right to assume you simply want a syntax 
 you're more familiar with?

 How do you mean people "type docs in the way they've been doing 
 for decades"? Are you implying doxygen not only has been around 
 for decades, but it's been some sort of ubiquitous standard? 
 Honest question, I'm definitely not getting that.


doxygen is mostly javadoc compatible and that one is the standard.
(in fact the \ can be replaced by   every where) and it allows 
direct html tags. The good thing about doxygen is that it 
supports also markdown.
There are also some niceties like auto brief and autoreferencing 
(when a symbol is written in a text will be clickable if its body 
was reachable during generation of the doc).


 Does this boil down to - if we replace the macro syntax with 
 one closer to doxygen things will just click? (That may as well 
 be the case.)


Probably. But be reminded of your remark in the SDLang vs. JSON 
thread from last year:
"...this is a strategic error. Please throw s/SDL/DDOC/ away and 
use a standardized file format. -- Andrei
Sep 15 2016
parent Bastiaan Veelo <Bastiaan Veelo.net> writes:
On Thursday, 15 September 2016 at 09:46:07 UTC, Patrick Schluter 
wrote:

 There are also some niceties like auto brief and 
 autoreferencing (when a symbol is written in a text will be 
 clickable if its body was reachable during generation of the 
 doc).


Not only that, by using tag files [1] Doxygen even generates 
clickable symbols to third party libraries and other online 
documentation, without having access to the respective code.

[1] 
http://www.stack.nl/~dimitri/doxygen/manual/config.html#config_external
Sep 15 2016
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/14/2016 2:31 PM, Andrei Alexandrescu wrote:

 The resulting look is alien too, as opposed to the more familiar $MACRO(a1, a2,
 a3) or the CPP-style MACRO(a1, a2, a3).


The $(MACRO a1, a2, a3) syntax comes from makefiles.
Sep 14 2016
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/14/2016 11:11 PM, Walter Bright wrote:

 On 9/14/2016 2:31 PM, Andrei Alexandrescu wrote:

 The resulting look is alien too, as opposed to the more familiar $MACRO(a1, a2,
 a3) or the CPP-style MACRO(a1, a2, a3).


 The $(MACRO a1, a2, a3) syntax comes from makefiles.


The original idea was to be able to write documentation without needing many 
macros, if any. Such as this I wrote recently:

/**
  * Provides a safe interface to the Posix stat() function.
  *
  * Detailed description that I didn't write for this function,
  * but this is where it goes.
  *
  * Another paragraph.
  *
  * Params:
  *	path = file name
  *	buf = stat instance to be filled in
  * Returns:
  *	0 for success, system errno for error
  */

I suspect that adding Markdown would help a lot (yes, I changed my mind about 
that). Also auto-detecting email addresses and urls and turning them into 
clickable links would help.
Sep 14 2016
next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 15 September 2016 at 16:44, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:

 On 9/14/2016 11:11 PM, Walter Bright wrote:

 On 9/14/2016 2:31 PM, Andrei Alexandrescu wrote:

 The resulting look is alien too, as opposed to the more familiar
 $MACRO(a1, a2,
 a3) or the CPP-style MACRO(a1, a2, a3).



 The $(MACRO a1, a2, a3) syntax comes from makefiles.



 The original idea was to be able to write documentation without needing many
 macros, if any. Such as this I wrote recently:

 /**
  * Provides a safe interface to the Posix stat() function.
  *
  * Detailed description that I didn't write for this function,
  * but this is where it goes.
  *
  * Another paragraph.
  *
  * Params:
  *      path = file name
  *      buf = stat instance to be filled in
  * Returns:
  *      0 for success, system errno for error
  */

 I suspect that adding Markdown would help a lot (yes, I changed my mind
 about that). Also auto-detecting email addresses and urls and turning them
 into clickable links would help.


+1
Sep 15 2016
prev sibling next sibling parent Paolo Invernizzi <paolo.invernizzi no.address> writes:
On Thursday, 15 September 2016 at 06:44:16 UTC, Walter Bright 
wrote:

 On 9/14/2016 11:11 PM, Walter Bright wrote:

 I suspect that adding Markdown would help a lot (yes, I changed 
 my mind about that). Also auto-detecting email addresses and 
 urls and turning them into clickable links would help.


I **really** love the pragmatism of this man!

;-)

/Paolo
Sep 15 2016
prev sibling next sibling parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Wed, 14 Sep 2016 23:44:16 -0700
schrieb Walter Bright <newshound2 digitalmars.com>:


 I suspect that adding Markdown would help a lot (yes, I changed my mind about 
 that).


+1

-- 
Marco
Sep 15 2016
parent reply Nemanja Boric <4burgos gmail.com> writes:
On Thursday, 15 September 2016 at 09:45:26 UTC, Marco Leise wrote:

 Am Wed, 14 Sep 2016 23:44:16 -0700
 schrieb Walter Bright <newshound2 digitalmars.com>:

 I suspect that adding Markdown would help a lot (yes, I changed 
 my mind about that).



https://github.com/kiith-sa/harbored-mod provides support for 
mixing ddoc and markdown. That's what we're using, and it's nice 
(especially because docs in source files look very close to 
generated ones), but it would be nicer if we could just use dmd 
for that, of course.
Sep 15 2016
parent Walter Bright <newshound2 digitalmars.com> writes:
On 9/15/2016 4:59 AM, Nemanja Boric wrote:

 On Thursday, 15 September 2016 at 09:45:26 UTC, Marco Leise wrote:

 Am Wed, 14 Sep 2016 23:44:16 -0700
 schrieb Walter Bright <newshound2 digitalmars.com>:

 I suspect that adding Markdown would help a lot (yes, I changed my mind about
 that).



 https://github.com/kiith-sa/harbored-mod provides support for mixing ddoc and
 markdown. That's what we're using, and it's nice (especially because docs in
 source files look very close to generated ones), but it would be nicer if we
 could just use dmd for that, of course.


It looks like you've done some great work with this! And Boost licensed, too! I 
think we can leverage this for Ddoc markdown support. Thanks!
Sep 15 2016
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2016-09-15 08:44, Walter Bright wrote:


 The original idea was to be able to write documentation without needing
 many macros, if any. Such as this I wrote recently:

 /**
  * Provides a safe interface to the Posix stat() function.
  *
  * Detailed description that I didn't write for this function,
  * but this is where it goes.
  *
  * Another paragraph.
  *
  * Params:
  *    path = file name
  *    buf = stat instance to be filled in
  * Returns:
  *    0 for success, system errno for error
  */

 I suspect that adding Markdown would help a lot (yes, I changed my mind
 about that). Also auto-detecting email addresses and urls and turning
 them into clickable links would help.


Yeah, if you already have defined all the standard ddoc macros because 
the default styling is not useful at all.

This is an example of the documentation generator for Ruby called Yard 
[1]. The default output view/style might not be the best in the world 
but I think it's pretty good. Here's an example of how to write the 
documentation [2].

[1] http://www.rubydoc.info/gems/httparty/0.14.0/HTTParty/ClassMethods
[2] http://www.rubydoc.info/gems/yard/file/docs/GettingStarted.md

-- 
/Jacob Carlborg
Sep 15 2016
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/15/2016 10:37 AM, Jacob Carlborg wrote:

 Yeah, if you already have defined all the standard ddoc macros because
 the default styling is not useful at all.


Glad to hear that. It becomes a matter of improving the default-provided 
macros and/or distribute some .ddoc batteries.

Jacob, I'd also like to thank for the great constructive ideas for 
improving things and for formulating them as enhancement requests on 
bugsilla. This is the kind of communication that pushes things forward.


Andrei
Sep 15 2016
parent Jacob Carlborg <doob me.com> writes:
On 2016-09-15 16:45, Andrei Alexandrescu wrote:


 Glad to hear that. It becomes a matter of improving the default-provided
 macros and/or distribute some .ddoc batteries.

 Jacob, I'd also like to thank for the great constructive ideas for
 improving things and for formulating them as enhancement requests on
 bugsilla. This is the kind of communication that pushes things forward.


The oldest bugzilla issue I referred to is more than four years old and 
this is not the first discussion on the topic :(

-- 
/Jacob Carlborg
Sep 15 2016
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/15/2016 7:37 AM, Jacob Carlborg wrote:

 Yeah, if you already have defined all the standard ddoc macros because the
 default styling is not useful at all.


The default styling was set up to produce legible output while not requiring
any 
configuration file, nor a .css style file nor any knowledge of HTML at all.
Sep 15 2016
parent reply Jacob Carlborg <doob me.com> writes:
On 2016-09-16 02:02, Walter Bright wrote:


 The default styling was set up to produce legible output while not
 requiring any configuration file, nor a .css style file nor any
 knowledge of HTML at all.


I don't see how that's relevant. Currently one needs to have knowledge 
of HTML and CSS because the default output is not usable.

With Yard (documentation generator for Ruby) one just invokes "yard -f 
html -o output intput.rb" and it generates all files and put them into 
"output". The result looks something like this [1].

How is that requiring any HTML or CSS knowledge?

[1] 
http://www.rubydoc.info/gems/rails-bootstrap-helpers/0.1.0/RailsBootstrapHelpers/Helpers/ButtonHelper

-- 
/Jacob Carlborg
Sep 16 2016
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/16/16 7:30 AM, Jacob Carlborg wrote:

 I don't see how that's relevant. Currently one needs to have knowledge
 of HTML and CSS because the default output is not usable.


Jacob, would you want to try your hand at tweaking the defaults? They 
are in a string in https://github.com/dlang/dmd/blob/master/src/doc.d. 
Clearly there's agreement these can be improved, and it's incredibly 
more productive to just improve the content instead of arguing how it 
could. Thanks!! -- Andrei
Sep 16 2016
parent reply Jacob Carlborg <doob me.com> writes:
On 2016-09-16 16:39, Andrei Alexandrescu wrote:


 Jacob, would you want to try your hand at tweaking the defaults? They
 are in a string in https://github.com/dlang/dmd/blob/master/src/doc.d.
 Clearly there's agreement these can be improved, and it's incredibly
 more productive to just improve the content instead of arguing how it
 could. Thanks!! -- Andrei


My design skills are quite limited. The theme I'm using for TextMate is 
basically the same as Apple is using for their documentation. But now 
when I tried to find a link to it, I see that they've changed the design.

If I can use what I already have it might not be that much work.

Two questions:

* Can we have DMD output a separate CSS file? What will avoid the colon 
issue mentioned before and avoid the need to have 300-500 lines of CSS 
code in each HTML document

* Can we create a new macro which will be used for each source code line 
in examples enclosed in "---"? That would allow to add line numbers with CSS

-- 
/Jacob Carlborg
Sep 16 2016
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/16/2016 12:07 PM, Jacob Carlborg wrote:

 On 2016-09-16 16:39, Andrei Alexandrescu wrote:


 Jacob, would you want to try your hand at tweaking the defaults? They
 are in a string in https://github.com/dlang/dmd/blob/master/src/doc.d.
 Clearly there's agreement these can be improved, and it's incredibly
 more productive to just improve the content instead of arguing how it
 could. Thanks!! -- Andrei


 My design skills are quite limited. The theme I'm using for TextMate is
 basically the same as Apple is using for their documentation. But now
 when I tried to find a link to it, I see that they've changed the design.

 If I can use what I already have it might not be that much work.


That would be awesome!


 Two questions:

 * Can we have DMD output a separate CSS file? What will avoid the colon
 issue mentioned before and avoid the need to have 300-500 lines of CSS
 code in each HTML document


I'd say we start without a separate CSS file. When the need for it 
becomes obvious, we can do that. What would be great is that your 
initiative isn't forestalled by the need for some other work.


 * Can we create a new macro which will be used for each source code line
 in examples enclosed in "---"? That would allow to add line numbers with
 CSS


We can, and we should (there are js solutions too). Again, let's get the 
ball rolling and amplify this as needed.


Thanks!

Andrei
Sep 16 2016
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/15/16 2:11 AM, Walter Bright wrote:

 On 9/14/2016 2:31 PM, Andrei Alexandrescu wrote:

 The resulting look is alien too, as opposed to the more familiar
 $MACRO(a1, a2,
 a3) or the CPP-style MACRO(a1, a2, a3).


 The $(MACRO a1, a2, a3) syntax comes from makefiles.


In the words of Seinfeld, "And that doesn't help either!"

I was thinking more about it and figured we could introduce a special 
macro at the very beginning of a .ddoc file or "Macros:" section called 
DDOC_MACRO_CALL_SYNTAX. It may be defined one of the following:

DDOC_MACRO_CALL_SYNTAX = DDOC

That's today's syntax.

DDOC_MACRO_CALL_SYNTAX = CPP

A macro will be invoked as MACRO(a1, a2).

DDOC_MACRO_CALL_SYNTAX = $CPP

A macro will be invoked as $MACRO(a1, a2)

DDOC_MACRO_CALL_SYNTAX = LaTeX

Macro invocation is \MACRO{a1}{a2}.

This definition is modular, i.e. it doesn't carry beyond the current 
document (otherwise the semantics of a .dd file depends on the .ddoc 
file(s) it's used with).

Macro definitions stay the same using $1, $2 etc. but macros invoked by 
them would need to obey the DDOC_MACRO_CALL_SYNTAX defined at the top.

Thoughts?


Andrei
Sep 15 2016
parent Walter Bright <newshound2 digitalmars.com> writes:
Started a new thread on this, "Ddoc macro syntax". Please direct further 
discussion there.
Sep 15 2016
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 9/14/2016 5:47 AM, Meta wrote:

 DDOC sections such as Returns, Args, etc.


s/Args/Params/
Sep 14 2016
prev sibling parent Johan Engelen <j j.nl> writes:
On Monday, 12 September 2016 at 04:14:27 UTC, Manu wrote:

 I think I'm about as happy with my colour lib as I'm going to 
 be. It really needs reviews.


I don't know enough about colors and what people do with it to 
give you helpful feedback, but I was wondering if you had looked 
at the (auto) vectorization of the code.
I see some matrix math code, which perhaps would benefit from 
LDC's  fastmath (or similar). It's just a thought. You can shoot 
me some example that you think should vectorize.

Anyway, I think you are mostly interested in API feedback, 
implementation optimization can come later.

(others: let's stop depressing Manu further and move all ddoc and 
other irrelevant discussions to other topics! :)
Sep 16 2016