www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - colour lib needs reviewers

reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
I think I'm about as happy with my colour lib as I'm going to be.
It really needs reviews.

I added packed-RGB support, including weird micro-float and
shared-exponent formats.
They're important for interacting with any real-time rendering libraries.
There is only one texture format I'm aware of that isn't supported,
and that is u7e3 floats, only available on Xbox360 ;)

Chromatic adaptation functions are in.
I've done a tidy/reorg pass.

I'm not sure what else should be in the scope of this lib.

PR: https://github.com/dlang/phobos/pull/2845
dub: https://code.dlang.org/packages/color
docs: http://dtest.thecybershadow.net/artifact/website-04a64e024cf75be39700bebd3a50d26f6c7bd163-7185c8ec7b15c9e785880cab6d512e6f/web/library-prerelease/std/experimental/color.html

- Manu
Sep 11 2016
next sibling parent reply Guillaume Chatelet <chatelet.guillaume gmail.com> writes:
On Monday, 12 September 2016 at 04:14:27 UTC, Manu wrote:
 I think I'm about as happy with my colour lib as I'm going to 
 be. It really needs reviews.

 I added packed-RGB support, including weird micro-float and
 shared-exponent formats.
 They're important for interacting with any real-time rendering 
 libraries.
 There is only one texture format I'm aware of that isn't 
 supported,
 and that is u7e3 floats, only available on Xbox360 ;)

 Chromatic adaptation functions are in.
 I've done a tidy/reorg pass.

 I'm not sure what else should be in the scope of this lib.

 PR: https://github.com/dlang/phobos/pull/2845
 dub: https://code.dlang.org/packages/color
 docs: 
 http://dtest.thecybershadow.net/artifact/website-04a64e024cf75be39700bebd3a50d26f6c7bd163-7185c8ec7b15c9e785880cab6d512e6f/web/library-prerelease/std/experimental/color.html

 - Manu
Looks pretty good to me. Quick question though. Do you handle endianness? DPX might be a dead file format but it's main usage relies on 10bits packed RGB into 32 bits (last two bits are discarded or alpha) but it can be either little or big endian. Good job on the library though!
Sep 12 2016
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
I thought about it, but it added (even more) complexity, and I'm not
really sure it's the MO of a color lib to worry about the endian-ness
of storage.

I could support "_le"/"_be" suffix on the PackedRGB format string, but
then you have to also specify the word size, ie, is the packed color
32bit words (eg, 11,11,10) or 16bit words (16f,16f,16f,16f)?
It's only really practically applicable to PackedRGB, so I wonder why
just one color type should worry about it, but not all the others...

Consider an unpacked RGB with 16bit elements... that may also suffer
endian problems, but RGB is a runtime type, not a storage type, which
means it doesn't have a pack/unpack process... so there's no
meaningful palace to do it.

On 12 September 2016 at 18:09, Guillaume Chatelet via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 12 September 2016 at 04:14:27 UTC, Manu wrote:
 I think I'm about as happy with my colour lib as I'm going to be. It
 really needs reviews.

 I added packed-RGB support, including weird micro-float and
 shared-exponent formats.
 They're important for interacting with any real-time rendering libraries.
 There is only one texture format I'm aware of that isn't supported,
 and that is u7e3 floats, only available on Xbox360 ;)

 Chromatic adaptation functions are in.
 I've done a tidy/reorg pass.

 I'm not sure what else should be in the scope of this lib.

 PR: https://github.com/dlang/phobos/pull/2845
 dub: https://code.dlang.org/packages/color
 docs:
 http://dtest.thecybershadow.net/artifact/website-04a64e024cf75be39700bebd3a50d26f6c7bd163-7185c8ec7b15c9e785880cab6d512e6f/web/library-prerelease/std/experimental/color.html

 - Manu
Looks pretty good to me. Quick question though. Do you handle endianness? DPX might be a dead file format but it's main usage relies on 10bits packed RGB into 32 bits (last two bits are discarded or alpha) but it can be either little or big endian. Good job on the library though!
Sep 12 2016
parent reply Guillaume Chatelet <chatelet.guillaume gmail.com> writes:
On Monday, 12 September 2016 at 09:09:29 UTC, Manu wrote:
 I thought about it, but it added (even more) complexity, and 
 I'm not really sure it's the MO of a color lib to worry about 
 the endian-ness of storage.

 I could support "_le"/"_be" suffix on the PackedRGB format 
 string, but
 then you have to also specify the word size, ie, is the packed 
 color
 32bit words (eg, 11,11,10) or 16bit words (16f,16f,16f,16f)?
 It's only really practically applicable to PackedRGB, so I 
 wonder why
 just one color type should worry about it, but not all the 
 others...

 Consider an unpacked RGB with 16bit elements... that may also 
 suffer endian problems, but RGB is a runtime type, not a 
 storage type, which means it doesn't have a pack/unpack 
 process... so there's no meaningful palace to do it.
Yep I totally understand the rationale - I've been there too :) I don't think it belong to Color but maybe Buffer(1) could be templated on endianness? Buffer is indeed the place where you deal with storage. If you don't want to add it to the library I think you should add a paragraph in the documentation explaining why it's not in there and how people can overcome it (copy + change endianness or use an adapter?). Also, it would be nice to have some code snippets in the documentation. Last thing, for the white points you didn't specify which CIE standard observer(2) version you used: CIE 1931 2° Standard Observer or CIE 1964 10° Standard Observer? IIRC the standard illuminant are not the same depending on the observer. 1. http://dtest.thecybershadow.net/artifact/website-04a64e024cf75be39700bebd3a50d26f6c7bd163-7185c8ec7b15c9e785880cab6d512e6f/web/library-prerelease/std/experimental/color/packedrgb/buffer.html 2. https://en.wikipedia.org/wiki/CIE_1931_color_space#CIE_standard_observer
Sep 12 2016
parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 12 September 2016 at 21:13, Guillaume Chatelet via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 12 September 2016 at 09:09:29 UTC, Manu wrote:
 I thought about it, but it added (even more) complexity, and I'm not
 really sure it's the MO of a color lib to worry about the endian-ness of
 storage.

 I could support "_le"/"_be" suffix on the PackedRGB format string, but
 then you have to also specify the word size, ie, is the packed color
 32bit words (eg, 11,11,10) or 16bit words (16f,16f,16f,16f)?
 It's only really practically applicable to PackedRGB, so I wonder why
 just one color type should worry about it, but not all the others...

 Consider an unpacked RGB with 16bit elements... that may also suffer
 endian problems, but RGB is a runtime type, not a storage type, which means
 it doesn't have a pack/unpack process... so there's no meaningful palace to
 do it.
Yep I totally understand the rationale - I've been there too :) I don't think it belong to Color but maybe Buffer(1) could be templated on endianness? Buffer is indeed the place where you deal with storage. If you don't want to add it to the library I think you should add a paragraph in the documentation explaining why it's not in there and how people can overcome it (copy + change endianness or use an adapter?). Also, it would be nice to have some code snippets in the documentation. Last thing, for the white points you didn't specify which CIE standard observer(2) version you used: CIE 1931 2° Standard Observer or CIE 1964 10° Standard Observer? IIRC the standard illuminant are not the same depending on the observer.
That's very true. The data is all CIE 1931 reference. I haven't seen deployed use of the updated standard... I don't know why.
Sep 12 2016
prev sibling next sibling parent Ilya Yaroshenko <ilyayaroshenko gmail.com> writes:
On Monday, 12 September 2016 at 04:14:27 UTC, Manu wrote:
 I think I'm about as happy with my colour lib as I'm going to 
 be. It really needs reviews.

 [...]
Looks great! 1. packedrgb has not enough docs to understand what it is. 2. colorFromString should have scope argument. - Ilya
Sep 12 2016
prev sibling next sibling parent reply Basile B. <b2.temp gmx.com> writes:
On Monday, 12 September 2016 at 04:14:27 UTC, Manu wrote:
 I think I'm about as happy with my colour lib as I'm going to 
 be. It really needs reviews.

 I'm not sure what else should be in the scope of this lib.

 PR: https://github.com/dlang/phobos/pull/2845
 dub: https://code.dlang.org/packages/color
 docs: 
 http://dtest.thecybershadow.net/artifact/website-04a64e024cf75be39700bebd3a50d26f6c7bd163-7185c8ec7b15c9e785880cab6d512e6f/web/library-prerelease/std/experimental/color.html

 [...]
 - Manu
1. Maybe that the each package ddoc comment should mention that everything is always safe nothrow nogc...For me it wasn't directly clear: - seen that there's opCast: Q: safe ? - seen in the sources that opCast call a templatized function - seen that the unittest for this function are not annotated - then only seen that the leading " safe nothrow nogc:" at the beg of the package. 2. Q: is there anything to convert a color to grey scale ?
Sep 12 2016
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 12 September 2016 at 19:30, Basile B. via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 12 September 2016 at 04:14:27 UTC, Manu wrote:
 I think I'm about as happy with my colour lib as I'm going to be. It
 really needs reviews.

 I'm not sure what else should be in the scope of this lib.

 PR: https://github.com/dlang/phobos/pull/2845
 dub: https://code.dlang.org/packages/color
 docs:
 http://dtest.thecybershadow.net/artifact/website-04a64e024cf75be39700bebd3a50d26f6c7bd163-7185c8ec7b15c9e785880cab6d512e6f/web/library-prerelease/std/experimental/color.html

 [...]
 - Manu
1. Maybe that the each package ddoc comment should mention that everything is always safe nothrow nogc...For me it wasn't directly clear: - seen that there's opCast: Q: safe ? - seen in the sources that opCast call a templatized function - seen that the unittest for this function are not annotated - then only seen that the leading " safe nothrow nogc:" at the beg of the package.
How do you think this should this look? (it's best to leave these comments on the PR)
 2. Q: is there anything to convert a color to grey scale ?
This isn't a short-answer question :) .. There are many ways depending on the job. A simple way might be: L8 grey = cast(L8)RGB8(r, g, b); // L8 is luminance only, casting will do the right thing This is the 'normal' way, but there are better ways... A superior way if you want a top-quality result would be to use the Lab luminance value, since Lab is perceptually uniform, when you desaturate, it maintains the colors relative luminance correctly, something like: Lab!float lab = cast(Lab!float)RGB8(r, g, b); // cast your color to Lab lab.a = 0; // eliminate saturation on a axis lab.b = 0; // eliminate saturation on b axis // lab.L is now pure luminance RGB8 c = cast(RGB8)lab; // cast it back to RGB, and your RGB will be greyscale, but with more perceptually correct conversion. Lots of pow's in that conversion, so it's slower. This link demonstrates some competing techniques: https://en.wikipedia.org/wiki/HSL_and_HSV#/media/File:Fire-breather_CIELAB_L*.jpg Press left/right to cycle through them. The first method (cast to L8) is: Rec. 601 luma Y′. (or more accurately, my code above would be: sRGB luma Y') The second method (Lab.L) is: CIELAB L*. L* performs much better in low light.
Sep 12 2016
parent reply John Colvin <john.loughran.colvin gmail.com> writes:
On Monday, 12 September 2016 at 13:04:49 UTC, Manu wrote:
 2. Q: is there anything to convert a color to grey scale ?
This isn't a short-answer question :) .. There are many ways depending on the job. A simple way might be: L8 grey = cast(L8)RGB8(r, g, b); // L8 is luminance only, casting will do the right thing This is the 'normal' way, but there are better ways... A superior way if you want a top-quality result would be to use the Lab luminance value, since Lab is perceptually uniform, when you desaturate, it maintains the colors relative luminance correctly, something like: Lab!float lab = cast(Lab!float)RGB8(r, g, b); // cast your color to Lab lab.a = 0; // eliminate saturation on a axis lab.b = 0; // eliminate saturation on b axis // lab.L is now pure luminance RGB8 c = cast(RGB8)lab; // cast it back to RGB, and your RGB will be greyscale, but with more perceptually correct conversion. Lots of pow's in that conversion, so it's slower. This link demonstrates some competing techniques: https://en.wikipedia.org/wiki/HSL_and_HSV#/media/File:Fire-breather_CIELAB_L*.jpg Press left/right to cycle through them. The first method (cast to L8) is: Rec. 601 luma Y′. (or more accurately, my code above would be: sRGB luma Y') The second method (Lab.L) is: CIELAB L*. L* performs much better in low light.
Is there some uniform policy w.r.t. casts here? If I take one colour type and cast it to another will it always do something unambiguously right or not compile? Seems to me that a function that takes a target type would be more obvious for the user. I think the multiple different meanings of cast are a mistake in D. We can't go back now, but I wish there was only reinterpet_cast and everything else was a function.
Sep 12 2016
parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 13 September 2016 at 01:30, John Colvin via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 12 September 2016 at 13:04:49 UTC, Manu wrote:
 2. Q: is there anything to convert a color to grey scale ?
This isn't a short-answer question :) .. There are many ways depending on the job. A simple way might be: L8 grey = cast(L8)RGB8(r, g, b); // L8 is luminance only, casting will do the right thing This is the 'normal' way, but there are better ways... A superior way if you want a top-quality result would be to use the Lab luminance value, since Lab is perceptually uniform, when you desaturate, it maintains the colors relative luminance correctly, something like: Lab!float lab = cast(Lab!float)RGB8(r, g, b); // cast your color to Lab lab.a = 0; // eliminate saturation on a axis lab.b = 0; // eliminate saturation on b axis // lab.L is now pure luminance RGB8 c = cast(RGB8)lab; // cast it back to RGB, and your RGB will be greyscale, but with more perceptually correct conversion. Lots of pow's in that conversion, so it's slower. This link demonstrates some competing techniques: https://en.wikipedia.org/wiki/HSL_and_HSV#/media/File:Fire-breather_CIELAB_L*.jpg Press left/right to cycle through them. The first method (cast to L8) is: Rec. 601 luma Y′. (or more accurately, my code above would be: sRGB luma Y') The second method (Lab.L) is: CIELAB L*. L* performs much better in low light.
Is there some uniform policy w.r.t. casts here? If I take one colour type and cast it to another will it always do something unambiguously right or not compile?
I think, yes? So, the logic here is, if you cast, and then cast back, you will have the same thing barring loss of precision. Casts in this lib just change format (+precision), they don't change value at all. I feel it's compatible with casting an int to a float... it's the same sort of conversion.
 Seems to me that a function that takes a target type would be
 more obvious for the user.
Really? I like the cast this way. There is an explicit function in package.d though...
Sep 12 2016
prev sibling next sibling parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Mon, 12 Sep 2016 14:14:27 +1000
schrieb Manu via Digitalmars-d <digitalmars-d puremagic.com>:

 I think I'm about as happy with my colour lib as I'm going to be.
 It really needs reviews.
 
 I added packed-RGB support, including weird micro-float and
 shared-exponent formats.
 They're important for interacting with any real-time rendering libraries.
 There is only one texture format I'm aware of that isn't supported,
 and that is u7e3 floats, only available on Xbox360 ;)
 
 Chromatic adaptation functions are in.
 I've done a tidy/reorg pass.
 
 I'm not sure what else should be in the scope of this lib.
 
 PR: https://github.com/dlang/phobos/pull/2845
 dub: https://code.dlang.org/packages/color
 docs: http://dtest.thecybershadow.net/artifact/website-04a64e024cf75be39700bebd3a50d26f6c7bd163-7185c8ec7b15c9e785880cab6d512e6f/web/library-prerelease/std/experimental/color.html
 
 - Manu
Nice work. What's u7e3? 3*10-bit-floats + 2-bit padding? In the example for convert color you show that converting from a color space without alpha (XYZ) to one with alpha (RGBA) turns the pixel transparent. I believe you meant this as something to lookout for when converting colors. What's the reasoning for not setting maximum alpha as a more useful and natural default? Cool stuff there in the colorspace module, to get started with getting photos or DV/MPEG into a linear color space. Should BT.709 also have prefab conversion methods? It would seem HD video is more wide spread nowadays. There is a typo in "Function that converts a linear luminance to gamme space." (colorspace.d) I like how RGB color spaces are described as primaries in xyY + gamma functions. It seems inconsistent that you can specify all kinds of exponents when the exponent is shared but are forced to 5 bit exponents when mixed into each component, while the mantissa can still be adjusted. Would m7e3 have worked? (With f* being the IEEE standard formats.) The final package should contain a fair amount of documentation, in my opinion also on the topics of sRGB vs. linear and what "hybrid gamma" is. What is your stance on pixel formats that are arranged in planes? I assume that e.g. YUV is handled by a single channel "sRGB luminance" + 2 signed 8-bit color planes? It may be worth adding some hints in that direction. All in all good work with a scientific feel to it. -- Marco
Sep 12 2016
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 12 September 2016 at 21:28, Marco Leise via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 Am Mon, 12 Sep 2016 14:14:27 +1000
 schrieb Manu via Digitalmars-d <digitalmars-d puremagic.com>:

 I think I'm about as happy with my colour lib as I'm going to be.
 It really needs reviews.

 I added packed-RGB support, including weird micro-float and
 shared-exponent formats.
 They're important for interacting with any real-time rendering libraries.
 There is only one texture format I'm aware of that isn't supported,
 and that is u7e3 floats, only available on Xbox360 ;)

 Chromatic adaptation functions are in.
 I've done a tidy/reorg pass.

 I'm not sure what else should be in the scope of this lib.

 PR: https://github.com/dlang/phobos/pull/2845
 dub: https://code.dlang.org/packages/color
 docs: http://dtest.thecybershadow.net/artifact/website-04a64e024cf75be39700bebd3a50d26f6c7bd163-7185c8ec7b15c9e785880cab6d512e6f/web/library-prerelease/std/experimental/color.html

 - Manu
Nice work. What's u7e3? 3*10-bit-floats + 2-bit padding?
Yup.
 In the example for convert color you show that converting from
 a color space without alpha (XYZ) to one with alpha (RGBA)
 turns the pixel transparent. I believe you meant this as
 something to lookout for when converting colors. What's the
 reasoning for not setting maximum alpha as a more useful and
 natural default?
I flip-flopped on this multiple times. It's not so simple. 1. Alpha doesn't necessarily mean transparently, that's just one (extremely common) use 2. 1 is the multiplication identity, but 0 is the additive identity. I cant find either addition or multiplication to take precedence over eachother. 3. With those 2 points above already making me feel a bit worried about it, I think that appearing numbers out of nowhere is just generally bad, so I went with a=0. If alpha defaults to 1, then a*b works as expected (object does not become transparent by this operation). If alpha defaults to 0, then b-a works as expected (object does not become transparent by this operation). With that in mind, you realise alpha will *always* require explicit handling... so I think 0 is a better default at that stage. It may be possible to convince me otherwise ;)
 Cool stuff there in the colorspace module, to get started with
 getting photos or DV/MPEG into a linear color space. Should
 BT.709 also have prefab conversion methods? It would seem HD
 video is more wide spread nowadays.
BT.709's gamma func isn't changed from 601. Rec.2020 introduced a higher precision version.
 There is a typo in "Function that converts a linear luminance
 to gamme space." (colorspace.d)
Thx. But please log comments like this on the PR so I don't lose track of them? :)
 I like how RGB color spaces are described as primaries in xyY
 + gamma functions.
Seems natural :) So, I was initially planning to make that struct the RGB template argument, but then I realised that would make the symbols gigantic (heaps of CS data in the symbol names), so I introduced the enum... sadly though, this means you can't define a custom colour without introducing a new custom-colour type to do what I originally intended. I'm upset about this either way... I think I should leave it how it is... custom colours are probably more rare than gigantic symbol names ;)
 It seems inconsistent that you can specify all kinds of
 exponents when the exponent is shared but are forced to 5 bit
 exponents when mixed into each component, while the mantissa
 can still be adjusted. Would m7e3 have worked? (With f* being
 the IEEE standard formats.)
I was thinking that, and mentioned as such in the comments near those lines ;) Thing is, there are no realtime small floats with exponent != 5bits except the unique 7e3 format on xbox 360. I can extend this in an unbreaking way. I might: f## = ieee-like, s##e# = signed float, (u)##e# = unsgned float. It's just pretty verbose is all: "rgb_11e5_11e5_10e5". I mean, it's fine... but maybe people wouldn't find this intuitive? Many programmers don't know what an 'exponent' or 'mantissa' is... :/
 The final package should contain a fair amount of
 documentation, in my opinion also on the topics of sRGB vs.
 linear and what "hybrid gamma" is.
Yeah, but is it really my job to teach someone about colour space? Wikipedia can do that much better than I can... :/ I agree, but I don't really know where to draw the line, or how to organise it. Perhaps people with strong opinions on how this should be presented can create a PR?
 What is your stance on pixel formats that are arranged in
 planes? I assume that e.g. YUV is handled by a single channel
 "sRGB luminance" + 2 signed 8-bit color planes? It may be
 worth adding some hints in that direction.
I haven't supported YUV, but it is as you say. Nitpick: most of those YUV sinals are probably broadcast signals, with rec.601 luminance ;) YUV is a pretty big family, I'll do it if it's requested... There are a lot of micro-permutations. It's easy to get carried away (ie; as you can see in HSx). I don't think 'types' can really express bitplanes well. We need to express array operations to support them. I want to extend this to support efficient array operations, but I don't really know how to work this into D's ranges efficiently, and my other topic on the matter died before anything worthwhile emerged.
 All in all good work with a scientific feel to it.
Thanks, I struggled with this balance a lot. I'm fairly happy with it now.
Sep 12 2016
parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Tue, 13 Sep 2016 00:37:22 +1000
schrieb Manu via Digitalmars-d <digitalmars-d puremagic.com>:
 I flip-flopped on this multiple times.
 It's not so simple.
 1. Alpha doesn't necessarily mean transparently, that's just one
 (extremely common) use
 2. 1 is the multiplication identity, but 0 is the additive identity. I
 cant find either addition or multiplication to take precedence over
 eachother.
 3. With those 2 points above already making me feel a bit worried
 about it, I think that appearing numbers out of nowhere is just
 generally bad, so I went with a=0.
 If alpha defaults to 1, then a*b works as expected (object does not
 become transparent by this operation).
 If alpha defaults to 0, then b-a works as expected (object does not
 become transparent by this operation).
Additive color components can work like that, but the alpha component just doesn't represent an amount of light. Don't try too hard to make the arithmetics make sense, when the practical use lies in 100% alpha by default. Converting every color to full transparent is not helpful when you want to save a 24-bit BMP as 32-bit PNG or get an RGBA color from an RGB texture. I don't know who declared that 100% alpha means opaque. For what it's worth it could have been the other way around and all the (1-a) and (a) in blending functions swapped. And a blending function is what is needed here to give any meaning at all to arithmetics on alpha. An additive one may be: color = color_dst + color_src * alpha_src alpha = alpha_dst
 With that in mind, you realise alpha will *always* require explicit
 handling... so I think 0 is a better default at that stage. It may be
 possible to convince me otherwise ;)
I fully agree with the first line, I just think that for practical uses alpha is considered to be 100% in pixel formats without alpha channel. Otherwise the most popular blending functions would do nothing on RGB colors. SDL for example also uses 100% alpha by default: "If the specified pixel format has an alpha component it will be returned as all 1 bits (fully opaque)."
 Cool stuff there in the colorspace module, to get started with
 getting photos or DV/MPEG into a linear color space. Should
 BT.709 also have prefab conversion methods? It would seem HD
 video is more wide spread nowadays.  
BT.709's gamma func isn't changed from 601. Rec.2020 introduced a higher precision version.
 There is a typo in "Function that converts a linear luminance
 to gamme space." (colorspace.d)  
Thx. But please log comments like this on the PR so I don't lose track of them? :)
Will do!
 It seems inconsistent that you can specify all kinds of
 exponents when the exponent is shared but are forced to 5 bit
 exponents when mixed into each component, while the mantissa
 can still be adjusted. Would m7e3 have worked? (With f* being
 the IEEE standard formats.)  
I was thinking that, and mentioned as such in the comments near those lines ;) Thing is, there are no realtime small floats with exponent != 5bits except the unique 7e3 format on xbox 360. I can extend this in an unbreaking way. I might: f## = ieee-like, s##e# = signed float, (u)##e# = unsgned float. It's just pretty verbose is all: "rgb_11e5_11e5_10e5". I mean, it's fine... but maybe people wouldn't find this intuitive? Many programmers don't know what an 'exponent' or 'mantissa' is... :/
Yeah I thought that the common formats would be the IEEE ones, especially f16 and that anything less are very niche custom formats that come and go and need verbose descriptions. Well, either way works.
 The final package should contain a fair amount of
 documentation, in my opinion also on the topics of sRGB vs.
 linear and what "hybrid gamma" is.  
Yeah, but is it really my job to teach someone about colour space? Wikipedia can do that much better than I can... :/ I agree, but I don't really know where to draw the line, or how to organise it. Perhaps people with strong opinions on how this should be presented can create a PR?
Alright, but hybrid gamma is really not something that can be googled. Or rather I end up at Toyota's Gamma Hybrid product page. :D And as always, don't let the turkeys get you down. -- Marco
Sep 12 2016
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 13 September 2016 at 07:00, Marco Leise via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 Am Tue, 13 Sep 2016 00:37:22 +1000
 schrieb Manu via Digitalmars-d <digitalmars-d puremagic.com>:
 I flip-flopped on this multiple times.
 It's not so simple.
 1. Alpha doesn't necessarily mean transparently, that's just one
 (extremely common) use
 2. 1 is the multiplication identity, but 0 is the additive identity. I
 cant find either addition or multiplication to take precedence over
 eachother.
 3. With those 2 points above already making me feel a bit worried
 about it, I think that appearing numbers out of nowhere is just
 generally bad, so I went with a=0.
 If alpha defaults to 1, then a*b works as expected (object does not
 become transparent by this operation).
 If alpha defaults to 0, then b-a works as expected (object does not
 become transparent by this operation).
Additive color components can work like that, but the alpha component just doesn't represent an amount of light. Don't try too hard to make the arithmetics make sense, when the practical use lies in 100% alpha by default.
I understand the argument, I have stressed over this to no end. I still think it's just not 'right'. :/
 Converting every
 color to full transparent is not helpful when you want to save
 a 24-bit BMP as 32-bit PNG or get an RGBA color from an RGB
 texture.
What is the worth of storing alpha data if it's uniform 0xFF anyway? It sounds like you mean rgbx, not rgba (ie, 32bit, but no alpha). There should only be an alpha channel if there's actually alpha data... right?
 I don't know who declared that 100% alpha means opaque. For
 what it's worth it could have been the other way around and
 all the (1-a) and (a) in blending functions swapped.
 And a blending function is what is needed here to give any
 meaning at all to arithmetics on alpha. An additive one may be:

   color = color_dst + color_src * alpha_src
   alpha = alpha_dst
I thought about adding blend's, but I stopped short on that. I think that's firmly entering image processing territory, and I felt that was one step beyond the MO of this particular lib... no? Blending opens up a whole world.
 With that in mind, you realise alpha will *always* require explicit
 handling... so I think 0 is a better default at that stage. It may be
 possible to convince me otherwise ;)
I fully agree with the first line, I just think that for practical uses alpha is considered to be 100% in pixel formats without alpha channel.
'rgbx' is for image formats without alpha channels...
 Otherwise the most popular blending
 functions would do nothing on RGB colors.
'rgbx' colors fed through a blend function would always properly imply alpha=1.
 SDL for example also uses 100% alpha by default: "If the
 specified pixel format has an alpha component it will be
 returned as all 1 bits (fully opaque)."
From which functions? Link me?
I'd love to see more precedents.
 It seems inconsistent that you can specify all kinds of
 exponents when the exponent is shared but are forced to 5 bit
 exponents when mixed into each component, while the mantissa
 can still be adjusted. Would m7e3 have worked? (With f* being
 the IEEE standard formats.)
I was thinking that, and mentioned as such in the comments near those lines ;) Thing is, there are no realtime small floats with exponent != 5bits except the unique 7e3 format on xbox 360. I can extend this in an unbreaking way. I might: f## = ieee-like, s##e# = signed float, (u)##e# = unsgned float. It's just pretty verbose is all: "rgb_11e5_11e5_10e5". I mean, it's fine... but maybe people wouldn't find this intuitive? Many programmers don't know what an 'exponent' or 'mantissa' is... :/
Yeah I thought that the common formats would be the IEEE ones, especially f16 and that anything less are very niche custom formats that come and go and need verbose descriptions. Well, either way works.
The verbose idea I said above is my best idea. I'll roll with that if enough people agree. There was the other fellow who posted before who feels like it's already too complicated for him though. I've been very carefully trying to put the complicated bits just one level back from arms reach, so a casual user isn't confronted with those details. Putting that complexity in the format string makes it very user-facing. That's the part I'm not wild about.
 The final package should contain a fair amount of
 documentation, in my opinion also on the topics of sRGB vs.
 linear and what "hybrid gamma" is.
Yeah, but is it really my job to teach someone about colour space? Wikipedia can do that much better than I can... :/ I agree, but I don't really know where to draw the line, or how to organise it. Perhaps people with strong opinions on how this should be presented can create a PR?
Alright, but hybrid gamma is really not something that can be googled. Or rather I end up at Toyota's Gamma Hybrid product page. :D
True. I'm not even sure what the technical term for this sort of gamma function is... I just made that up! :/ As Walter and others have asked, I'll have to start adding links to reference material I guess, although that still feels really weird to me for some reason.
Sep 12 2016
next sibling parent reply Random D user <no email.com> writes:
On Tuesday, 13 September 2016 at 02:00:44 UTC, Manu wrote:
 On 13 September 2016 at 07:00, Marco Leise via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:
 Am Tue, 13 Sep 2016 00:37:22 +1000
 schrieb Manu via Digitalmars-d <digitalmars-d puremagic.com>:
 Alright, but hybrid gamma is really not something that can be 
 googled. Or rather I end up at Toyota's Gamma Hybrid product 
 page. :D
True. I'm not even sure what the technical term for this sort of gamma function is... I just made that up! :/ As Walter and others have asked, I'll have to start adding links to reference material I guess, although that still feels really weird to me for some reason.
FWIW I stumbled into this while double-checking HDR standards for my previous post. (I'm not a HDR expert, only somewhat interested because it's the future of displays/graphics) https://en.wikipedia.org/wiki/Hybrid_Log-Gamma
Sep 13 2016
parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 14 September 2016 at 04:25, Random D user via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Tuesday, 13 September 2016 at 02:00:44 UTC, Manu wrote:
 On 13 September 2016 at 07:00, Marco Leise via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 Am Tue, 13 Sep 2016 00:37:22 +1000
 schrieb Manu via Digitalmars-d <digitalmars-d puremagic.com>:
 Alright, but hybrid gamma is really not something that can be googled. Or
 rather I end up at Toyota's Gamma Hybrid product page. :D
True. I'm not even sure what the technical term for this sort of gamma function is... I just made that up! :/ As Walter and others have asked, I'll have to start adding links to reference material I guess, although that still feels really weird to me for some reason.
FWIW I stumbled into this while double-checking HDR standards for my previous post. (I'm not a HDR expert, only somewhat interested because it's the future of displays/graphics) https://en.wikipedia.org/wiki/Hybrid_Log-Gamma
Perfect, thanks!
Sep 13 2016
prev sibling next sibling parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Tue, 13 Sep 2016 12:00:44 +1000
schrieb Manu via Digitalmars-d <digitalmars-d puremagic.com>:

 What is the worth of storing alpha data if it's uniform 0xFF anyway?
 It sounds like you mean rgbx, not rgba (ie, 32bit, but no alpha).
 There should only be an alpha channel if there's actually alpha data... r=
ight? I don't mean RGBX. JavaScript's canvas works that way for example. I.e. the only pixel format is RGBA for simplicity's sake and I'm not surprised it actually draws something if I load it with a 24-bit graphic. ;)
 [=E2=80=A6] An additive one may be:

   color =3D color_dst + color_src * alpha_src
   alpha =3D alpha_dst =20
=20 I thought about adding blend's, but I stopped short on that. I think that's firmly entering image processing territory, and I felt that was one step beyond the MO of this particular lib... no? Blending opens up a whole world.
I agree with that decision, and that it entails that arithmetics are undefined for alpha channels. :-( Yeah bummer. The idea that basic (saturating) arithmetics work on colors is a great simplification that works for the most part, but let's be fair, multiplying two HSV colors isn't exactly going to yield a well defined hue either, just as multiplying two angles doesn't give you a new angle. I.e.: http://math.stackexchange.com/a/47880
 [=E2=80=A6]
[=E2=80=A6] From which functions? Link me? =20 I'd love to see more precedents.
Yep, that's better than arguing :) So here are all graphics APIs I know and what they do when they encounter colors without alpha and need a default value: SDL: https://wiki.libsdl.org/SDL_MapRGB "If the specified pixel format has an alpha component it will be returned a= s all 1 bits (fully opaque)." Allegro: https://github.com/liballeg/allegro5/blob/master/include/allegro5/internal/= aintern_pixels.h#L59 (No docs, just source code that defaults to 255 for alpha when converting a color from a bitmap with non-alpha pixel format to an ALLEGRO_COLOR.) Cairo: https://www.cairographics.org/manual/cairo-cairo-t.html#cairo-set-source-rgb "Sets the source pattern within cr to an opaque color." Microsoft GDI+: https://msdn.microsoft.com/de-de/library/windows/desktop/ms536255%28v=3Dvs.= 85%29.aspx "The default value of the alpha component for this Color object is 255." Gnome GDK: https://developer.gnome.org/gdk-pixbuf/2.33/gdk-pixbuf-Utilities.html#gdk-p= ixbuf-add-alpha "[=E2=80=A6] the alpha channel is initialized to 255 (full opacity)." Qt: http://doc.qt.io/qt-4.8/qcolor.html#alpha-blended-drawing "By default, the alpha-channel is set to 255 (opaque)." OpenGL: https://www.opengl.org/sdk/docs/man2/xhtml/glColor.xml "glColor3 variants specify new red, green, and blue values explicitly and set the current alpha value to 1.0 (full intensity) implicitly." (Note: The color can be queried and shows a=3D1.0 without blending operations setting it internally if needed.) Java (AWT): https://docs.oracle.com/javase/7/docs/api/java/awt/Color.html#Color%28int,%= 20boolean%29 "If the hasalpha argument is false, alpha is defaulted to 255." Apple's Quartz does not seem to provide color space conversions and always requires the user to give the alpha value for new colors, so there is no default: https://developer.apple.com/library/tvos/documentation/GraphicsImaging/Refe= rence/CGColor/index.html#//apple_ref/c/func/CGColorCreate One thing I noticed is that many of those strictly separate color spaces from alpha as concepts. For example in Quartz *all* color spaces have alpha. In Allegro color space conversions are ignorant of alpha. That begs the question what should happen when you convert RGBA to a HLx color space and back to RGBA. Can you retain the alpha value? CSS3 for example has HSLA colors that raise the bar a bit. --=20 Marco
Sep 13 2016
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 14 September 2016 at 04:34, Marco Leise via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 Am Tue, 13 Sep 2016 12:00:44 +1000
 schrieb Manu via Digitalmars-d <digitalmars-d puremagic.com>:

 What is the worth of storing alpha data if it's uniform 0xFF anyway?
 It sounds like you mean rgbx, not rgba (ie, 32bit, but no alpha).
 There should only be an alpha channel if there's actually alpha data... right?
I don't mean RGBX. JavaScript's canvas works that way for example. I.e. the only pixel format is RGBA for simplicity's sake and I'm not surprised it actually draws something if I load it with a 24-bit graphic. ;)
Given this example, isn't it the job of the image loader to populate the image with data? If you ask for an RGBA image loaded from a 24bit BMP or something, sure, it should put 0xFF in there, but I don't know if it's right that that logic belongs here...?
 […] An additive one may be:

   color = color_dst + color_src * alpha_src
   alpha = alpha_dst
I thought about adding blend's, but I stopped short on that. I think that's firmly entering image processing territory, and I felt that was one step beyond the MO of this particular lib... no? Blending opens up a whole world.
I agree with that decision, and that it entails that arithmetics are undefined for alpha channels. :-( Yeah bummer. The idea that basic (saturating) arithmetics work on colors is a great simplification that works for the most part, but let's be fair, multiplying two HSV colors isn't exactly going to yield a well defined hue either,
You'll notice I didn't add arithmetic operators to the HSx type ;) If you have HSx colors, and want to do arithmetic, cast it to RGB first.
 just as multiplying two
 angles doesn't give you a new angle. I.e.:
 http://math.stackexchange.com/a/47880
I went through a brief phase where I thought about adding an angle type (beside normint), but I felt it was overkill. I still wonder if it's the right thing to do though... some type that understands a circle, and making angle arithmetic work as expected. I like my solution of not providing arithmetic operators for HSx and LCh for now though ;)
 […]
[…] From which functions? Link me? I'd love to see more precedents.
Yep, that's better than arguing :) So here are all graphics APIs I know and what they do when they encounter colors without alpha and need a default value: SDL: https://wiki.libsdl.org/SDL_MapRGB "If the specified pixel format has an alpha component it will be returned as all 1 bits (fully opaque)." Allegro: https://github.com/liballeg/allegro5/blob/master/include/allegro5/internal/aintern_pixels.h#L59 (No docs, just source code that defaults to 255 for alpha when converting a color from a bitmap with non-alpha pixel format to an ALLEGRO_COLOR.) Cairo: https://www.cairographics.org/manual/cairo-cairo-t.html#cairo-set-source-rgb "Sets the source pattern within cr to an opaque color." Microsoft GDI+: https://msdn.microsoft.com/de-de/library/windows/desktop/ms536255%28v=vs.85%29.aspx "The default value of the alpha component for this Color object is 255." Gnome GDK: https://developer.gnome.org/gdk-pixbuf/2.33/gdk-pixbuf-Utilities.html#gdk-pixbuf-add-alpha "[…] the alpha channel is initialized to 255 (full opacity)." Qt: http://doc.qt.io/qt-4.8/qcolor.html#alpha-blended-drawing "By default, the alpha-channel is set to 255 (opaque)." OpenGL: https://www.opengl.org/sdk/docs/man2/xhtml/glColor.xml "glColor3 variants specify new red, green, and blue values explicitly and set the current alpha value to 1.0 (full intensity) implicitly." (Note: The color can be queried and shows a=1.0 without blending operations setting it internally if needed.) Java (AWT): https://docs.oracle.com/javase/7/docs/api/java/awt/Color.html#Color%28int,%20boolean%29 "If the hasalpha argument is false, alpha is defaulted to 255." Apple's Quartz does not seem to provide color space conversions and always requires the user to give the alpha value for new colors, so there is no default: https://developer.apple.com/library/tvos/documentation/GraphicsImaging/Reference/CGColor/index.html#//apple_ref/c/func/CGColorCreate
Touche! ;)
 One thing I noticed is that many of those strictly separate
 color spaces from alpha as concepts. For example in Quartz
 *all* color spaces have alpha. In Allegro color space
 conversions are ignorant of alpha. That begs the question
 what should happen when you convert RGBA to a HLx
 color space and back to RGBA. Can you retain the alpha value?
 CSS3 for example has HSLA colors that raise the bar a bit.
I've actually had the thought that I should remove alpha from RGB, and instead provide an aggregate colour type where you could make RGBA by RGB + alpha, and each element in the aggregate would follow its own natural arithmetic rules... again, it feels like massive overkill though, and RGBA 8 is just way simpler for the 99% case. That aggregate thing can appear, or be added in the future, or by users. The library and conversions are all extensible.
Sep 13 2016
parent Marco Leise <Marco.Leise gmx.de> writes:
Am Wed, 14 Sep 2016 11:36:12 +1000
schrieb Manu via Digitalmars-d <digitalmars-d puremagic.com>:

 On 14 September 2016 at 04:34, Marco Leise via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 JavaScript's canvas works that way for
 example. I.e. the only pixel format is RGBA for simplicity's
 sake and I'm not surprised it actually draws something if I
 load it with a 24-bit graphic. ;) =20
=20 Given this example, isn't it the job of the image loader to populate the image with data?2 [=E2=80=A6]
I admit is was a constructed example. You can't load an image directly into a HTML5 canvas. Shame on me.
 You'll notice I didn't add arithmetic operators to the HSx type ;)
 If you have HSx colors,  and want to do arithmetic, cast it to RGB first.
Now that you say it, yes I was wondering how the arithmetics work since I couldn't find any opBinary. =20
 I went through a brief phase where I thought about adding an angle
 type (beside normint), but I felt it was overkill.
 I still wonder if it's the right thing to do though... some type that
 understands a circle, and making angle arithmetic work as expected.
I see. I'm also wondering what equality means for a color with floating point hue 10=C2=B0 and another with hue 730=C2=B0. I guess it is ok the way it is now, because wrapping the values doesn't guarantee they map to the same value either. Floating point equality is flawed either way. --=20 Marco
Sep 14 2016
prev sibling parent Basile B. <b2.temp gmx.com> writes:
On Tuesday, 13 September 2016 at 02:00:44 UTC, Manu wrote:
 What is the worth of storing alpha data if it's uniform 0xFF 
 anyway?
He he, that's the big problem with classic image formats. When a graphic is described in term of vector, fill, stroke, gradient, layer, etc (not to name SVG) there's no more waste.
Sep 13 2016
prev sibling next sibling parent reply Guillaume Chatelet <chatelet.guillaume gmail.com> writes:
On Monday, 12 September 2016 at 04:14:27 UTC, Manu wrote:
 I think I'm about as happy with my colour lib as I'm going to 
 be. It really needs reviews.

 I added packed-RGB support, including weird micro-float and
 shared-exponent formats.
 They're important for interacting with any real-time rendering 
 libraries.
 There is only one texture format I'm aware of that isn't 
 supported,
 and that is u7e3 floats, only available on Xbox360 ;)

 Chromatic adaptation functions are in.
 I've done a tidy/reorg pass.

 I'm not sure what else should be in the scope of this lib.

 PR: https://github.com/dlang/phobos/pull/2845
 dub: https://code.dlang.org/packages/color
 docs: 
 http://dtest.thecybershadow.net/artifact/website-04a64e024cf75be39700bebd3a50d26f6c7bd163-7185c8ec7b15c9e785880cab6d512e6f/web/library-prerelease/std/experimental/color.html

 - Manu
And also, do you handle limited (16-235) to full dynamic range (0-255) conversions? It's a setting some acquisition/graphic card allow. https://wiki.videolan.org/VSG:Video:Color_washed_out/ From this page as well http://www.digitalpreservation.gov/formats/fdd/fdd000352.shtml "Video and wide range. The ITU-R specifications support values for Y, Cb, and Cr that conform to what is sometimes called "video range," "legal range," or "studio swing" levels. Expressed in terms of 8-bit tonal range values, video range has a 16-235 levels for Y and 16-240 levels for Cr and Cb. The term video range is used to contrast with "wide range" or "super white " values from 0 to 255, sometimes used when video signals are created using computer-based graphics applications."
Sep 12 2016
next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Mon, 12 Sep 2016 11:31:13 +0000
schrieb Guillaume Chatelet <chatelet.guillaume gmail.com>:

 On Monday, 12 September 2016 at 04:14:27 UTC, Manu wrote:
 I think I'm about as happy with my colour lib as I'm going to 
 be. It really needs reviews.

 I added packed-RGB support, including weird micro-float and
 shared-exponent formats.
 They're important for interacting with any real-time rendering 
 libraries.
 There is only one texture format I'm aware of that isn't 
 supported,
 and that is u7e3 floats, only available on Xbox360 ;)

 Chromatic adaptation functions are in.
 I've done a tidy/reorg pass.

 I'm not sure what else should be in the scope of this lib.

 PR: https://github.com/dlang/phobos/pull/2845
 dub: https://code.dlang.org/packages/color
 docs: 
 http://dtest.thecybershadow.net/artifact/website-04a64e024cf75be39700bebd3a50d26f6c7bd163-7185c8ec7b15c9e785880cab6d512e6f/web/library-prerelease/std/experimental/color.html

 - Manu  
And also, do you handle limited (16-235) to full dynamic range (0-255) conversions? It's a setting some acquisition/graphic card allow. https://wiki.videolan.org/VSG:Video:Color_washed_out/ [...]
Practically all video content is encoded with the limited range, from VHS to HDTV, IIRC and HiFi equipment uses it as well, only computers are not so familiar with it and when connected to a TV over HDMI one or the other may think that the other part thinks ... well colors are often washed out, in effect. On the other hand this luminance compression can apply to any color format, so it is probably best to leave it to the user to compress colors if the output is a video file. -- Marco
Sep 12 2016
prev sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 12 September 2016 at 21:31, Guillaume Chatelet via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 12 September 2016 at 04:14:27 UTC, Manu wrote:
 I think I'm about as happy with my colour lib as I'm going to be. It
 really needs reviews.

 I added packed-RGB support, including weird micro-float and
 shared-exponent formats.
 They're important for interacting with any real-time rendering libraries.
 There is only one texture format I'm aware of that isn't supported,
 and that is u7e3 floats, only available on Xbox360 ;)

 Chromatic adaptation functions are in.
 I've done a tidy/reorg pass.

 I'm not sure what else should be in the scope of this lib.

 PR: https://github.com/dlang/phobos/pull/2845
 dub: https://code.dlang.org/packages/color
 docs:
 http://dtest.thecybershadow.net/artifact/website-04a64e024cf75be39700bebd3a50d26f6c7bd163-7185c8ec7b15c9e785880cab6d512e6f/web/library-prerelease/std/experimental/color.html

 - Manu
And also, do you handle limited (16-235) to full dynamic range (0-255) conversions?
Actually, I spent quite some time stressing about dynamic range. Trouble is, there's no real format spec's relating to dynamic range, it tends to be by convention. I'm not sure what the API would look like, so I haven't approached that problem. It's very easy to do the transform yourself, I don't really think it needs to be part of the type, particularly since there's no conventional way to express it. Note also that I didn't do any YUV formats (this case you discuss is usually related to older broadcast formats)... didn't know if they'd be particularly useful to people. At least, it's not in my initial offering.
Sep 12 2016
next sibling parent Guillaume Chatelet <chatelet.guillaume gmail.com> writes:
On Monday, 12 September 2016 at 14:12:35 UTC, Manu wrote:
 On 12 September 2016 at 21:31, Guillaume Chatelet via 
 Digitalmars-d <digitalmars-d puremagic.com> wrote:
 [...]
Actually, I spent quite some time stressing about dynamic range. Trouble is, there's no real format spec's relating to dynamic range, it tends to be by convention. I'm not sure what the API would look like, so I haven't approached that problem. It's very easy to do the transform yourself, I don't really think it needs to be part of the type, particularly since there's no conventional way to express it. Note also that I didn't do any YUV formats (this case you discuss is usually related to older broadcast formats)... didn't know if they'd be particularly useful to people. At least, it's not in my initial offering.
Yes I agree, as you and Marco said: I think it's best to leave it to clients of the API.
Sep 12 2016
prev sibling parent Guillaume Piolat <first.last gmail.com> writes:
On Monday, 12 September 2016 at 14:12:35 UTC, Manu wrote:
 Note also that I didn't do any YUV formats (this case you 
 discuss is usually related to older broadcast formats)... 
 didn't know if they'd be particularly useful to people. At 
 least, it's not in my initial offering.
As someone who had to write lots of code involving such video color spaces, I'd say to don't bother. - Video engineers will write their own highly-optimized version anyway, - Dynamic range is indeed very often 16-235 / 16-240 but often with no meta-date allowing to know if this is the case, - you often don't want clamping on each intermediate results
Sep 12 2016
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/12/2016 12:14 AM, Manu via Digitalmars-d wrote:
 I think I'm about as happy with my colour lib as I'm going to be.
 It really needs reviews.
Upon a quick look it seems to have the right amount of "notes", as Emperor Joseph II would say. Well balanced, nothing jumping at you. Nice work. Regarding Phobos suitability: One thing I'd like to hear from the community is the applicability of this work. I know very little about colors (only the RGB was familiar from HTML etc, but then there was no YUL), but that doesn't mean much. The library is quite isolated from the rest of Phobos, e.g. a stronger case would be made if there were some std.experimental.canvas package which uses the color package. Or at least a std.experimental.terminal package or something. A good case for inclusion would motivate the presence of the library as a catalyst for many user-level abstractions. -- Andrei
Sep 12 2016
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 13 September 2016 at 00:37, Andrei Alexandrescu via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 09/12/2016 12:14 AM, Manu via Digitalmars-d wrote:
 I think I'm about as happy with my colour lib as I'm going to be.
 It really needs reviews.
Upon a quick look it seems to have the right amount of "notes", as Emperor Joseph II would say. Well balanced, nothing jumping at you. Nice work.
Thanks. It has taken a lot more work than I actually expected to make this not suck!
 Regarding Phobos suitability: One thing I'd like to hear from the community
 is the applicability of this work. I know very little about colors (only the
 RGB was familiar from HTML etc, but then there was no YUL), but that doesn't
 mean much. The library is quite isolated from the rest of Phobos, e.g. a
 stronger case would be made if there were some std.experimental.canvas
 package which uses the color package. Or at least a
 std.experimental.terminal package or something. A good case for inclusion
 would motivate the presence of the library as a catalyst for many user-level
 abstractions. -- Andrei
So, the main reason I wanted to push for this in phobos is because it's an enabler. There is no multimedia in phobos at all. I think that's a shame, and this is a starting point. Any conceivable image based work depends on this as foundation. If pixel format types are phobos-supplied, then D image libraries will naturally be compatible, as opposed to naturally incompatible since every image library will otherwise re-implement their own colour types (and almost always incorrectly). The thing about colours is, they're HARD to get right, but more importantly, most programmers have no idea that it's actually a hard problem and won't tend to go looking for a lib for a colour type (Ie, struct { ubyte r, g, b;}... simple! *cough*). They probably will use it if it's in phobos though for compatibility reasons. Image processing is super easy by contrast. One of my first port-of-calls after this work would be a graph/plot library... I'd use the hell out of that. I constantly want to output data visualisations! It also enables seamless interaction with realtime rendering api's. Ie, if OpenGL/D3D/Vulkan bindings support phobos supplied colour types, then image decoding libraries, and other sources of image data become easy to render without any requirement for adaptation. Rendering libs and image/video data source libs are almost always separate, independent libraries. We need to maximise probability that they are compatible out of the gate. I didn't want to touch 'images' in this work, because there are countless subjective designs to present those API's, but I don't think there's many competing designs for a colour library, so it felt it was the best and most bang-for-buck starting place overall. Also, I think this has very high synergy with ndslice. I think ndslice + color is the foundation for image processing, or at least image data transport.
Sep 12 2016
next sibling parent Adam D. Ruppe <destructionator gmail.com> writes:
On Monday, 12 September 2016 at 15:06:29 UTC, Manu wrote:
 So, the main reason I wanted to push for this in phobos is 
 because it's an enabler.
Indeed. I know there's a window library that was waiting on this work to try for Phobos. Also, worth noting that the one module I import more than any other in my libs is.... color.d. Image load/save depends on it. Windowing display depends on it. Graphing depends on one or the other, GUI depends on windowing, terminal I managed to keep independent... but it is REALLY tempting to make it depend on color too. It is a very useful base building block, and moreover, having it in Phobos means User X's graphing lib and my image libs could conceivably be able to interchange data with ease since we both use the same underlying data structure!
Sep 12 2016
prev sibling next sibling parent Edwin van Leeuwen <edder tkwsping.nl> writes:
On Monday, 12 September 2016 at 15:06:29 UTC, Manu wrote:
 One of my first port-of-calls after this work would be a 
 graph/plot library... I'd use the hell out of that. I 
 constantly want to output data visualisations!
Just for your information. ggplotd currently uses an old version of your library on the backend. I'm planning on porting it soon to the latest (dub) version.
Sep 12 2016
prev sibling next sibling parent Guillaume Chatelet <chatelet.guillaume gmail.com> writes:
On Monday, 12 September 2016 at 15:06:29 UTC, Manu wrote:
 On 13 September 2016 at 00:37, Andrei Alexandrescu via
 Regarding Phobos suitability: One thing I'd like to hear from 
 the community is the applicability of this work. I know very 
 little about colors (only the RGB was familiar from HTML etc, 
 but then there was no YUL), but that doesn't mean much. The 
 library is quite isolated from the rest of Phobos, e.g. a 
 stronger case would be made if there were some 
 std.experimental.canvas package which uses the color package. 
 Or at least a std.experimental.terminal package or something. 
 A good case for inclusion would motivate the presence of the 
 library as a catalyst for many user-level abstractions. -- 
 Andrei
So, the main reason I wanted to push for this in phobos is because it's an enabler. There is no multimedia in phobos at all. I think that's a shame, and this is a starting point. Any conceivable image based work depends on this as foundation. If pixel format types are phobos-supplied, then D image libraries will naturally be compatible, as opposed to naturally incompatible since every image library will otherwise re-implement their own colour types (and almost always incorrectly). The thing about colours is, they're HARD to get right, but more importantly, most programmers have no idea that it's actually a hard problem and won't tend to go looking for a lib for a colour type (Ie, struct { ubyte r, g, b;}... simple! *cough*). They probably will use it if it's in phobos though for compatibility reasons. Image processing is super easy by contrast. One of my first port-of-calls after this work would be a graph/plot library... I'd use the hell out of that. I constantly want to output data visualisations! It also enables seamless interaction with realtime rendering api's. Ie, if OpenGL/D3D/Vulkan bindings support phobos supplied colour types, then image decoding libraries, and other sources of image data become easy to render without any requirement for adaptation. Rendering libs and image/video data source libs are almost always separate, independent libraries. We need to maximise probability that they are compatible out of the gate. I didn't want to touch 'images' in this work, because there are countless subjective designs to present those API's, but I don't think there's many competing designs for a colour library, so it felt it was the best and most bang-for-buck starting place overall. Also, I think this has very high synergy with ndslice. I think ndslice + color is the foundation for image processing, or at least image data transport.
+1 It's simply amazing how the concept of color is well understood by everyone and yet utterly hard to understand in the details. Color is a Universe in itself! Just have a look at Metamerism(1) (colors that look the same but aren't), Human Tetrachromacy(2) (some people - mostly women - perceive more colors than others). I'm not even scratching the surface: it connects with everything: Physics, Physiology, Art, Entertainement, Evolution(3)... Having a high quality Color API in Phobos is a differentiator and an enabler: think Raytracer, image batch processing, scientific analysis, interactions with GUI or rendering APIs. I think it's an important addition to Phobos - I might be biased though :-P 1. https://en.wikipedia.org/wiki/Metamerism_(color) 2. https://en.wikipedia.org/wiki/Tetrachromacy#Humans 3. https://www.youtube.com/watch?v=qrKZBh8BL_U
Sep 12 2016
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/12/2016 8:06 AM, Manu via Digitalmars-d wrote:
 So, the main reason I wanted to push for this in phobos is because
 it's an enabler.
 There is no multimedia in phobos at all. I think that's a shame, and
 this is a starting point. Any conceivable image based work depends on
 this as foundation. If pixel format types are phobos-supplied, then D
 image libraries will naturally be compatible, as opposed to naturally
 incompatible since every image library will otherwise re-implement
 their own colour types (and almost always incorrectly).
 The thing about colours is, they're HARD to get right, but more
 importantly, most programmers have no idea that it's actually a hard
 problem and won't tend to go looking for a lib for a colour type (Ie,
 struct { ubyte r, g, b;}... simple! *cough*). They probably will use
 it if it's in phobos though for compatibility reasons.
 Image processing is super easy by contrast.

 One of my first port-of-calls after this work would be a graph/plot
 library... I'd use the hell out of that. I constantly want to output
 data visualisations!

 It also enables seamless interaction with realtime rendering api's.
 Ie, if OpenGL/D3D/Vulkan bindings support phobos supplied colour
 types, then image decoding libraries, and other sources of image data
 become easy to render without any requirement for adaptation.
 Rendering libs and image/video data source libs are almost always
 separate, independent libraries. We need to maximise probability that
 they are compatible out of the gate.

 I didn't want to touch 'images' in this work, because there are
 countless subjective designs to present those API's, but I don't think
 there's many competing designs for a colour library, so it felt it was
 the best and most bang-for-buck starting place overall.

 Also, I think this has very high synergy with ndslice. I think ndslice
 +  color is the foundation for image processing, or at least image
 data transport.
Thanks for the rationale. I'm glad to see you're planning follow-on libraries! I'd use a graph/plot library myself. Some of the rationale above should go into the documentation for the library.
Sep 12 2016
parent jmh530 <john.michael.hall gmail.com> writes:
On Monday, 12 September 2016 at 22:55:01 UTC, Walter Bright wrote:
 Thanks for the rationale. I'm glad to see you're planning 
 follow-on libraries! I'd use a graph/plot library myself. Some 
 of the rationale above should go into the documentation for the 
 library.
You might try https://github.com/BlackEdder/ggplotd
Sep 12 2016
prev sibling next sibling parent reply Guillaume Piolat <first.last gmail.com> writes:
On Monday, 12 September 2016 at 04:14:27 UTC, Manu wrote:
 I think I'm about as happy with my colour lib as I'm going to 
 be. It really needs reviews.


 - Manu
My thoughts: - I've wanted a function like colorFromString many times. It's especially nice with the added #RGBA and #RRGGBBAA syntax that eg. SVG lacks. - The knowledge encoded in this library is great in itself. - This could form the basis of the next-gen DbI image library so it is much needed. => I would certainly vote for inclusion.
Sep 12 2016
parent reply Edwin van Leeuwen <edder tkwsping.nl> writes:
On Monday, 12 September 2016 at 19:55:57 UTC, Guillaume Piolat 
wrote:
 - I've wanted a function like colorFromString many times. It's 
 especially nice with the added #RGBA and #RRGGBBAA syntax that 
 eg. SVG lacks.
What happens when the string is invalid? Does it throw an error?
Sep 12 2016
next sibling parent Guillaume Piolat <first.last gmail.com> writes:
On Monday, 12 September 2016 at 19:59:58 UTC, Edwin van Leeuwen 
wrote:
 On Monday, 12 September 2016 at 19:55:57 UTC, Guillaume Piolat 
 wrote:
 - I've wanted a function like colorFromString many times. It's 
 especially nice with the added #RGBA and #RRGGBBAA syntax that 
 eg. SVG lacks.
What happens when the string is invalid? Does it throw an error?
It throws an Exception, like it should.
Sep 12 2016
prev sibling parent reply John Colvin <john.loughran.colvin gmail.com> writes:
On Monday, 12 September 2016 at 19:59:58 UTC, Edwin van Leeuwen 
wrote:
 On Monday, 12 September 2016 at 19:55:57 UTC, Guillaume Piolat 
 wrote:
 - I've wanted a function like colorFromString many times. It's 
 especially nice with the added #RGBA and #RRGGBBAA syntax that 
 eg. SVG lacks.
What happens when the string is invalid? Does it throw an error?
It should be integrated with std.conv.parse and std.conv.to and follow their conventions.
Sep 12 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/12/2016 2:08 PM, John Colvin wrote:
 On Monday, 12 September 2016 at 19:59:58 UTC, Edwin van Leeuwen wrote:
 On Monday, 12 September 2016 at 19:55:57 UTC, Guillaume Piolat wrote:
 - I've wanted a function like colorFromString many times. It's especially
 nice with the added #RGBA and #RRGGBBAA syntax that eg. SVG lacks.
What happens when the string is invalid? Does it throw an error?
It should be integrated with std.conv.parse and std.conv.to and follow their conventions.
I'm not so sure. Follow the conventions, ok, but integrating them in - they are already large and complex.
Sep 12 2016
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 13 September 2016 at 12:05, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 9/12/2016 2:08 PM, John Colvin wrote:
 On Monday, 12 September 2016 at 19:59:58 UTC, Edwin van Leeuwen wrote:
 On Monday, 12 September 2016 at 19:55:57 UTC, Guillaume Piolat wrote:
 - I've wanted a function like colorFromString many times. It's
 especially
 nice with the added #RGBA and #RRGGBBAA syntax that eg. SVG lacks.
What happens when the string is invalid? Does it throw an error?
It should be integrated with std.conv.parse and std.conv.to and follow their conventions.
I'm not so sure. Follow the conventions, ok, but integrating them in - they are already large and complex.
std.conv.to seems to 'just work' for color type conversion. I'd like to support string -> color conversion via std.conv.to; ie: auto c = "#FFFF00".to!RGB8; Is there a simple way to do that?
Sep 12 2016
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/12/16 10:50 PM, Manu via Digitalmars-d wrote:
 std.conv.to  seems to 'just work' for color type conversion.
 I'd like to support string -> color conversion via std.conv.to; ie:
 auto c = "#FFFF00".to!RGB8;
 Is there a simple way to do that?
That was the purpose of toImpl, which I recall has been recently removed... -- Andrei
Sep 13 2016
prev sibling parent reply ketmar <ketmar ketmar.no-ip.org> writes:
On Tuesday, 13 September 2016 at 02:50:02 UTC, Manu wrote:
 std.conv.to  seems to 'just work' for color type conversion.
 I'd like to support string -> color conversion via std.conv.to; 
 ie:
 auto c = "#FFFF00".to!RGB8;
 Is there a simple way to do that?
yes. just define a ctor that accepts string, and you're there. it works both for classes and structs.
Sep 13 2016
parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 13 September 2016 at 20:19, ketmar via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Tuesday, 13 September 2016 at 02:50:02 UTC, Manu wrote:
 std.conv.to  seems to 'just work' for color type conversion.
 I'd like to support string -> color conversion via std.conv.to; ie:
 auto c = "#FFFF00".to!RGB8;
 Is there a simple way to do that?
yes. just define a ctor that accepts string, and you're there. it works both for classes and structs.
Oh... it already works! :P
Sep 13 2016
prev sibling next sibling parent reply Random D user <no email.com> writes:
On Monday, 12 September 2016 at 04:14:27 UTC, Manu wrote:
 I think I'm about as happy with my colour lib as I'm going to 
 be. It really needs reviews.

 - Manu
Hi. I'm just a random forum lurker, but here's my feedback. It needs more docs/examples for the basic usage cases (i.e. how to skip docs and start working). Now at first glance it looks like working with RGBA (most common case (with srgb)) is complicated and while reading the docs I kind of felt like I'd just write my own. Few basic things I'd be interested in (examples): 1. How to pack/unpack color into uint/rgba channels. Also can I swizzle channels directly? How would I extract a single channel from color? 2. Can I just cast an array of bytes into a RGBA array and operate on that? 3. How to scale color channels e.g. 5 bits to 8 bits and vice versa. How to convert color formats? Can I combine scale/conversion with single channel or channel swizzling? 4. Which way are RGBA format names defined ("byte array order" r is first byte or "uint order" r is in high bits)? 5. Is it the same for packed rgba formats? I'm not sure if packedrgb is a necessary distinction. Aren't they all kind of packed (small structs)? There's no guarantee that 1 byte is not just a part of a bigger field. Btw. Your RGBA8 example is a bit weird: static assert(RGBA8("0x908000FF") == RGBA8(0x80,0x00,0xFF,0x90)); I'd assume RGBA8(r,g,b,a) would be either "uint order" r(0x90), g(0x80), b(0x00), a(0xFF) or "byte array order" (le) r(0xFF), g(0x00), b(0x80), a(0x90) 6. Perhaps you should define all the usual color formats, since everybody and their cat has their own preference for rgba channel orders. In my experience all these 4 variants for RGBA8888 are pretty common: RGBA8888 BGRA8888 ARGB8888 ABGR8888 and their X and 24bit variants. For 16 bits fairly common are: RGB565 and RGBA5551, also sometimes you see one of RGBA4444 permutations (like RGBA8 above). Also L16 is pretty common for heightfields or other data images/textures. I guess there are also planar formats, but I think those are more of a Image problem space. Just as a completely random idea - How about aliases for HDR formats like HDR10 and Dolby Vision? Kind of looks like they're just combinations what you already have.
Sep 12 2016
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 13 September 2016 at 06:29, Random D user via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 12 September 2016 at 04:14:27 UTC, Manu wrote:
 I think I'm about as happy with my colour lib as I'm going to be. It
 really needs reviews.

 - Manu
Hi. I'm just a random forum lurker, but here's my feedback. It needs more docs/examples for the basic usage cases (i.e. how to skip docs and start working). Now at first glance it looks like working with RGBA (most common case (with srgb)) is complicated and while reading the docs I kind of felt like I'd just write my own.
You'll get it wrong though! Can you describe what you perceive to be hard? import std.experimental.color; RGBA8 a(0xFF808000); RGBA8 b("GoldenRodYellow"); auto r = a + b; // do normal operations Or: auto image = new RGBA8[height][width]; image[x][y] = RGBA8("Aquamarine");
 Few basic things I'd be interested in (examples):
 1. How to pack/unpack color into uint/rgba channels.
auto c = RGBA8(r, g, b, a); uint r = c.r.value; // 'r' is a 'NormalizedInt", so you need '.value' to get the raw integer value float g = cast(float)c.g; // cast to float knows about the normalized 0-1 range
 Also can I swizzle channels directly?
I could add something like: auto brg = c.swizzle!"brg"; The result would be strongly typed to the new arrangement.
 How would I extract a single channel from color?
uint r = c.r.value; // like this?
 2. Can I just cast an array of bytes into a RGBA array and operate on that?
Yes. Of course, if the image library returned an array of colors, you wouldn't have an array of raw bytes in the first place.
 3. How to scale color channels e.g. 5 bits to 8 bits and vice versa. How to
 convert color formats? Can I combine scale/conversion with single channel or
 channel swizzling?
alias CustomPackedColor = PackedRGB!("BGR_5_6_5", ubyte); // <- ordered bgr CustomPackedColor[] image = loadPackedImage(); auto unpacked = cast(RGB8)image[0]; // unpack and swizzle I'm not sure what you mean by "with single channel"?
 4. Which way are RGBA format names defined ("byte array order" r is first
 byte or "uint order" r is in high bits)?
I describe formats in byte (bit) order. It's the only reasonable way that works outside of a uint.
 5. Is it the same for packed rgba formats?
Yes.
 I'm not sure if packedrgb is a necessary distinction. Aren't they all kind
 of packed (small structs)? There's no guarantee that 1 byte is not just a
 part of a bigger field.
RGBA8 has 4 ubytes, and you can operate on them. ubyte a, b; a += b; // <- is valid D code Unpacked types can do maths. Packed types can't do maths, they need to be unpacked before you can do anything, and you don't generally want the result to be immediately packed again. If I allowed maths on PackedRGB for instance, it would be expected that the result is re-packed at the end of every operation, and that would be useless. PackedRGB is for storage, not for doing work. This is why they're distinct.
 Btw. Your RGBA8 example is a bit weird:
 static assert(RGBA8("0x908000FF") == RGBA8(0x80,0x00,0xFF,0x90));
 I'd assume RGBA8(r,g,b,a) would be either "uint order" r(0x90), g(0x80),
 b(0x00), a(0xFF) or "byte array order" (le) r(0xFF), g(0x00), b(0x80),
 a(0x90)
I'm showing that parsing colors from string's follow the web convention. I can't redefine how colours are expressed on the web for decades. I think you've just highlighted that I need to remove the '0x' notation though, that is definitely misleading! Use '#', that should be clear.
 6. Perhaps you should define all the usual color formats, since everybody
 and their cat has their own preference for rgba channel orders.
 In my experience all these 4 variants for RGBA8888 are pretty common:
 RGBA8888
 BGRA8888
 ARGB8888
 ABGR8888
 and their X and 24bit variants.
I don't think they're common at all. Only 2 are common (one of which is deprecated), the other 2 are very niche. I was trying to balance a tasteful amount of instantiations. I don't really want to provide an instantiation of every single realtime format I've ever encountered out of the box.
 For 16 bits fairly common are:
 RGB565 and RGBA5551, also sometimes you see one of RGBA4444 permutations
 (like RGBA8 above).
Nobody uses 16bit textures anymore. Compressed textures are both MUCH smaller, and generally higher quality.
 Also L16 is pretty common for heightfields or other data images/textures.
I could probably put that one.
 I guess there are also planar formats, but I think those are more of a Image
 problem space.
I think if people want to provide instantiations for convenience, it should be in the libs that interact with those formats. For instance, I imagine in OpenGL binding there might be: alias C = ColorFor!GL_R11F_G11F_B10F; // <- instantiation of the proper type for the GL enum. Most people really just want RGB8, or RGBA8. If you're interacting with PackedRGB, you're already in specialist territory.
 Just as a completely random idea - How about aliases for HDR formats like
 HDR10 and Dolby Vision? Kind of looks like they're just combinations what
 you already have.
This is very specialist. Instantiations are expensive. If they're not used, it's a waste of compiler effort.
Sep 12 2016
next sibling parent reply John Colvin <john.loughran.colvin gmail.com> writes:
On Tuesday, 13 September 2016 at 01:05:56 UTC, Manu wrote:
 Also can I swizzle channels directly?
I could add something like: auto brg = c.swizzle!"brg"; The result would be strongly typed to the new arrangement.
Perfect use-case for opDispatch like in gl3n.
Sep 13 2016
parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 13 September 2016 at 17:47, John Colvin via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Tuesday, 13 September 2016 at 01:05:56 UTC, Manu wrote:
 Also can I swizzle channels directly?
I could add something like: auto brg = c.swizzle!"brg"; The result would be strongly typed to the new arrangement.
Perfect use-case for opDispatch like in gl3n.
One trouble with arbitrary swizzling is that people often want to write expressions like "c.xyzy", or whatever, but repeating components... and that's not something my colours can do. It's useful in realtime code, but it doesn't mean anything, and you can't interpret that value as a colour anymore after you do that. This sort of thing is used when you're not actually storing colours in textures, but instead just some arbitrary data. Don't use a colour type for that, use a vector type instead. What that sort of swizzling really is, are vector operations, not colour operations. Maybe I could add an API to populate a vector from the components of colours, in-order... but then we don't actually have linear algebra functions in phobos either! So colour->vectors and arbitrary swizzling is probably not something we need immediately. In my lib, colours are colours. If you have `BGR8 x` and `RGB8 y`, and add them, you don't get x.b+y.r, x.g+y.g, x.r+y.b... that's not a colour operation, that's an element-wise vector operation.
Sep 13 2016
next sibling parent Guillaume Piolat <first.last gmail.com> writes:
On Tuesday, 13 September 2016 at 09:31:53 UTC, Manu wrote:
 In my lib, colours are colours. If you have `BGR8 x` and `RGB8 
 y`, and add them, you don't get x.b+y.r, x.g+y.g, x.r+y.b... 
 that's not a colour operation, that's an element-wise vector 
 operation.
I think swizzling is way overrated, especially swizzling assignment. Which is the most readable? color.rb = color.br; // swizzling swap(color.r, color.b); // no swizzling auto color = RGBA8(c.rgb, 255); // swizzling auto color = RGBA8(c.r, c.g, c.b, 255); // no swizzling And finally: auto rb = color.rgrb; // When do you ever need that?
Sep 13 2016
prev sibling parent reply John Colvin <john.loughran.colvin gmail.com> writes:
On Tuesday, 13 September 2016 at 09:31:53 UTC, Manu wrote:
 On 13 September 2016 at 17:47, John Colvin via Digitalmars-d 
 <digitalmars-d puremagic.com> wrote:
 On Tuesday, 13 September 2016 at 01:05:56 UTC, Manu wrote:
 Also can I swizzle channels directly?
I could add something like: auto brg = c.swizzle!"brg"; The result would be strongly typed to the new arrangement.
Perfect use-case for opDispatch like in gl3n.
One trouble with arbitrary swizzling is that people often want to write expressions like "c.xyzy", or whatever, but repeating components... and that's not something my colours can do. It's useful in realtime code, but it doesn't mean anything, and you can't interpret that value as a colour anymore after you do that. This sort of thing is used when you're not actually storing colours in textures, but instead just some arbitrary data. Don't use a colour type for that, use a vector type instead. What that sort of swizzling really is, are vector operations, not colour operations. Maybe I could add an API to populate a vector from the components of colours, in-order... but then we don't actually have linear algebra functions in phobos either! So colour->vectors and arbitrary swizzling is probably not something we need immediately. In my lib, colours are colours. If you have `BGR8 x` and `RGB8 y`, and add them, you don't get x.b+y.r, x.g+y.g, x.r+y.b... that's not a colour operation, that's an element-wise vector operation.
Fair enough, you know much better than me how useful it would be. I was just suggesting that if you do support some sorts of swizzling then opDispatch would allow you to avoid users having to use strings. It would be as simple to implement as `alias opDispatch = swizzle;` given the swizzle function you were using before.
Sep 13 2016
parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 14 September 2016 at 00:56, John Colvin via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Tuesday, 13 September 2016 at 09:31:53 UTC, Manu wrote:
 On 13 September 2016 at 17:47, John Colvin via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On Tuesday, 13 September 2016 at 01:05:56 UTC, Manu wrote:
 Also can I swizzle channels directly?
I could add something like: auto brg = c.swizzle!"brg"; The result would be strongly typed to the new arrangement.
Perfect use-case for opDispatch like in gl3n.
One trouble with arbitrary swizzling is that people often want to write expressions like "c.xyzy", or whatever, but repeating components... and that's not something my colours can do. It's useful in realtime code, but it doesn't mean anything, and you can't interpret that value as a colour anymore after you do that. This sort of thing is used when you're not actually storing colours in textures, but instead just some arbitrary data. Don't use a colour type for that, use a vector type instead. What that sort of swizzling really is, are vector operations, not colour operations. Maybe I could add an API to populate a vector from the components of colours, in-order... but then we don't actually have linear algebra functions in phobos either! So colour->vectors and arbitrary swizzling is probably not something we need immediately. In my lib, colours are colours. If you have `BGR8 x` and `RGB8 y`, and add them, you don't get x.b+y.r, x.g+y.g, x.r+y.b... that's not a colour operation, that's an element-wise vector operation.
Fair enough, you know much better than me how useful it would be. I was just suggesting that if you do support some sorts of swizzling then opDispatch would allow you to avoid users having to use strings. It would be as simple to implement as `alias opDispatch = swizzle;` given the swizzle function you were using before.
Oh, it would be super-useful! Just that it's not a colour operation, its liner algebra... I think it's a job for another lib. Getting a colour as a vector is all that's needed.
Sep 13 2016
prev sibling parent reply Random D user <no email.com> writes:
On Tuesday, 13 September 2016 at 01:05:56 UTC, Manu wrote:
 Can you describe what  you  perceive to be hard?
Well, I just skimmed through the docs and I didn't look at the code, so that sense it was an "honest" view for phobos proposal. Also I was trying to convey that based on the docs it "looks hard", although I was suspecting that it isn't really. So the list of things was a list of examples that I would've wanted to see in the docs. Maybe in an overview or "here's how you use it" page. I can read the details if I really need them, but I think it's important to have a cook book examples for quick start. In general, I think basics should be dead simple (even over simplified for the very basic case), but the API and the docs should provide me layers of increasing detail and sophistication, which I can study if (or when) I really need control and care about the details.
 Few basic things I'd be interested in (examples):
 1. How to pack/unpack color into uint/rgba channels.
auto c = RGBA8(r, g, b, a); uint r = c.r.value; // 'r' is a 'NormalizedInt", so you need '.value' to get the raw integer value float g = cast(float)c.g; // cast to float knows about the normalized 0-1 range
Let's put these in the docs.
 Also can I swizzle channels directly?
I could add something like: auto brg = c.swizzle!"brg";
It's just a nicety. I guess I could just define exact color formats and convert (let the lib do the rest), or just unpack and repack. Also the docs might want to mention which way is the preferred/designed way.
 2. Can I just cast an array of bytes into a RGBA array and 
 operate on that?
Yes. Of course, if the image library returned an array of colors, you wouldn't have an array of raw bytes in the first place.
Cool. I just wanted to make sure something like this possible. Also should be in the docs as an example, so that people don't have to try this out themselves. Or search through github for details.
 3. How to scale color channels e.g. 5 bits to 8 bits and vice 
 versa. How to convert color formats? Can I combine 
 scale/conversion with single channel or channel swizzling?
alias CustomPackedColor = PackedRGB!("BGR_5_6_5", ubyte); // <- ordered bgr CustomPackedColor[] image = loadPackedImage(); auto unpacked = cast(RGB8)image[0]; // unpack and swizzle I'm not sure what you mean by "with single channel"?
For example if you first unpack RGBA_uint32 into R,G,B,A then you want to take G (or just swizzle color.g) and scale that to 5 bits or 16 bits for what ever use case you might have. Looks to me like this can be handled with something like PackedRGB!("G_5") and then do color conversion into that.
 4. Which way are RGBA format names defined ("byte array order" 
 r is first byte or "uint order" r is in high bits)?
I describe formats in byte (bit) order. It's the only reasonable way that works outside of a uint.
Agreed. This should be in the docs, so that it's clear to users. (especially, since RGBA8888 is the common case and it does fit into uint)
 I'm showing that parsing colors from string's follow the web
 convention. I can't redefine how colours are expressed on the 
 web for
 decades.
 I think you've just highlighted that I need to remove the '0x'
 notation though, that is definitely misleading! Use '#', that 
 should
 be clear.
Right. I was interpreting 0x........ as c hex literal. You should document that it's following the web convention.
 6. Perhaps you should define all the usual color formats, 
 since everybody
 and their cat has their own preference for rgba channel orders.
 In my experience all these 4 variants for RGBA8888 are pretty 
 common:
 RGBA8888
 BGRA8888
 ARGB8888
 ABGR8888
 and their X and 24bit variants.
I don't think they're common at all. Only 2 are common (one of which is deprecated), the other 2 are very niche.
Deprecated by who? Shouldn't phobos grade lib include all reasonable platforms? I agree that you probably don't see too much variation within windows APIs, images (BMP etc.) or with D3D GPU textures (they're compressed anyway and asset pipelines to do the conversions beforehand), but I was more of thinking image libraries and the numerous, sometimes old and quirky, image formats. Or perhaps someone wants to implement a software blitter for their favorite toy device/embedded project.
 I was trying to balance a tasteful amount of instantiations. I 
 don't
 really want to provide an instantiation of every single realtime
 format I've ever encountered out of the box.
Seems reasonable.
 For 16 bits fairly common are:
 RGB565 and RGBA5551, also sometimes you see one of RGBA4444 
 permutations
 (like RGBA8 above).
Nobody uses 16bit textures anymore. Compressed textures are both MUCH smaller, and generally higher quality.
Sure, agreed. These are not too useful as GPU textures these days (even on mobile), but if you do software 2d, load older image formats (image viewer etc.) or something even more specialized, these are pretty useful. I guess I was going for comprehensive (within reason), where as you were going for "90% of 2016 colors/image data", which is fine. Anyways, it would be nice to see color (and hopefully image + simple blitter) in phobos. There's no need to write the same convert/copy/stretch/blend loop over and over again. And cpu-side code is nice to have for quick prototyping and small-scale work.
 Just as a completely random idea - How about aliases for HDR 
 formats like HDR10 and Dolby Vision? Kind of looks like 
 they're just combinations what you already have.
This is very specialist. Instantiations are expensive. If they're not used, it's a waste of compiler effort.
HDR TVs and displays are coming fast and with that I'm sure HDR editing and photography will be common, which of course means HDR color formats and apps. But these can be added later.
Sep 13 2016
parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 14 September 2016 at 04:10, Random D user via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 In general, I think basics should be dead simple (even over simplified for
 the very basic case)
I think they are: import std.experimental.color; RGB8 color("red"); I don't think i's possible to make it simpler than that?
 but the API and the docs should provide me layers of
 increasing detail and sophistication, which I can study if (or when) I
 really need control and care about the details.
I'll work on the docs some more.
 Deprecated by who? Shouldn't phobos grade lib include all reasonable
 platforms?
They're all comprehensively supported, I just didn't pre-instantiate every possible format imaginable. That would be devastating to compile times, and have absolutely no value. BGR really just exists because WINAPI. Microsoft haven't recommended people write GDI based software for decades. Look at DX10(,11,12), BGR isn't even supported in modern DirectX. Microsoft left BGR to die years ago (and good riddance). That said, there is an instantiation for BGR in package.d... the reversed versions (ie, alpha first) are completely unusual; just some big-endian games consoles. Modern formats have the channels in order; RGB, this is true for all data types, ubyte, short, float, etc.
 I agree that you probably don't see too much variation within windows APIs,
 images (BMP etc.) or with D3D GPU textures (they're compressed anyway and
 asset pipelines to do the conversions beforehand), but I was more of
 thinking image libraries and the numerous, sometimes old and quirky, image
 formats. Or perhaps someone wants to implement a software blitter for their
 favorite toy device/embedded project.
"favorite toy device/embedded project" <- this is the definition of specialist task. I don't mind if that person has to type 'alias' at the top of their app.
 For 16 bits fairly common are:
 RGB565 and RGBA5551, also sometimes you see one of RGBA4444 permutations
 (like RGBA8 above).
Nobody uses 16bit textures anymore. Compressed textures are both MUCH smaller, and generally higher quality.
Sure, agreed. These are not too useful as GPU textures these days (even on mobile), but if you do software 2d, load older image formats (image viewer etc.) or something even more specialized, these are pretty useful. I guess I was going for comprehensive (within reason), where as you were going for "90% of 2016 colors/image data", which is fine.
There is comprehensive support for all these formats... I just didn't pre-instantiate the templates. It's not hard to do at the top of your program: alias MyRGBType = RGB!(...details...);
 Anyways, it would be nice to see color (and hopefully image + simple
 blitter) in phobos.
I think that'll come. This is a pre-requisite.
 There's no need to write the same convert/copy/stretch/blend loop over and
 over again. And cpu-side code is nice to have for quick prototyping and
 small-scale work.
Sure.
 Just as a completely random idea - How about aliases for HDR formats like
 HDR10 and Dolby Vision? Kind of looks like they're just combinations what
 you already have.
This is very specialist. Instantiations are expensive. If they're not used, it's a waste of compiler effort.
HDR TVs and displays are coming fast and with that I'm sure HDR editing and photography will be common, which of course means HDR color formats and apps. But these can be added later.
There is already support for HDR. Use a floating point RGB with UHDTV colour space: alias HDR = RGB!("rgb", float, linear(?), RGBColorSpace.UHDTV); CompressedRGB also has comprehensive support for all the HDR formats I've ever encountered. My point was, I don't want to provide instantiations for these types... most people don't need it. People writing HDR rendering software will instantiate the types they need themselves. It's very specialist work.
Sep 13 2016
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/11/2016 9:14 PM, Manu via Digitalmars-d wrote:
 I think I'm about as happy with my colour lib as I'm going to be.
 It really needs reviews.
I have zero expertise in color programming, and have no idea what a color library might do or what it would be used for. I looked at the front page of the documentation. I would like to see some sort of explanation of who, what, where, why and how of a color library. Like what is a color space, etc. Maybe add some links to explanatory Wikipedia pages? "Set of colors defined by X11, adopted by the W3C, SVG, and other popular libraries." Would be a great place to add clickable links for the relevant X11, W3C, SVG, etc. specification pages. It really, really needs a "References:" section, with links like: https://en.wikipedia.org/wiki/RGB_color_model Are there any definitive books on the topic? What is a BGR color type? What is a UV color type? What is a UVW color type? What is an alpha color type? What is luminance? "This module implements HSV, HSL, HSI, HCY, HWB, HCG color types." "angular color spaces" Some references to what these are would be nice. Contrast with: http://dlang.org/phobos/std_regex.html where general information is given and links for more are provided (although that is weak and could be improved). ------- My last comment is what can I do with a color library that would not require some additional library?
Sep 12 2016
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/12/16 6:51 PM, Walter Bright wrote:
 My last comment is what can I do with a color library that would not
 require some additional library?
Great question and well asked. It was mine as well but I could not put it as succinctly. -- Andrei
Sep 12 2016
prev sibling next sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 13 September 2016 at 08:51, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 9/11/2016 9:14 PM, Manu via Digitalmars-d wrote:
 I think I'm about as happy with my colour lib as I'm going to be.
 It really needs reviews.
I have zero expertise in color programming, and have no idea what a color library might do or what it would be used for. I looked at the front page of the documentation. I would like to see some sort of explanation of who, what, where, why and how of a color library. Like what is a color space, etc. Maybe add some links to explanatory Wikipedia pages? "Set of colors defined by X11, adopted by the W3C, SVG, and other popular libraries." Would be a great place to add clickable links for the relevant X11, W3C, SVG, etc. specification pages. It really, really needs a "References:" section, with links like: https://en.wikipedia.org/wiki/RGB_color_model Are there any definitive books on the topic? What is a BGR color type? What is a UV color type? What is a UVW color type? What is an alpha color type? What is luminance? "This module implements HSV, HSL, HSI, HCY, HWB, HCG color types." "angular color spaces" Some references to what these are would be nice. Contrast with: http://dlang.org/phobos/std_regex.html where general information is given and links for more are provided (although that is weak and could be improved).
Okay. I'll work on this feedback.
 My last comment is what can I do with a color library that would not require
 some additional library?
With color, ndslice, and std.algorithm, you can do quite a lot of useful image processing already. But my point before was that I see this primarily as an enabler. We really need a graph/plot library for instance! Also, multimedia libraries will all become naturally compatible when the primitive working type is defined in phobos.
Sep 12 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/12/2016 6:11 PM, Manu via Digitalmars-d wrote:
 Okay. I'll work on this feedback.
Wonderful!
 My last comment is what can I do with a color library that would not require
 some additional library?
With color, ndslice, and std.algorithm, you can do quite a lot of useful image processing already.
Ok, but I have no idea how. Perhaps include something as an example? Phobos in general lacks short, pithy, and *useful* examples. Such can go a long way towards selling the color library to users.
 But my point before was that I see this primarily as an enabler. We
 really need a graph/plot library for instance! Also, multimedia
 libraries will all become naturally compatible when the primitive
 working type is defined in phobos.
I had read your post on this topic after making my post. I think it's a strong answer.
Sep 12 2016
parent Walter Bright <newshound2 digitalmars.com> writes:
On 9/12/2016 7:03 PM, Walter Bright wrote:
 On 9/12/2016 6:11 PM, Manu via Digitalmars-d wrote:
 Okay. I'll work on this feedback.
Wonderful!
BTW, linking to authoritative reference material has the effect of reassuring the user that the library is designed by someone who knows his stuff. It has the same effect as notes and references appendices have in any good technical book.
Sep 12 2016
prev sibling next sibling parent Andrea Fontana <nospam example.com> writes:
On Monday, 12 September 2016 at 22:51:14 UTC, Walter Bright wrote:
 My last comment is what can I do with a color library that 
 would not require some additional library?
I think it's easy to do some image processing using netbpm [1] as format. Reading/Writing images doesn't require any other library. And using ndslices a lot of things can be done :) Andrea [1] https://en.wikipedia.org/wiki/Netpbm_format
Sep 13 2016
prev sibling parent Guillaume Piolat <first.last gmail.com> writes:
On Monday, 12 September 2016 at 22:51:14 UTC, Walter Bright wrote:
 My last comment is what can I do with a color library that 
 would not require some additional library?
Having correct conversions. Converting sRGB to linear RGB is typically not done correctly, pow isn't enough. It also dissipates most misunderstanding about color space (yes Lab also are dependent on the color point, it was something I wasn't sure about and this library says it clearly). Right now I rely on the OS for Apple RGB to sRGB conversion, this library would allow me to do it without using a system call. Other uses I can imagine: - Downsampling an image preserving colors (compressed RGB isn't necessarily the right sep to do those) - Perceptually correct color gradients. In font rendering, - HSV is very useful for colouring stuff in video games, it's what the color selection does in painter-programs like GIMP. So lots of library out there have a HSV <-> RGB conversions - converting to packed RGB is useful for programs using GPUs. Same reason than half-float support is useful.
Sep 13 2016
prev sibling next sibling parent reply Meta <jared771 gmail.com> writes:
On Monday, 12 September 2016 at 04:14:27 UTC, Manu wrote:
 I think I'm about as happy with my colour lib as I'm going to 
 be. It really needs reviews.

 I added packed-RGB support, including weird micro-float and
 shared-exponent formats.
 They're important for interacting with any real-time rendering 
 libraries.
 There is only one texture format I'm aware of that isn't 
 supported,
 and that is u7e3 floats, only available on Xbox360 ;)

 Chromatic adaptation functions are in.
 I've done a tidy/reorg pass.

 I'm not sure what else should be in the scope of this lib.

 PR: https://github.com/dlang/phobos/pull/2845
 dub: https://code.dlang.org/packages/color
 docs: 
 http://dtest.thecybershadow.net/artifact/website-04a64e024cf75be39700bebd3a50d26f6c7bd163-7185c8ec7b15c9e785880cab6d512e6f/web/library-prerelease/std/experimental/color.html

 - Manu
Looks like a nice and simple module. My only complaint is that DDOC sections such as Returns, Args, etc. don't seem to be used at all. Also maybe a bit more detailed of an overview in package.d, which I assume is the entry point for the documentation.
Sep 14 2016
next sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 14 September 2016 at 22:47, Meta via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 12 September 2016 at 04:14:27 UTC, Manu wrote:
 I think I'm about as happy with my colour lib as I'm going to be. It
 really needs reviews.

 I added packed-RGB support, including weird micro-float and
 shared-exponent formats.
 They're important for interacting with any real-time rendering libraries.
 There is only one texture format I'm aware of that isn't supported,
 and that is u7e3 floats, only available on Xbox360 ;)

 Chromatic adaptation functions are in.
 I've done a tidy/reorg pass.

 I'm not sure what else should be in the scope of this lib.

 PR: https://github.com/dlang/phobos/pull/2845
 dub: https://code.dlang.org/packages/color
 docs:
 http://dtest.thecybershadow.net/artifact/website-04a64e024cf75be39700bebd3a50d26f6c7bd163-7185c8ec7b15c9e785880cab6d512e6f/web/library-prerelease/std/experimental/color.html

 - Manu
Looks like a nice and simple module. My only complaint is that DDOC sections such as Returns, Args, etc. don't seem to be used at all. Also maybe a bit more detailed of an overview in package.d, which I assume is the entry point for the documentation.
Cheers. Yeah, I need to do better with ddoc. ... I'm just gonna go on the record and say that I am really, really not enjoying ddoc ;) If I could get people starting to review the code, I'll polish up the docs over the next few days.
Sep 14 2016
next sibling parent reply Meta <jared771 gmail.com> writes:
On Wednesday, 14 September 2016 at 13:28:23 UTC, Manu wrote:
 Cheers.
 Yeah, I need to do better with ddoc.

 ... I'm just gonna go on the record and say that I am really, 
 really not enjoying ddoc ;)
I can't say I'm a fan either.
Sep 14 2016
next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Wed, 14 Sep 2016 13:47:43 +0000
schrieb Meta <jared771 gmail.com>:

 On Wednesday, 14 September 2016 at 13:28:23 UTC, Manu wrote:
 Cheers.
 Yeah, I need to do better with ddoc.

 ... I'm just gonna go on the record and say that I am really,=20
 really not enjoying ddoc ;) =20
=20 I can't say I'm a fan either.
You can replace $(D =E2=80=A6) with `=E2=80=A6` nowadays. It made me a bit = more of a fan. :) --=20 Marco
Sep 14 2016
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/14/2016 09:47 AM, Meta wrote:
 On Wednesday, 14 September 2016 at 13:28:23 UTC, Manu wrote:
 Cheers.
 Yeah, I need to do better with ddoc.

 ... I'm just gonna go on the record and say that I am really, really
 not enjoying ddoc ;)
I can't say I'm a fan either.
Why? -- Andrei
Sep 14 2016
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2016-09-14 22:14, Andrei Alexandrescu wrote:

 Why? -- Andrei
I've recently worked on a theme for Ddoc that will be used in TextMate. For convince I started by adding the CSS inline in the head tag. Turns out if you have CSS looking like this: .foo { font-weight: bold; background-color: white; } Everything works perfectly fine. Then I added text color as well: color: black; Guess what happens? It cuts, it cuts everything after "color". The solution is to use a colon macro: Macros: COLON = : color$(COLON) black; It took me quite a while and a lot of frustration until I figured that out. But I guess that's an inherit problem with a macro system. That is what I'm missing from Ddoc: * Inline source code for each symbol [1]. The [1] example both has a button to show the source code inline and a direct link to GitHub * Link to GitHub for each symbol [1] and for the whole module. The way it's done now with $(PHOBOSSRC std/net/_isemail.d) is not standardized and feels like a hack. Why slashes and why the underscore? If anything it should be the fully qualified name. Ideally it should be automatic, or a compiler recognized macro that inserts the link * Ddoc should organize symbols by visibility [2] * Ddoc should generate index table [3] * Ddoc should organize a module by symbol category [4] * Add a way to inherit documentation from the parent class [5] * Ddoc should give a list of inherited members [6]. JavaDoc does this by default [7] * Consistent way of cross-referencing. Ideally one would just pass the fully qualified name (or local name for symbols in the same module) to a compiler recognized macro and it would generate the appropriate links * An index page with all symbols, including a filter [8]. This is super convenient when you know the symbol to use but you need to double check how it's used or the behavior. I use it almost every day with Ruby All this should be done within Ddoc without the need for any post processing. In general I feel that Ddoc is too much focused on a generic macro system instead of focusing on being a good tool for writing documentation for code. [1] http://api.rubyonrails.org/classes/ActiveRecord/FinderMethods.html#method-i-first [2] https://issues.dlang.org/show_bug.cgi?id=11893 [3] https://issues.dlang.org/show_bug.cgi?id=11891 [4] https://issues.dlang.org/show_bug.cgi?id=11892 [5] https://issues.dlang.org/show_bug.cgi?id=10399 [6] https://issues.dlang.org/show_bug.cgi?id=7688 [7] http://help.eclipse.org/luna/index.jsp?topic=%2Forg.eclipse.jdt.doc.isv%2Freference%2Fapi%2Forg%2Feclipse%2Fjdt%2Fcore%2Fdom%2FSimpleName.html [8] http://ruby-doc.org/core-2.3.1 -- /Jacob Carlborg
Sep 14 2016
next sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
Do we mind moving this discussion to the other topic I started
actually on this topic?

It's slightly upsetting that this discussion about ddoc is more
popular than the colour lib I spent months workinng on :(


On 15 September 2016 at 16:55, Jacob Carlborg via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 2016-09-14 22:14, Andrei Alexandrescu wrote:

 Why? -- Andrei
I've recently worked on a theme for Ddoc that will be used in TextMate. For convince I started by adding the CSS inline in the head tag. Turns out if you have CSS looking like this: .foo { font-weight: bold; background-color: white; } Everything works perfectly fine. Then I added text color as well: color: black; Guess what happens? It cuts, it cuts everything after "color". The solution is to use a colon macro: Macros: COLON = : color$(COLON) black; It took me quite a while and a lot of frustration until I figured that out. But I guess that's an inherit problem with a macro system. That is what I'm missing from Ddoc: * Inline source code for each symbol [1]. The [1] example both has a button to show the source code inline and a direct link to GitHub * Link to GitHub for each symbol [1] and for the whole module. The way it's done now with $(PHOBOSSRC std/net/_isemail.d) is not standardized and feels like a hack. Why slashes and why the underscore? If anything it should be the fully qualified name. Ideally it should be automatic, or a compiler recognized macro that inserts the link * Ddoc should organize symbols by visibility [2] * Ddoc should generate index table [3] * Ddoc should organize a module by symbol category [4] * Add a way to inherit documentation from the parent class [5] * Ddoc should give a list of inherited members [6]. JavaDoc does this by default [7] * Consistent way of cross-referencing. Ideally one would just pass the fully qualified name (or local name for symbols in the same module) to a compiler recognized macro and it would generate the appropriate links * An index page with all symbols, including a filter [8]. This is super convenient when you know the symbol to use but you need to double check how it's used or the behavior. I use it almost every day with Ruby All this should be done within Ddoc without the need for any post processing. In general I feel that Ddoc is too much focused on a generic macro system instead of focusing on being a good tool for writing documentation for code. [1] http://api.rubyonrails.org/classes/ActiveRecord/FinderMethods.html#method-i-first [2] https://issues.dlang.org/show_bug.cgi?id=11893 [3] https://issues.dlang.org/show_bug.cgi?id=11891 [4] https://issues.dlang.org/show_bug.cgi?id=11892 [5] https://issues.dlang.org/show_bug.cgi?id=10399 [6] https://issues.dlang.org/show_bug.cgi?id=7688 [7] http://help.eclipse.org/luna/index.jsp?topic=%2Forg.eclipse.jdt.doc.isv%2Freference%2Fapi%2Forg%2Feclipse%2Fjdt%2Fcore%2Fdom%2FSimpleName.html [8] http://ruby-doc.org/core-2.3.1 -- /Jacob Carlborg
Sep 15 2016
parent reply Jacob Carlborg <doob me.com> writes:
On 2016-09-15 09:11, Manu via Digitalmars-d wrote:
 Do we mind moving this discussion to the other topic I started
 actually on this topic?

 It's slightly upsetting that this discussion about ddoc is more
 popular than the colour lib I spent months workinng on :(
Sorry. Is it the "ddoc latex/formulas?" topic? That's the only recent topic by you related to ddoc. But I have no interest in either latex or formulas. -- /Jacob Carlborg
Sep 15 2016
parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 15 September 2016 at 19:23, Jacob Carlborg via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 2016-09-15 09:11, Manu via Digitalmars-d wrote:
 Do we mind moving this discussion to the other topic I started
 actually on this topic?

 It's slightly upsetting that this discussion about ddoc is more
 popular than the colour lib I spent months workinng on :(
Sorry. Is it the "ddoc latex/formulas?" topic? That's the only recent topic by you related to ddoc. But I have no interest in either latex or formulas.
Yeah, sorry I forgot that detail. That thread meandered in the same direction though.
Sep 15 2016
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/15/2016 02:55 AM, Jacob Carlborg wrote:
 On 2016-09-14 22:14, Andrei Alexandrescu wrote:

 Why? -- Andrei
I've recently worked on a theme for Ddoc that will be used in TextMate. For convince I started by adding the CSS inline in the head tag. Turns out if you have CSS looking like this: .foo { font-weight: bold; background-color: white; } Everything works perfectly fine. Then I added text color as well: color: black; Guess what happens? It cuts, it cuts everything after "color". The solution is to use a colon macro: Macros: COLON = : color$(COLON) black; It took me quite a while and a lot of frustration until I figured that out. But I guess that's an inherit problem with a macro system.
Yah, this is terrible (it happens with http: all the time). We need to suppress the special meaning of ":" to only the cases when it's preceded by only one word and followed by a newline, e,g, "\ncolor:\n".
 That is what I'm missing from Ddoc:

 * Inline source code for each symbol [1]. The [1] example both has a
 button to show the source code inline and a direct link to GitHub
Can this be done by defining the expansion macros appropriately?
 * Link to GitHub for each symbol [1] and for the whole module. The way
 it's done now with $(PHOBOSSRC std/net/_isemail.d) is not standardized
 and feels like a hack. Why slashes and why the underscore?
Slashes are tolerable. The underscore I agree it's a quirk.
 If anything
 it should be the fully qualified name.
That'd be nice.
 Ideally it should be automatic,
 or a compiler recognized macro that inserts the link
Ironically the underscore is necessary because ddoc does too much automatically.
 * Ddoc should organize symbols by visibility [2]
 * Ddoc should generate index table [3]
 * Ddoc should organize a module by symbol category [4]
 * Add a way to inherit documentation from the parent class [5]

 * Ddoc should give a list of inherited members [6]. JavaDoc does this by
 default [7]

 * Consistent way of cross-referencing. Ideally one would just pass the
 fully qualified name (or local name for symbols in the same module) to a
 compiler recognized macro and it would generate the appropriate links

 * An index page with all symbols, including a filter [8]. This is super
 convenient when you know the symbol to use but you need to double check
 how it's used or the behavior. I use it almost every day with Ruby

 All this should be done within Ddoc without the need for any post
 processing.
I think we should start with a postprocessing-based approach and see what capabilities need to be added to obviate the need for postprocessing.
 In general I feel that Ddoc is too much focused on a generic macro
 system instead of focusing on being a good tool for writing
 documentation for code.
That's a fair assessment, and an actionable one. Andrei
Sep 15 2016
parent reply Jacob Carlborg <doob me.com> writes:
On 2016-09-15 14:54, Andrei Alexandrescu wrote:

 Yah, this is terrible (it happens with http: all the time). We need to
 suppress the special meaning of ":" to only the cases when it's preceded
 by only one word and followed by a newline, e,g, "\ncolor:\n".
It's useful for many of the sections to allow the section name on the same line as the content: /// Returns: some useful value int foo(); See the Ddoc documentation for more examples [1].
 Can this be done by defining the expansion macros appropriately?
What are "the expansion macros" ?
 Slashes are tolerable. The underscore I agree it's a quirk.
There are several (other) problems: * How do I know I should use slashes? There's no documentation * It's not a standard macro. If it's not here [2] nobody will find it or use it
 That's a fair assessment, and an actionable one.
Perhaps the current implementation of Ddoc could be moved to a separate tool that would focus on being a macro system. That can still be used for the site and other documents. Ddoc in the compiler could be improved to be a better tool for documenting code. I think these are two separate use cases that might not be best accomplished with the same tool. [1] http://dlang.org/spec/ddoc.html#standard_srction [2] http://dlang.org/spec/ddoc.html#predefined_macros -- /Jacob Carlborg
Sep 15 2016
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/15/2016 02:34 PM, Jacob Carlborg wrote:
 On 2016-09-15 14:54, Andrei Alexandrescu wrote:

 Yah, this is terrible (it happens with http: all the time). We need to
 suppress the special meaning of ":" to only the cases when it's preceded
 by only one word and followed by a newline, e,g, "\ncolor:\n".
It's useful for many of the sections to allow the section name on the same line as the content: /// Returns: some useful value int foo(); See the Ddoc documentation for more examples [1].
That's fine because "Returns" is special-cased. Random words at the beginning of a line followed by ":" should not receive special treatment.
 Can this be done by defining the expansion macros appropriately?
What are "the expansion macros" ?
Those DDOC_XXX things that are used for formatting declarations.
 Slashes are tolerable. The underscore I agree it's a quirk.
There are several (other) problems: * How do I know I should use slashes? There's no documentation
"Monkey see, monkey do." :o)
 * It's not a standard macro. If it's not here [2] nobody will find it or
 use it
Guess we can improve the (d)docs.
 That's a fair assessment, and an actionable one.
Perhaps the current implementation of Ddoc could be moved to a separate tool that would focus on being a macro system. That can still be used for the site and other documents. Ddoc in the compiler could be improved to be a better tool for documenting code. I think these are two separate use cases that might not be best accomplished with the same tool.
That'd be neat. As always, somebody's got to do it... Andrei
Sep 15 2016
parent Jacob Carlborg <doob me.com> writes:
On 2016-09-15 20:55, Andrei Alexandrescu wrote:

 That's fine because "Returns" is special-cased. Random words at the
 beginning of a line followed by ":" should not receive special treatment.
Ah, so you mean all non-standard non-special sections? Basically any section that is not listed in the ddoc documentation.
 Those DDOC_XXX things that are used for formatting declarations.
Hmm, I don't think there's a DDOC_ macro that evaluates to the full source code a given symbol. For a function that would include the source code of the body.
 Guess we can improve the (d)docs.
But then we need to have a standardized macro for that.
 That'd be neat. As always, somebody's got to do it...
As always. -- /Jacob Carlborg
Sep 16 2016
prev sibling parent Meta <jared771 gmail.com> writes:
On Wednesday, 14 September 2016 at 20:14:48 UTC, Andrei 
Alexandrescu wrote:
 On 09/14/2016 09:47 AM, Meta wrote:
 On Wednesday, 14 September 2016 at 13:28:23 UTC, Manu wrote:
 Cheers.
 Yeah, I need to do better with ddoc.

 ... I'm just gonna go on the record and say that I am really, 
 really
 not enjoying ddoc ;)
I can't say I'm a fan either.
Why? -- Andrei
The syntax is clunky and it's poorly documented. I've been lurking this mailing list since late 2012 and sporadically contributing since late 2013, and I had no idea you could do something like your example for Manu using DDOC. It seems like a real case of NiH syndrome; there are plenty of more widely used systems that would probably fit D just fine, but instead we have our own special macro documentation language. Really the only thing DDOC has going for it in comparison to something like Doxygen, IMO, is momentum. It's just too much of a pain to switch now.
Sep 15 2016
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/14/2016 09:28 AM, Manu via Digitalmars-d wrote:
 ... I'm just gonna go on the record and say that I am really, really
 not enjoying ddoc ;)
I was using ddoc today for the Greeks in DIP1000 and was musing about what makes it annoying. I think a major part is the macro invocation syntax: $(MACRO a1, a2, a3). After writing it for a few dozen times today, I figured it's objectively awkward to type (at least on the US keyboard): the dollar sign followed by the left paren is just unpleasant for mechanical reasons. The resulting look is alien too, as opposed to the more familiar $MACRO(a1, a2, a3) or the CPP-style MACRO(a1, a2, a3). There are a couple of un(der)documented tricks without which life is very difficult. For example, people commonly believe that in order to embed a literal comma they need to go like this: COMMA = , and then $(MACRO Hello$(COMMA) world, a2). That scales poorly with the size of the argument. In fact, there's a much better solution: ARGS = $0 and then $(MACRO $(ARGS Hello, world), a2) allows any amount of text with commas. Sadly no similar solution for unpaired parens. Also, the fact that the doc opens with the text and then has an obscure "Macros:" line at a point is unpleasant. The macro definitions should come first, followed by a separator and then the content. That would make for a much nicer flow. There are a few other macros I never leave home without: ARGS = $0 COMMA = , DOLLAR = $ LPAREN = ( RPAREN = ) TAIL = $+ COLON = : LF = $(LF) $(LF) COMMENT = SP = $(COMMENT space coming) $(COMMENT that was a space) TAIL sadly doesn't work properly. Overall: I'm trying to build a list of ddoc grievances that can be fixed. It's the de jure and de facto standard for D documentation (with the ddoc processor or ddox), so we'd do good to improve it. Please add your own ideas. Andrei
Sep 14 2016
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/14/2016 05:31 PM, Andrei Alexandrescu wrote:
 After writing it for a few dozen times today, I figured it's objectively
 awkward to type (at least on the US keyboard): the dollar sign followed
 by the left paren is just unpleasant for mechanical reasons.

 The resulting look is alien too, as opposed to the more familiar
 $MACRO(a1, a2, a3) or the CPP-style MACRO(a1, a2, a3).
I meant to add here: we can improve this in a backward-compatible way by allowing users to choose syntax by means of some notation, e.g. this thing as the first macro: DD0C_SYNTAX = $(X) for the existing syntax, vs. DD0C_SYNTAX = $X() for the $MACRO(a1, a2) syntax. We may also allow users to define the argument separator, also on a per-document basis. By default: DDOC_ARGUMENT_SEPARATOR = , We may also allow custom parens and so on. Andrei
Sep 14 2016
prev sibling next sibling parent reply bachmeier <no spam.net> writes:
On Wednesday, 14 September 2016 at 21:31:51 UTC, Andrei 
Alexandrescu wrote:

 Please add your own ideas.


 Andrei
One thing that I didn't like when I used DDOC was that I couldn't find a way to add a block of text directly to the output file. I know that can be done with html, but it's not the same. When I am programming I want to make notes (the way I make comments) but not get bogged down with all the macros/html stuff. Unformatted text is way better than unwritten text. Blocks of markdown would be better, but blocks of text would have helped. [Maybe this is possible but I was a new D user at that time and haven't had a reason to look into it.]
Sep 14 2016
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/14/16 7:04 PM, bachmeier wrote:
 On Wednesday, 14 September 2016 at 21:31:51 UTC, Andrei Alexandrescu wrote:

 Please add your own ideas.


 Andrei
One thing that I didn't like when I used DDOC was that I couldn't find a way to add a block of text directly to the output file. I know that can be done with html, but it's not the same. When I am programming I want to make notes (the way I make comments) but not get bogged down with all the macros/html stuff. Unformatted text is way better than unwritten text. Blocks of markdown would be better, but blocks of text would have helped. [Maybe this is possible but I was a new D user at that time and haven't had a reason to look into it.]
Do you mean like a macro $(VERBATIM ... text with balanced parens ...)? -- Andrei
Sep 14 2016
parent reply bachmeier <no spam.net> writes:
On Thursday, 15 September 2016 at 00:42:57 UTC, Andrei 
Alexandrescu wrote:

 Do you mean like a macro $(VERBATIM ... text with balanced 
 parens ...)? -- Andrei
Yes. Unfortunately I cannot find any mention of it on this page: http://dlang.org/spec/ddoc.html
Sep 15 2016
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/15/2016 11:25 AM, bachmeier wrote:
 On Thursday, 15 September 2016 at 00:42:57 UTC, Andrei Alexandrescu wrote:

 Do you mean like a macro $(VERBATIM ... text with balanced parens
 ...)? -- Andrei
Yes. Unfortunately I cannot find any mention of it on this page: http://dlang.org/spec/ddoc.html
It doesn't exist :o). I agree it's needed, I've hit the matter a few times. See https://issues.dlang.org/show_bug.cgi?id=16498. -- Andrei
Sep 15 2016
prev sibling next sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 15 September 2016 at 07:31, Andrei Alexandrescu via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 09/14/2016 09:28 AM, Manu via Digitalmars-d wrote:
 ... I'm just gonna go on the record and say that I am really, really
 not enjoying ddoc ;)
I was using ddoc today for the Greeks in DIP1000 and was musing about what makes it annoying. I think a major part is the macro invocation syntax: $(MACRO a1, a2, a3). After writing it for a few dozen times today, I figured it's objectively awkward to type (at least on the US keyboard): the dollar sign followed by the left paren is just unpleasant for mechanical reasons. The resulting look is alien too, as opposed to the more familiar $MACRO(a1, a2, a3) or the CPP-style MACRO(a1, a2, a3). There are a couple of un(der)documented tricks without which life is very difficult. For example, people commonly believe that in order to embed a literal comma they need to go like this: COMMA = , and then $(MACRO Hello$(COMMA) world, a2). That scales poorly with the size of the argument. In fact, there's a much better solution: ARGS = $0 and then $(MACRO $(ARGS Hello, world), a2) allows any amount of text with commas. Sadly no similar solution for unpaired parens. Also, the fact that the doc opens with the text and then has an obscure "Macros:" line at a point is unpleasant. The macro definitions should come first, followed by a separator and then the content. That would make for a much nicer flow. There are a few other macros I never leave home without: ARGS = $0 COMMA = , DOLLAR = $ LPAREN = ( RPAREN = ) TAIL = $+ COLON = : LF = $(LF) $(LF) COMMENT = SP = $(COMMENT space coming) $(COMMENT that was a space) TAIL sadly doesn't work properly. Overall: I'm trying to build a list of ddoc grievances that can be fixed. It's the de jure and de facto standard for D documentation (with the ddoc processor or ddox), so we'd do good to improve it. Please add your own ideas.
I haven't had enough experience for meaningful input. I have experienced the comma and parens things myself, and I generally just hate that everything is a macro, like you said yourself above. It didn't occur to me to attempt the sort of workarounds you illustrated here, I rather just factored the commas and parens out of my docs and moved on ;) Like, I really just don't care enough to try and understand ddoc sufficiently to have a bag of tricks like those you demonstrated above to workaround these issues. It's not a skill I *want* to possess, rather, in this case, it's a skill I'm being forced into. You're welcome to call me lazy, I'd suggest I'm being realistic. Perhaps a phobos contributor will be required to work it out (as seems to be a requirement for me right now), but normal programmers wouldn't. In 7 years, I've never been motivated to find workarounds to ddoc shortcomings before now, and now, it's only because people are hassling me. There's no intrinsic motivation here... one reason I use D is because I hate the C preprocessor ;) If I had to suggest, I'd introduce doxygen style \tags alongside the macros, then when people try and type docs in the way they've been doing for decades, it'll just work, and they can get on with their code. Nobody likes writing documentation, it needs to have the minimum possible friction or people just won't. Added bonus; when you cut&paste C code to adapt bindings, the existing docs on the C code would remain valid documentation without additional amendment. I bind C code regularly, and my goto 'solution' is to just remove the doxygen doco comments from the translated code and point them to the C/C++ docs instead. Again, call me what you like, I think my behaviour is probably typical though. I'm just being honest.
Sep 14 2016
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/14/16 9:28 PM, Manu via Digitalmars-d wrote:
 Like, I really just don't care enough to try and understand ddoc
 sufficiently to have a bag of tricks like those you demonstrated above
 to workaround these issues. It's not a skill I *want* to possess,
 rather, in this case, it's a skill I'm being forced into.
 You're welcome to call me lazy, I'd suggest I'm being realistic.
 Perhaps a phobos contributor will be required to work it out (as seems
 to be a requirement for me right now), but normal programmers
 wouldn't. In 7 years, I've never been motivated to find workarounds to
 ddoc shortcomings before now, and now, it's only because people are
 hassling me. There's no intrinsic motivation here... one reason I use
 D is because I hate the C preprocessor ;)
I don't see those workarounds to ddoc shortcomings. m4 and all macro systems I know of have similar idioms. It's the nature of macro processing. Are you simply saying you're familiar with other documentation tools and are not motivated to get into another one? That's entirely fair.
 If I had to suggest, I'd introduce doxygen style \tags alongside the
 macros, then when people try and type docs in the way they've been
 doing for decades, it'll just work, and they can get on with their
 code. Nobody likes writing documentation, it needs to have the minimum
 possible friction or people just won't.
I have difficulty understanding this. I haven't looked at Doxygen in a long time and never really used it, but from what I see at https://www.stack.nl/~dimitri/doxygen/manual/commands.html it seems the \tags you refer to are just a form of macros. The syntax is different, i.e. you'd write \a hello whereas in html you'd write <i>hello</i>, in latex \textit{hello}, in ddoc $(I hello). It's a matter of syntax and though I agree syntax matters (and it would be nice to make ddoc's more configurable), is it right to assume you simply want a syntax you're more familiar with? How do you mean people "type docs in the way they've been doing for decades"? Are you implying doxygen not only has been around for decades, but it's been some sort of ubiquitous standard? Honest question, I'm definitely not getting that. Does this boil down to - if we replace the macro syntax with one closer to doxygen things will just click? (That may as well be the case.) Andrei
Sep 14 2016
next sibling parent reply Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 15 September 2016 at 13:49, Andrei Alexandrescu via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 9/14/16 9:28 PM, Manu via Digitalmars-d wrote:
 Like, I really just don't care enough to try and understand ddoc
 sufficiently to have a bag of tricks like those you demonstrated above
 to workaround these issues. It's not a skill I *want* to possess,
 rather, in this case, it's a skill I'm being forced into.
 You're welcome to call me lazy, I'd suggest I'm being realistic.
 Perhaps a phobos contributor will be required to work it out (as seems
 to be a requirement for me right now), but normal programmers
 wouldn't. In 7 years, I've never been motivated to find workarounds to
 ddoc shortcomings before now, and now, it's only because people are
 hassling me. There's no intrinsic motivation here... one reason I use
 D is because I hate the C preprocessor ;)
I don't see those workarounds to ddoc shortcomings. m4 and all macro systems I know of have similar idioms. It's the nature of macro processing. Are you simply saying you're familiar with other documentation tools and are not motivated to get into another one? That's entirely fair.
That's absolutely what I'm saying, but not only that, given that I'm being forced into it now, I'm actively not enjoying it. I understand it's a macro system, and I'm saying that's the problem as I perceive it :) I clearly recall when I learned doxygen. I generally liked it, and I was motivated to populate my code with docs. Perhaps there's an obvious counter-pressure in this case since I already know doxygen, but I think that's insignificant compared to the fact that I just don't like ddoc. This macro system is not what I would have done, and it's just not compelling. I don't want documentation-realted macro definitions in my code. I also hate the paren spam.
 If I had to suggest, I'd introduce doxygen style \tags alongside the
 macros, then when people try and type docs in the way they've been
 doing for decades, it'll just work, and they can get on with their
 code. Nobody likes writing documentation, it needs to have the minimum
 possible friction or people just won't.
I have difficulty understanding this. I haven't looked at Doxygen in a long time and never really used it, but from what I see at https://www.stack.nl/~dimitri/doxygen/manual/commands.html it seems the \tags you refer to are just a form of macros. The syntax is different, i.e. you'd write \a hello whereas in html you'd write <i>hello</i>, in latex \textit{hello}, in ddoc $(I hello). It's a matter of syntax and though I agree syntax matters (and it would be nice to make ddoc's more configurable), is it right to assume you simply want a syntax you're more familiar with?
I think of those examples, doxygen is objectively the least annoying syntax! It's 2 bytes, and doesn't introduce delimiters or parens of any kind. Sure, that's definitely an influence, I hate paren spam, but it's not all. If in practise I was able to make my docs be how I want as easily as I was able to the first time I encountered doxygen, I wouldn't be bothered. But I've had a number of contacts with ddoc now, and it's just not intuitive how to make my docs how I want, whereas with doxygen, I was walking within minutes, seconds even. I think I have a sense of uneasy-ness that comes from my intuition that macros are macros, and I don't know which ones are 'special', or which are just macros. Ie, some macros are obviously not actually macros, but keywords in disguise. Doxygen says things like "\a word" (style this as an argument), ddoc seems to be $(I word) (italic this), which says nothing about the semantic meaning of the thing allowing styling to be deferred. I refer to this page: https://dlang.org/spec/ddoc.html There's nothing there that says how to style a template argument for instance, it's all 'italic, big, small, bold' etc, and skimming other ddoc code, I see people using raw styling like $(I ..) (as you just demonstrated above). At face value, it feels clumsy and poorly defined. I'm sure I'm incorrect, but I just didn't have this same first-impression with doxygen, or C#'s doco standard (they're the only 2 others I've used before). The only reference I've ever looked at for doxygen is this page: https://www.stack.nl/~dimitri/doxygen/manual/commands.html I never read another line from their manuals, everything else emerged via osmosis, and I think osmosis is the correct way a programmer should learn this stuff, which means it needs to be as obvious and intuitive as possible.
 How do you mean people "type docs in the way they've been doing for
 decades"? Are you implying doxygen not only has been around for decades, but
 it's been some sort of ubiquitous standard? Honest question, I'm definitely
 not getting that.
Do you feel that doxygen is not a ubiquitous standard? It's been around for like 20 years. I can't think of any non-doxygen docs I interact with regularly other than Microsoft docs... Maybe this is just a C/C++ ecosystem thing?
 Does this boil down to - if we replace the macro syntax with one closer to
 doxygen things will just click? (That may as well be the case.)
Well, I think the core of my uneasy-ness is that the macros have no clear semantic meaning. Obviously I can design my own set of macros that express my semantic meaning... to hell with standards! ;) Your prior post said "here's a bunch of macros I never leave home without", that's exactly the problem as I see it. I don't want too see ddoc macro definition blocks appearing at random locations in my source code. And I'll also remind you of my past criticism that my solution to porting C code to D, is to block delete the doxygen comment blocks rather than bothering to port them to D. If I had a reason to say what you said above "if we replace the macro syntax with one closer to doxygen...", this is that reason, but it's the fact that they're macros rather than keywords that makes me upset, so replacing the macros with a different syntax probably won't make me feel less uneasy. Perhaps my uneasy-ness would subside if the list of 'builtins' (https://dlang.org/spec/ddoc.html) was more comprehensive (comparable to doxygen commands), and I never saw a ddoc macro definition in code anywhere. My gripe with the syntax would probably have passed without comment if that were the case and I would have just gotten used to it. And don't get me wrong, I think it's absolutely fantastic that D has a built-in documentation system. I just don't really like it.
Sep 15 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/15/2016 12:06 AM, Manu via Digitalmars-d wrote:
 I think I have a sense of uneasy-ness that comes from my intuition
 that macros are macros, and I don't know which ones are 'special', or
 which are just macros.
That's easy, they are all just macros, and can all be redefined.
Sep 15 2016
parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 15 September 2016 at 17:38, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 9/15/2016 12:06 AM, Manu via Digitalmars-d wrote:
 I think I have a sense of uneasy-ness that comes from my intuition
 that macros are macros, and I don't know which ones are 'special', or
 which are just macros.
That's easy, they are all just macros, and can all be redefined.
Yeah... I suspected this was the case ;)
Sep 15 2016
prev sibling parent reply Patrick Schluter <Patrick.Schluter bbox.fr> writes:
On Thursday, 15 September 2016 at 03:49:55 UTC, Andrei 
Alexandrescu wrote:
 On 9/14/16 9:28 PM, Manu via Digitalmars-d wrote:
 Like, I really just don't care enough to try and understand 
 ddoc
 sufficiently to have a bag of tricks like those you 
 demonstrated above
 to workaround these issues. It's not a skill I *want* to 
 possess,
 rather, in this case, it's a skill I'm being forced into.
 You're welcome to call me lazy, I'd suggest I'm being 
 realistic.
 Perhaps a phobos contributor will be required to work it out 
 (as seems
 to be a requirement for me right now), but normal programmers
 wouldn't. In 7 years, I've never been motivated to find 
 workarounds to
 ddoc shortcomings before now, and now, it's only because 
 people are
 hassling me. There's no intrinsic motivation here... one 
 reason I use
 D is because I hate the C preprocessor ;)
I don't see those workarounds to ddoc shortcomings. m4 and all macro systems I know of have similar idioms. It's the nature of macro processing. Are you simply saying you're familiar with other documentation tools and are not motivated to get into another one? That's entirely fair.
 If I had to suggest, I'd introduce doxygen style \tags 
 alongside the
 macros, then when people try and type docs in the way they've 
 been
 doing for decades, it'll just work, and they can get on with 
 their
 code. Nobody likes writing documentation, it needs to have the 
 minimum
 possible friction or people just won't.
I have difficulty understanding this. I haven't looked at Doxygen in a long time and never really used it, but from what I see at https://www.stack.nl/~dimitri/doxygen/manual/commands.html it seems the \tags you refer to are just a form of macros. The syntax is different, i.e. you'd write \a hello whereas in html you'd write <i>hello</i>, in latex \textit{hello}, in ddoc $(I hello). It's a matter of syntax and though I agree syntax matters (and it would be nice to make ddoc's more configurable), is it right to assume you simply want a syntax you're more familiar with? How do you mean people "type docs in the way they've been doing for decades"? Are you implying doxygen not only has been around for decades, but it's been some sort of ubiquitous standard? Honest question, I'm definitely not getting that.
doxygen is mostly javadoc compatible and that one is the standard. (in fact the \ can be replaced by every where) and it allows direct html tags. The good thing about doxygen is that it supports also markdown. There are also some niceties like auto brief and autoreferencing (when a symbol is written in a text will be clickable if its body was reachable during generation of the doc).
 Does this boil down to - if we replace the macro syntax with 
 one closer to doxygen things will just click? (That may as well 
 be the case.)
Probably. But be reminded of your remark in the SDLang vs. JSON thread from last year: "...this is a strategic error. Please throw s/SDL/DDOC/ away and use a standardized file format. -- Andrei
Sep 15 2016
parent Bastiaan Veelo <Bastiaan Veelo.net> writes:
On Thursday, 15 September 2016 at 09:46:07 UTC, Patrick Schluter 
wrote:
 There are also some niceties like auto brief and 
 autoreferencing (when a symbol is written in a text will be 
 clickable if its body was reachable during generation of the 
 doc).
Not only that, by using tag files [1] Doxygen even generates clickable symbols to third party libraries and other online documentation, without having access to the respective code. [1] http://www.stack.nl/~dimitri/doxygen/manual/config.html#config_external
Sep 15 2016
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/14/2016 2:31 PM, Andrei Alexandrescu wrote:
 The resulting look is alien too, as opposed to the more familiar $MACRO(a1, a2,
 a3) or the CPP-style MACRO(a1, a2, a3).
The $(MACRO a1, a2, a3) syntax comes from makefiles.
Sep 14 2016
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/14/2016 11:11 PM, Walter Bright wrote:
 On 9/14/2016 2:31 PM, Andrei Alexandrescu wrote:
 The resulting look is alien too, as opposed to the more familiar $MACRO(a1, a2,
 a3) or the CPP-style MACRO(a1, a2, a3).
The $(MACRO a1, a2, a3) syntax comes from makefiles.
The original idea was to be able to write documentation without needing many macros, if any. Such as this I wrote recently: /** * Provides a safe interface to the Posix stat() function. * * Detailed description that I didn't write for this function, * but this is where it goes. * * Another paragraph. * * Params: * path = file name * buf = stat instance to be filled in * Returns: * 0 for success, system errno for error */ I suspect that adding Markdown would help a lot (yes, I changed my mind about that). Also auto-detecting email addresses and urls and turning them into clickable links would help.
Sep 14 2016
next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 15 September 2016 at 16:44, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 9/14/2016 11:11 PM, Walter Bright wrote:
 On 9/14/2016 2:31 PM, Andrei Alexandrescu wrote:
 The resulting look is alien too, as opposed to the more familiar
 $MACRO(a1, a2,
 a3) or the CPP-style MACRO(a1, a2, a3).
The $(MACRO a1, a2, a3) syntax comes from makefiles.
The original idea was to be able to write documentation without needing many macros, if any. Such as this I wrote recently: /** * Provides a safe interface to the Posix stat() function. * * Detailed description that I didn't write for this function, * but this is where it goes. * * Another paragraph. * * Params: * path = file name * buf = stat instance to be filled in * Returns: * 0 for success, system errno for error */ I suspect that adding Markdown would help a lot (yes, I changed my mind about that). Also auto-detecting email addresses and urls and turning them into clickable links would help.
+1
Sep 15 2016
prev sibling next sibling parent Paolo Invernizzi <paolo.invernizzi no.address> writes:
On Thursday, 15 September 2016 at 06:44:16 UTC, Walter Bright 
wrote:
 On 9/14/2016 11:11 PM, Walter Bright wrote:

 I suspect that adding Markdown would help a lot (yes, I changed 
 my mind about that). Also auto-detecting email addresses and 
 urls and turning them into clickable links would help.
I **really** love the pragmatism of this man! ;-) /Paolo
Sep 15 2016
prev sibling next sibling parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Wed, 14 Sep 2016 23:44:16 -0700
schrieb Walter Bright <newshound2 digitalmars.com>:

 I suspect that adding Markdown would help a lot (yes, I changed my mind about 
 that).
+1 -- Marco
Sep 15 2016
parent reply Nemanja Boric <4burgos gmail.com> writes:
On Thursday, 15 September 2016 at 09:45:26 UTC, Marco Leise wrote:
 Am Wed, 14 Sep 2016 23:44:16 -0700
 schrieb Walter Bright <newshound2 digitalmars.com>:

 I suspect that adding Markdown would help a lot (yes, I changed 
 my mind about that).
https://github.com/kiith-sa/harbored-mod provides support for mixing ddoc and markdown. That's what we're using, and it's nice (especially because docs in source files look very close to generated ones), but it would be nicer if we could just use dmd for that, of course.
Sep 15 2016
parent Walter Bright <newshound2 digitalmars.com> writes:
On 9/15/2016 4:59 AM, Nemanja Boric wrote:
 On Thursday, 15 September 2016 at 09:45:26 UTC, Marco Leise wrote:
 Am Wed, 14 Sep 2016 23:44:16 -0700
 schrieb Walter Bright <newshound2 digitalmars.com>:

 I suspect that adding Markdown would help a lot (yes, I changed my mind about
 that).
https://github.com/kiith-sa/harbored-mod provides support for mixing ddoc and markdown. That's what we're using, and it's nice (especially because docs in source files look very close to generated ones), but it would be nicer if we could just use dmd for that, of course.
It looks like you've done some great work with this! And Boost licensed, too! I think we can leverage this for Ddoc markdown support. Thanks!
Sep 15 2016
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2016-09-15 08:44, Walter Bright wrote:

 The original idea was to be able to write documentation without needing
 many macros, if any. Such as this I wrote recently:

 /**
  * Provides a safe interface to the Posix stat() function.
  *
  * Detailed description that I didn't write for this function,
  * but this is where it goes.
  *
  * Another paragraph.
  *
  * Params:
  *    path = file name
  *    buf = stat instance to be filled in
  * Returns:
  *    0 for success, system errno for error
  */

 I suspect that adding Markdown would help a lot (yes, I changed my mind
 about that). Also auto-detecting email addresses and urls and turning
 them into clickable links would help.
Yeah, if you already have defined all the standard ddoc macros because the default styling is not useful at all. This is an example of the documentation generator for Ruby called Yard [1]. The default output view/style might not be the best in the world but I think it's pretty good. Here's an example of how to write the documentation [2]. [1] http://www.rubydoc.info/gems/httparty/0.14.0/HTTParty/ClassMethods [2] http://www.rubydoc.info/gems/yard/file/docs/GettingStarted.md -- /Jacob Carlborg
Sep 15 2016
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/15/2016 10:37 AM, Jacob Carlborg wrote:
 Yeah, if you already have defined all the standard ddoc macros because
 the default styling is not useful at all.
Glad to hear that. It becomes a matter of improving the default-provided macros and/or distribute some .ddoc batteries. Jacob, I'd also like to thank for the great constructive ideas for improving things and for formulating them as enhancement requests on bugsilla. This is the kind of communication that pushes things forward. Andrei
Sep 15 2016
parent Jacob Carlborg <doob me.com> writes:
On 2016-09-15 16:45, Andrei Alexandrescu wrote:

 Glad to hear that. It becomes a matter of improving the default-provided
 macros and/or distribute some .ddoc batteries.

 Jacob, I'd also like to thank for the great constructive ideas for
 improving things and for formulating them as enhancement requests on
 bugsilla. This is the kind of communication that pushes things forward.
The oldest bugzilla issue I referred to is more than four years old and this is not the first discussion on the topic :( -- /Jacob Carlborg
Sep 15 2016
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/15/2016 7:37 AM, Jacob Carlborg wrote:
 Yeah, if you already have defined all the standard ddoc macros because the
 default styling is not useful at all.
The default styling was set up to produce legible output while not requiring any configuration file, nor a .css style file nor any knowledge of HTML at all.
Sep 15 2016
parent reply Jacob Carlborg <doob me.com> writes:
On 2016-09-16 02:02, Walter Bright wrote:

 The default styling was set up to produce legible output while not
 requiring any configuration file, nor a .css style file nor any
 knowledge of HTML at all.
I don't see how that's relevant. Currently one needs to have knowledge of HTML and CSS because the default output is not usable. With Yard (documentation generator for Ruby) one just invokes "yard -f html -o output intput.rb" and it generates all files and put them into "output". The result looks something like this [1]. How is that requiring any HTML or CSS knowledge? [1] http://www.rubydoc.info/gems/rails-bootstrap-helpers/0.1.0/RailsBootstrapHelpers/Helpers/ButtonHelper -- /Jacob Carlborg
Sep 16 2016
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/16/16 7:30 AM, Jacob Carlborg wrote:
 I don't see how that's relevant. Currently one needs to have knowledge
 of HTML and CSS because the default output is not usable.
Jacob, would you want to try your hand at tweaking the defaults? They are in a string in https://github.com/dlang/dmd/blob/master/src/doc.d. Clearly there's agreement these can be improved, and it's incredibly more productive to just improve the content instead of arguing how it could. Thanks!! -- Andrei
Sep 16 2016
parent reply Jacob Carlborg <doob me.com> writes:
On 2016-09-16 16:39, Andrei Alexandrescu wrote:

 Jacob, would you want to try your hand at tweaking the defaults? They
 are in a string in https://github.com/dlang/dmd/blob/master/src/doc.d.
 Clearly there's agreement these can be improved, and it's incredibly
 more productive to just improve the content instead of arguing how it
 could. Thanks!! -- Andrei
My design skills are quite limited. The theme I'm using for TextMate is basically the same as Apple is using for their documentation. But now when I tried to find a link to it, I see that they've changed the design. If I can use what I already have it might not be that much work. Two questions: * Can we have DMD output a separate CSS file? What will avoid the colon issue mentioned before and avoid the need to have 300-500 lines of CSS code in each HTML document * Can we create a new macro which will be used for each source code line in examples enclosed in "---"? That would allow to add line numbers with CSS -- /Jacob Carlborg
Sep 16 2016
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 09/16/2016 12:07 PM, Jacob Carlborg wrote:
 On 2016-09-16 16:39, Andrei Alexandrescu wrote:

 Jacob, would you want to try your hand at tweaking the defaults? They
 are in a string in https://github.com/dlang/dmd/blob/master/src/doc.d.
 Clearly there's agreement these can be improved, and it's incredibly
 more productive to just improve the content instead of arguing how it
 could. Thanks!! -- Andrei
My design skills are quite limited. The theme I'm using for TextMate is basically the same as Apple is using for their documentation. But now when I tried to find a link to it, I see that they've changed the design. If I can use what I already have it might not be that much work.
That would be awesome!
 Two questions:

 * Can we have DMD output a separate CSS file? What will avoid the colon
 issue mentioned before and avoid the need to have 300-500 lines of CSS
 code in each HTML document
I'd say we start without a separate CSS file. When the need for it becomes obvious, we can do that. What would be great is that your initiative isn't forestalled by the need for some other work.
 * Can we create a new macro which will be used for each source code line
 in examples enclosed in "---"? That would allow to add line numbers with
 CSS
We can, and we should (there are js solutions too). Again, let's get the ball rolling and amplify this as needed. Thanks! Andrei
Sep 16 2016
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/15/16 2:11 AM, Walter Bright wrote:
 On 9/14/2016 2:31 PM, Andrei Alexandrescu wrote:
 The resulting look is alien too, as opposed to the more familiar
 $MACRO(a1, a2,
 a3) or the CPP-style MACRO(a1, a2, a3).
The $(MACRO a1, a2, a3) syntax comes from makefiles.
In the words of Seinfeld, "And that doesn't help either!" I was thinking more about it and figured we could introduce a special macro at the very beginning of a .ddoc file or "Macros:" section called DDOC_MACRO_CALL_SYNTAX. It may be defined one of the following: DDOC_MACRO_CALL_SYNTAX = DDOC That's today's syntax. DDOC_MACRO_CALL_SYNTAX = CPP A macro will be invoked as MACRO(a1, a2). DDOC_MACRO_CALL_SYNTAX = $CPP A macro will be invoked as $MACRO(a1, a2) DDOC_MACRO_CALL_SYNTAX = LaTeX Macro invocation is \MACRO{a1}{a2}. This definition is modular, i.e. it doesn't carry beyond the current document (otherwise the semantics of a .dd file depends on the .ddoc file(s) it's used with). Macro definitions stay the same using $1, $2 etc. but macros invoked by them would need to obey the DDOC_MACRO_CALL_SYNTAX defined at the top. Thoughts? Andrei
Sep 15 2016
parent Walter Bright <newshound2 digitalmars.com> writes:
Started a new thread on this, "Ddoc macro syntax". Please direct further 
discussion there.
Sep 15 2016
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 9/14/2016 5:47 AM, Meta wrote:
 DDOC sections such as Returns, Args, etc.
s/Args/Params/
Sep 14 2016
prev sibling parent Johan Engelen <j j.nl> writes:
On Monday, 12 September 2016 at 04:14:27 UTC, Manu wrote:
 I think I'm about as happy with my colour lib as I'm going to 
 be. It really needs reviews.
I don't know enough about colors and what people do with it to give you helpful feedback, but I was wondering if you had looked at the (auto) vectorization of the code. I see some matrix math code, which perhaps would benefit from LDC's fastmath (or similar). It's just a thought. You can shoot me some example that you think should vectorize. Anyway, I think you are mostly interested in API feedback, implementation optimization can come later. (others: let's stop depressing Manu further and move all ddoc and other irrelevant discussions to other topics! :)
Sep 16 2016