www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - arsd users: weigh in on proposed breaking change

reply Adam D Ruppe <destructionator gmail.com> writes:
I've been thinking for a while to add a module `arsd.core` upon 
which the others can depend. This would have some interop 
definitions and implementation of the pieces necessary to make a 
shared event loop.

I also wrote most of a new text layout class over this weekend. 
My original intention was to embed it in simpledisplay.d since it 
was going to be tightly coupled to it and only useful with its 
draw text functions anyway.

However, after writing most of it, turns out it isn't actually 
coupled to simpledisplay at all, except for one font-related 
interface, which doesn't even have to work in terms of pixels! 
This means it could potentially be used by terminal.d as well and 
be the base of both character grid / tui and gui applications. 
The terminal.d ScrollbackBuffer *might* be able to use it (though 
since that thing already works as-is, idk if i would actually 
change it) and some of the terminal.d LineGetter could 
potentially be merged with a minigui LineEdit.

The benefits aren't exactly groundbreaking, but there's some 
potential here.

On the other hand, this would break the build for anyone who 
updates the individual files, since it'd change the import web.

I'd also have to shuffle some basic definitions like Point and 
Rectangle out of color.d and into core.d, which could affect 
static imports.

Before:
    terminal.d = stand-alone
    cgi.d = stand-alone
    http2.d = stand-alone
    color.d = stand-alone
    simpledisplay.d = needs color.d

etc. etc.

After, if we make all the changes:

    terminal.d = needs core.d, textlayouter.d
    cgi.d = needs core.d
    http2.d = needs core.d
    color.d = stand-alone
    simpledisplay.d = needs color.d, core.d, textlayouter.d

Anyone who downloads the individual files - which I've been 
telling people they can do for many years now - will have to get 
more of them and update their build process.

Moreover, the core event loop stuff might require you to actually 
run it in your thread, but I think I can do that right without 
breaking interop with outside systems anyway... and public 
imports can fixup reshuffled module members, so the main breakage 
here is for users who download the files.



So my question to those of you who use my libs... does this sound 
worth it to you? Of course, you can always just not update the 
file again, and stay happy with the version you have, but it 
seems likely that something will come up eventually that forces 
you to resync with me, so even though this breakage might not 
affect you today, I ask you to reply anyway since it might affect 
you tomorrow.

I expect I stress out too much over this, but I do try to take my 
promises of long-term support seriously and this is something 
that's worked without breakage for about 7 years now (which is 
where I added color.d and renamed the imports) so while it is 
small, I still don't want to break things without a real benefit.

PS if you do use my libs and wanna show off the project, please 
do! I'm always curious if people actually find these things 
useful.
Dec 19 2022
next sibling parent Adam D Ruppe <destructionator gmail.com> writes:
BTW I am also tempted to change the policy entirely and say you 
should clone the repo and use dmd -i. Then I could reorganize 
stuff without making it a breaking change anymore (Though I 
wouldn't do it often, once I change the policy, I can do it 
without waiting for my annual batch of breaking changes.)

I'd be curious about user's thoughts on this too, even though I'm 
less likely to actually do it. (I ask about the core.d and 
textlayouter.d since I'm already leaning toward that direction.)
Dec 19 2022
prev sibling next sibling parent Dom Disc <dominikus scherkl.de> writes:
On Monday, 19 December 2022 at 13:27:37 UTC, Adam D Ruppe wrote:

 Before:
    terminal.d = stand-alone
    cgi.d = stand-alone
    http2.d = stand-alone
    color.d = stand-alone
    simpledisplay.d = needs color.d

 etc. etc.

 After, if we make all the changes:

    terminal.d = needs core.d, textlayouter.d
    cgi.d = needs core.d
    http2.d = needs core.d
    color.d = stand-alone
    simpledisplay.d = needs color.d, core.d, textlayouter.d
Go for it! The stand-alone policy may have been useful, but now I tend to use several of the files anyway, so the dependency doesn't matter (to me).
Dec 19 2022
prev sibling next sibling parent reply bachmeier <no spam.net> writes:
On Monday, 19 December 2022 at 13:27:37 UTC, Adam D Ruppe wrote:

 So my question to those of you who use my libs... does this 
 sound worth it to you?
You are best able to judge the benefits, but from my perspective, the cost of downloading an extra file is minor. You could always have a script that copies all the files for each use case into its own directory, or to create a .zip for each of them if there's an issue.
 I expect I stress out too much over this, but I do try to take 
 my promises of long-term support seriously and this is 
 something that's worked without breakage for about 7 years now 
 (which is where I added color.d and renamed the imports) so 
 while it is small, I still don't want to break things without a 
 real benefit.
I personally think avoiding breakage is very important. That's why I typically create new projects rather than making breaking changes to existing projects. What you're proposing barely qualifies as breakage.
 PS if you do use my libs and wanna show off the project, please 
 do! I'm always curious if people actually find these things 
 useful.
I'm not someone has lots of users of my projects. However, I have one that has actually been used by other people - I've even come across people recommending it in HN comments. The entire thing is this 25-line file: https://github.com/bachmeil/tiddlyd/blob/main/tiddly.d You made changes to the handling of data inside PUT requests in cgi.d so it would work. Your libraries have been helpful to me. I learned basic web development by reading your book and then doing related things with your libraries. Not that web development is part of my job or anything. It was just nice to learn. From time to time I'll add a web interface on localhost to a D utility I've written. That sort of thing.
Dec 19 2022
parent reply Adam D Ruppe <destructionator gmail.com> writes:
On Monday, 19 December 2022 at 16:03:36 UTC, bachmeier wrote:
 What you're proposing barely qualifies as breakage.
Yeah, and the dub users wouldn't even notice it since I'll just add another 50 lines to dub.json for them. (Though they will be the ones to see the version number update. I tell people to use
= instead of ~> like dub suggests since what I consider a 
breaking change is almost certainly not going to affect them, but since ~> is the default they might be trapped on an old version for some time. oh well)
 The entire thing is this 25-line file:
Oh yes, I remember this. It is a cute little program.
 From time to time I'll add a web interface on localhost to a D 
 utility I've written. That sort of thing.
Little web interfaces are actually not half bad for a lot of things. You might enjoy experimenting with that cgi dispatcher some day, with its `serveApi` function it can auto-generate html forms form D functions. I haven't written much documentation of this stuff yet though. But check it out: --- import arsd.cgi; class TheClass : WebObject { UrlName("") // indicate this is the root function string hello(string name) { return "Hello, " ~ name ~ "!"; } static struct Info { string name; string title; } AutomaticForm (Cgi.RequestMethod.POST) Info[] withAutomaticForm(Info[] info) { return info; } } mixin DispatcherMain!( "/".serveApi!TheClass ); --- For this one, you will want to compile with `dmd -i` with the arsd repo cloned locally, since it requires cgi.d, dom.d, and jsvar.d. Of course, you can just download them and build with the explicit list too. Relatively slow build for an arsd module, about 1.6s for this example on my computer, but I've still kept it kinda under control despite the code gen magic it is doing behind the scenes. But anyway, run it! Here, I put a binary up so you can try yourself: http://arsdnet.net/cgi-bin/ncgi3/?name=D+fan http://arsdnet.net/cgi-bin/ncgi3/with-automatic-form You can see how it builds the form and some data output all from the D struct and function definitions. Not ideal for user-facing stuff (that's where you get into all the other code like custom presenter classes or the arsd.webtemplate module etc), but if you want a quick ui to call some things i quite like what it does automatically. And of course it can do the json output too: http://arsdnet.net/cgi-bin/ncgi3/?name=D+fan&format=json
Dec 19 2022
parent bachmeier <no spam.net> writes:
On Monday, 19 December 2022 at 21:41:47 UTC, Adam D Ruppe wrote:

 From time to time I'll add a web interface on localhost to a D 
 utility I've written. That sort of thing.
Little web interfaces are actually not half bad for a lot of things. You might enjoy experimenting with that cgi dispatcher some day, with its `serveApi` function it can auto-generate html forms form D functions. I haven't written much documentation of this stuff yet though.
I wasn't aware of it, but I'll give it a try. This is kind of how we used to use Tcl to make GUIs for scripts, but without the 1980s appearance on Linux.
Dec 19 2022
prev sibling next sibling parent reply Ralph Amissah <ralph.amissah gmail.com> writes:
On Mon, Dec 19, 2022 at 8:31 AM Adam D Ruppe via Digitalmars-d wrote:
 Anyone who downloads the individual files - which I've been
 telling people they can do for many years now - will have to get
 more of them and update their build process.
My need is for the single file; but whatever you choose to do, hopefully it can be done in a way that will not be too disruptive to users of individual parts of your offerings like me.
 PS if you do use my libs and wanna show off the project, please
 do! I'm always curious if people actually find these things
 useful.
I have been happy to have cgi.d available as a standalone file, that I use for a simple web sqlite search widget for generated document collections which are available in a few formats (html, epub, pdf via latex). arsd/cgi.d is the cgi behind, eg: searches: anticipate OR legend https://sisudoc.org/spine_search?sf=anticipate%20OR%20legend&rt=idx&ec=on&url=on&sml=1000&smo=0 https://sisudoc.org/spine_search?sf=anticipate%20OR%20legend&rt=txt&ec=on&url=on&sml=1000&smo=0 searches: linux OR windows https://sisudoc.org/spine_search?sf=linux OR windows&rt=idx&ec=on&url=on&se=on&sml=1000&smo=0 https://sisudoc.org/spine_search?sf=linux OR windows&rt=txt&ec=on&url=on&se=on&sml=1000&smo=0 which searches a small test collection of commons permissible licensed books/documents: https://sisudoc.org/spine/authors.html https://sisudoc.org/spine/topics.html (The origins of this interest go back to what was claimed to be one of the worlds first 200 websites, (on international commercial law) available from 1993, in early days for html). I have not shared the D code (repo currently only contains earlier ruby version). I consider myself an outlier. I wish to be able to work with the tools of my choice, and have become happy to explore on my own. The two most notable "tools" here would be: D (dlang) and things Nix (including NixOS). https://dlang.org/ https://forum.dlang.org/group/general https://nixos.org/ https://discourse.nixos.org/ There appears to be not much overlap of interest (in each other) between the projects. https://github.com/lionello/dub2nix although I have been building with static downloaded locally git tracked dependencies off the file system. (I wish there were more in each community that used (even depended on) the other).
 So my question to those of you who use my libs... does this sound
 worth it to you?
Not sure how the whole shebang is used by others and what it will mean to your project as a whole. Would it just mean my having to download an additional file? Thank you for all your offerings. No doubt there are many others like me who benefit from the multitude of different parts of it.
Dec 19 2022
parent Adam D Ruppe <destructionator gmail.com> writes:
On Monday, 19 December 2022 at 16:03:07 UTC, Ralph Amissah wrote:
 Would it just mean my having to download an additional file?
For cgi.d yes, it would just mean downloading core.d too and compiling them together. So if you currently do dmd yourfile.d cgi.d you'd do dmd yourfile.d cgi.d core.d (or whatever is necessary with your build system to add the other file) And always download both files together if you update. I was considering changing the policy to require you to download the whole repo and use dmd -i to build it (dmd -i will automatically include any referenced modules as-needed, while leaving the others out, so you still get the benefits of separation, it isn't like all 80 files are included in your build every time, while I'd be able to factor things out more often), but that is an even bigger change and I don't think I want to go that far. But adding just the one or maybe two files to a build is much closer to a balanced decision. I know this isn't big in the grand scheme of things, but it is something I've resisted doing for years. Even when I do add inter-module dependencies, I'd be careful to make it opt-in somehow. For example, if you use the cgi.dispatcher() function, then it depends on dom.d too.... but if you never call that, you don't need dom.d at all. But for what I'm thinking about now, I'm going to use it in many functions so that approach of "you don't pay for it if you don't call it" isn't going to work. So I gotta decide to either do it or not at least for the one core.d file.
 Thank you for all your offerings. No doubt there are many 
 others like me who benefit from the multitude of different 
 parts of it.
No, thank you for answering! I've also long been curious how many users it actually has. I sometimes get emails from people asking questions, and a couple times been surprised to see some of my code pop up in projects I've never heard of before, but I still have very little actual knowledge as to how many people actually use it so it is cool to hear from people.
Dec 19 2022
prev sibling next sibling parent reply Guillaume Piolat <first.last spam.org> writes:
On Monday, 19 December 2022 at 13:27:37 UTC, Adam D Ruppe wrote:
 PS if you do use my libs and wanna show off the project, please 
 do! I'm always curious if people actually find these things 
 useful.
I'm a user of arsd.dom and a few others. It will be a very minimal annoyance if arsd breaks compat more. Besides, it can be non-breaking when using dub subpackages, using sourceFiles.
Dec 19 2022
parent reply Adam D Ruppe <destructionator gmail.com> writes:
On Monday, 19 December 2022 at 19:57:24 UTC, Guillaume Piolat 
wrote:
 Besides, it can be non-breaking when using dub subpackages, 
 using sourceFiles.
Yeah, the dub users probably won't notice it at all, since I'll add the dependency to the subpackage in my config. But sourceFiles btw isn't great for this since you can end up with multiple definitions of things if people use multiple subpackages together. I've found the best way to do it is to make each module that can be used alone into its own subpackage and then list that as a dependency. It is a huge hassle to essentially reinvent what the compiler does natively... but it also seems to avoid the most problems when people do start mix and matching. And that's something I still will continue to support - while I provide a little of everything, I try not to force you to buy into my worldview just to use a little bit. If you like my png.d but not my simpledisplay.d, that's ok, they work well together but they also work just fine independently. Or if you like my simpledisplay but prefer your own image formats lib, it shouldn't be too hard to make adapters for the interfaces since I still try to keep things decoupled. Just at the same time, having a common core to import that defines a few of these interfaces helps with that interop too, which is why I'm looking at this.
Dec 19 2022
parent "H. S. Teoh" <hsteoh qfbox.info> writes:
On Mon, Dec 19, 2022 at 09:56:37PM +0000, Adam D Ruppe via Digitalmars-d wrote:
[...]
 And that's something I still will continue to support - while I
 provide a little of everything, I try not to force you to buy into my
 worldview just to use a little bit. If you like my png.d but not my
 simpledisplay.d, that's ok, they work well together but they also work
 just fine independently. Or if you like my simpledisplay but prefer
 your own image formats lib, it shouldn't be too hard to make adapters
 for the interfaces since I still try to keep things decoupled.
In one of my projects, I actually used a subset of arsd.terminal.Terminal's API as the basis for introspecting terminal-like objects in the code. There are a bunch of generic functions that use DbI to implement terminal-like functionality on things like virtual screens, sub-screens, buffering screens, etc., and of course Terminal itself. One in particular is a function call capture proxy that records calls to (a subset of) Terminal functions and saves them to a file, that can later be played back by a separate program. The playback can be to an instance of Terminal, or to another terminal-like virtual screen that renders the playback into pixels using some user-configured font and outputs an animated gif. :-D All of these can be more-or-less transparently swapped (and to a lesser extent Terminal itself, because I was too lazy to implement DbI for all of its methods, I just arbitrarily chose the subset I use the most), giving rise to all sorts of cool tricks. Your modules being more-or-less independent of each other was an important factor in making this possible. If Terminal had, for example, used some type defined by another module into its API, it would have made these DbI adapters more klunky to write, because another terminal-like wouldn't be able to define its own equivalent of that type without running into potential conflicts.
 Just at the same time, having a common core to import that defines a
 few of these interfaces helps with that interop too, which is why I'm
 looking at this.
Makes sense, as long as core.d doesn't grow without bound. ;-) T -- Stop staring at me like that! It's offens... no, you'll hurt your eyes!
Dec 19 2022
prev sibling next sibling parent reply "H. S. Teoh" <hsteoh qfbox.info> writes:
On Mon, Dec 19, 2022 at 01:27:37PM +0000, Adam D Ruppe via Digitalmars-d wrote:
[...]
 Before:
    terminal.d = stand-alone
    cgi.d = stand-alone
    http2.d = stand-alone
    color.d = stand-alone
    simpledisplay.d = needs color.d
 
 etc. etc.
 
 After, if we make all the changes:
 
    terminal.d = needs core.d, textlayouter.d
    cgi.d = needs core.d
    http2.d = needs core.d
    color.d = stand-alone
    simpledisplay.d = needs color.d, core.d, textlayouter.d
[...] This will probably break a few of my projects that use arsd.terminal. But since the fix is so easy, and I use git submodules to pull in arsd (meaning I will get all the files anyway), this isn't a big deal. I'll just fix a couple of build scripts and that's all. On that note, though, it would be nice if D had some kind of feature where only actual dependencies end up in the executable, i.e., end-to-end pay-as-you-go. But that's just a dream... We've been trying to make druntime/Phobos pay-as-you-go for years, and still haven't come close to the ideal state yet. T -- Prosperity breeds contempt, and poverty breeds consent. -- Suck.com
Dec 19 2022
parent reply Adam D Ruppe <destructionator gmail.com> writes:
On Monday, 19 December 2022 at 20:18:54 UTC, H. S. Teoh wrote:
 On that note, though, it would be nice if D had some kind of 
 feature where only actual dependencies end up in the 
 executable, i.e., end-to-end pay-as-you-go.
It does, and it has for over five years now! This is the only reason why this change is even on the table at all. This is what `dmd -i` does. Only things actually imported are compiled at all. Then you keep your imports local in templates which means they now happen on function-level granularity. So for example, if you do --- import arsd.dom; void main() { auto document = new Document("<html></html"); } --- And then `dmd -i yourapp.d`, it will automatically pull in arsd.dom to the build... but not the rest of the arsd modules. Now change it to: --- import arsd.dom; void main() { auto document = new Document(); document.parseGarbage("<html></html"); } --- And the same `dmd -i yourapp.d` will now automatically pull in arsd.dom and arsd.characterencodings because parseGarbage uses that to convert things to utf8. Or try: --- import arsd.dom; void main() { auto document = Document.fromUrl("http://dlang.org/"); } --- And again, the same `dmd -i yourapp.d` will pull arsd.dom, arsd.characterencodings, and arsd.http2 automatically, again because it knows you called fromUrl which imports arsd.http2. It took a little bit of effort for me to write the code in a way to do all this correctly, but the compiler works with it. The linker can also further strip unreferenced things from the executable afterward, and if you're careful about your import web, this can be very effective (and it sometimes can be too even if you aren't careful, but any module constructors being present have a habit of defeating the linker's stripping since it can't know if it is actually necessary or not).
 We've been trying to make druntime/Phobos pay-as-you-go for 
 years, and still haven't come close to the ideal state yet.
That's because they wrote things very differently than I do. arsd explicitly avoids dependency webs - which is why adding this proposed `core.d` module would be a breaking change, since I need to make an exception to that policy (and like i said in a previous message, I did consider suspending that policy entirely, but I don't think I'm going to do that. One advantage of more modules is, when combined with `dmd -i`, you can get pretty granular on what is compiled in. Just this doesn't always actually bring you a win.)
Dec 19 2022
parent "H. S. Teoh" <hsteoh qfbox.info> writes:
On Mon, Dec 19, 2022 at 08:57:26PM +0000, Adam D Ruppe via Digitalmars-d wrote:
 On Monday, 19 December 2022 at 20:18:54 UTC, H. S. Teoh wrote:
 On that note, though, it would be nice if D had some kind of feature
 where only actual dependencies end up in the executable, i.e.,
 end-to-end pay-as-you-go.
It does, and it has for over five years now! This is the only reason why this change is even on the table at all. This is what `dmd -i` does. Only things actually imported are compiled at all. Then you keep your imports local in templates which means they now happen on function-level granularity.
Well I know what `dmd -i` does. :-P What I had in mind was something more radical: importing a module would *not* trigger inclusion of the entire module's code; only the parts that are actually referenced. Say module M contains functions A, B, C, each of which references symbols X, Y, Z, respectively, in modules J, K, L. As it currently stands, if you import X, then `dmd -i` will also import J, K, L for you. However, if you only call X.A, then in theory the symbols from K and L will never actually get used. In my ideal vision, K and L would *not* be imported in this case. Now, today this can already be done to some extent by moving the imports of J, K, L into the respective function bodies as local imports, and making them template functions. This however only works for templates, and isn't scalable to the case where module M contains multiple functions that import the same module; in that case it's probably better to move the import to module level instead of repeating the import inside every function that might need it. And if you can't use template functions for whatever reason, you're also stuck. An even more extreme (and probably impractical) extension of this the elimination of code paths that can be determined at compile-time never to execute. E.g., most programs would not need all of the format handlers inside writefln(). If you never need to format a float, then the float-formatting part of writefln()'s implementation is just dead weight and ought to be eliminated. I don't think the current implementation of format() is capable of doing this (the implementation inside is a hairball of interdependent pieces that are almost impossible to extricate independently). [...]
 The linker can also further strip unreferenced things from the
 executable afterward, and if you're careful about your import web,
 this can be very effective (and it sometimes can be too even if you
 aren't careful, but any module constructors being present have a habit
 of defeating the linker's stripping since it can't know if it is
 actually necessary or not).
[...] In an ideal world, there would be a list of symbols touched by the module ctor, and if the reference graph containing these symbols is disjoint from main(), then the corresponding code can also be elided. T -- MS Windows: 64-bit rehash of 32-bit extensions and a graphical shell for a 16-bit patch to an 8-bit operating system originally coded for a 4-bit microprocessor, written by a 2-bit company that can't stand 1-bit of competition.
Dec 19 2022
prev sibling next sibling parent TTK Ciar <ttk ciar.org> writes:
Yes, please, go for it!

I only use two of the impacted modules anyway, and fixing my 
build scripts will be trivial.

My habit is to clone the whole git repo already, so that's a 
no-op.
Dec 19 2022
prev sibling next sibling parent reply Greggor <Greggor notareal.email> writes:
I use some of your stuff so I'll chime in. I don't automatically 
pull in files and so the updates to modules is done by hand in my 
case, so you adding a few dependencies is not a problem.

But I do not pull in the full arsd repo & I'd hate to see that to 
become a requirement. In general I love the "its just a few files 
you can add" to dependency management, but I don't mind it 
becoming a couple of files.

As for event loop stuff, there is a golden rule in my eyes, as 
long as there is a way to own the loop, just call a poll/dispatch 
function and I can still use the thread, there should be no issue 
with it.

The moment that is not the case that's when that type of stuff 
turns painful :^)
Dec 22 2022
parent Adam D Ruppe <destructionator gmail.com> writes:
On Friday, 23 December 2022 at 07:55:31 UTC, Greggor wrote:
 But I do not pull in the full arsd repo & I'd hate to see that 
 to become a requirement. In general I love the "its just a few 
 files you can add" to dependency management, but I don't mind 
 it becoming a couple of files.
Yeah, there's some temptation to break out a few more interfaces... well, ok i admit it is more than just some temptation, there's actually four or five files I'd probably add if there was zero concern (I'd do the eventloop+error handling+validation helper core, then a circular buffer container, the text layouter, and then possibly a couple more)... but I think I'll resist that and just put a set of things in the one additional core.d file, that compromise I think will do what I need it to do. The others in the list aren't as important. And of course, it is possible to keep doing things how I do it now, with some duck-typed interfaces that happen to be compatible if you pass the right arguments, then I copy/paste what needs to be pasted. But that's getting a bit limiting since i don't copy *everything*.
 As for event loop stuff, there is a golden rule in my eyes, as 
 long as there is a way to own the loop, just call a 
 poll/dispatch function and I can still use the thread, there 
 should be no issue with it.
eeeeeh, you shouldn't have to! but yeah, if you look at my http2.d current thing that's the model I've been coming to like. There's an `advance` function, a `addToSimpledisplayEventLoop` function which offers a file descriptor to integrate with an external thing, then the individual tasks can `waitForCompletion` and such. then, of course, you do have a static `runEventLoop` kind of thing to just drop and run. So you get some control but it might be a lot of extra work depending on what you're doing around it. Whereas if you make use of the event model you ought not need to do other stuff in the middle; you might even run the events in a separate thread and just queue messages back to this one. I haven't worked out the exact details yet but I'm fairly happy with how that http2.d thing has worked out api-wise, and the simpledisplay core is pretty solid allowing for integrations, so I'll probably just generalize those. Either way, yes, I do try to keep things decomposable so you can mix and match what you want. It might be a lil extra work for both of us but p sure I can keep it working for all these cases. (and I plan to keep my existing apis forwarding to the new thing, to avoid code breakage, and this also forces me to do it in a flexible way since my own existing things will require some of it)
Dec 23 2022
prev sibling parent Doigt <labog outlook.com> writes:
I don't mind the changes, go for it!
Dec 24 2022