www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - std.fileformats?

reply berni44 <dlang d-ecke.de> writes:
In Phobos there are several toplevel modules which each are about 
one special file format. Wouldn't it be better to put them all 
into std.fileformats (or some similar place)?

std.base64
std.csv
std.json
std.xml
std.zip

If yes, there are some more questions poping up:

a) Should we use the change to replace some of the modules? At 
least for std.xml I found std-experimental-xml [1], which seems 
to be thought to replace xml eventually. And there are lot's of 
packages which address json. I don't know if there is one, which 
is *the* candidate for replacement.

b) Should we instead remove some of these? Probably std.zip is 
here the first candidate. (I put some work in it in the last few 
weeks, but it would be fine for me throwing this away.)

It's on a gut level - just wondering, what you think about this.

[1] https://code.dlang.org/packages/std-experimental-xml
Jan 06
next sibling parent reply JN <666total wp.pl> writes:
On Monday, 6 January 2020 at 19:40:10 UTC, berni44 wrote:
 b) Should we instead remove some of these? Probably std.zip is 
 here the first candidate. (I put some work in it in the last 
 few weeks, but it would be fine for me throwing this away.)

 It's on a gut level - just wondering, what you think about this.

 [1] https://code.dlang.org/packages/std-experimental-xml
IMO all except base64 could be removed. Putting everything into the standard library made a lot of sense in the times before we got a package manager. Nowadays it might be better to simplify the standard library and just have XML, JSON, ZIP, CSV as "blessed" packages.
Jan 06
parent reply Laeeth Isharc <laeeth kaleidic.io> writes:
On Monday, 6 January 2020 at 21:05:37 UTC, JN wrote:
 On Monday, 6 January 2020 at 19:40:10 UTC, berni44 wrote:
 b) Should we instead remove some of these? Probably std.zip is 
 here the first candidate. (I put some work in it in the last 
 few weeks, but it would be fine for me throwing this away.)

 It's on a gut level - just wondering, what you think about 
 this.

 [1] https://code.dlang.org/packages/std-experimental-xml
IMO all except base64 could be removed. Putting everything into the standard library made a lot of sense in the times before we got a package manager. Nowadays it might be better to simplify the standard library and just have XML, JSON, ZIP, CSV as "blessed" packages.
Dependencies will be our doom. Whereas if you use something in Phobos then you have some confidence it will still build across platforms in a couple of years. It also seems to make sense to work on making code.dlang.org ultra-reliable before making it necessary to use libraries before doing even basic things.
Jan 06
next sibling parent reply "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Mon, Jan 06, 2020 at 11:38:25PM +0000, Laeeth Isharc via Digitalmars-d wrote:
 On Monday, 6 January 2020 at 21:05:37 UTC, JN wrote:
 On Monday, 6 January 2020 at 19:40:10 UTC, berni44 wrote:
 b) Should we instead remove some of these? Probably std.zip is
 here the first candidate. (I put some work in it in the last few
 weeks, but it would be fine for me throwing this away.)
 
 It's on a gut level - just wondering, what you think about this.
 
 [1] https://code.dlang.org/packages/std-experimental-xml
IMO all except base64 could be removed. Putting everything into the standard library made a lot of sense in the times before we got a package manager. Nowadays it might be better to simplify the standard library and just have XML, JSON, ZIP, CSV as "blessed" packages.
Dependencies will be our doom. Whereas if you use something in Phobos then you have some confidence it will still build across platforms in a couple of years.
[...] I agree, recent experiences have led me to the conclusion that dependencies are a liability rather than a benefit. The only exception is if you copy the code into your source tree, and periodically (manually) update it. Dependency resolution is NP complete (gives a whole new meaning to "dependency hell"); that should be a big red flag that dependencies are something to be avoided where possible, or at least treated with care, not relished. Even std.zip has been the source of trouble in the past: users would install dmd, code away happily, then get slammed with linker errors they don't know how to resolve. Eventually, the solution is to bundle zlib with the dmd distribution packages. Again, you see, dependencies are a liability, and the solution is to bundle the dependency with your main package so that the user never has to do any dependency resolution. This is why I've found that Adam's arsd libraries have been the best out of the D libraries out there: his philosophy of minimal (preferably no) dependencies, and everything bundled in a single source file, has been a boon. You just copy the source file into the right place in your source tree, check it in as part of your code repo, and never have to worry about sudden breakage beyond your control. Every now and then, just to stay up to date, git pull Adam's repo and copy the new file(s) over. Doing this manually may seem tedious in this day and age of instant gratification, but it's actually a benefit: (1) Since you manually copy the file(s) over, you're aware of what dependencies exactly you're pulling in, and (hopefully) are taking measures to prevent pulling in unnecessary extra cruft; (2) You will likely have enough sense to make sure your code compiles with the new version of the file before committing to the repo, thus avoiding the all-too-common problem of code breakage due to incompatibility with newer versions of the dependency: if it's checked in, it compiles, 100% guaranteed. Your buildability does not depend on the volatile state of some random server somewhere out there on the Internet. (3) Your collaborators will never be compiling using different versions of your dependencies and getting different results, which cause a lot of headaches trying to track down problems (everyone's build behaves slightly differently). (4) Should the upstream authors of the dependency abandon their project, vanish into the ether, or the dependency otherwise becomes unavailable, your code will still compile and still work. Again, you remove the possibility of random breakage caused by random internet outages. Future readers of your code will appreciate that *all* the code is there in the repo, without big missing chunks from dependencies that may no longer exist 10 or 20 years down the road. Even if the code will no longer compile by then, they can at least still see how it works. (And if you take this to the logical conclusion, bundling the exact version of the compiler you used to build the executables seems a logical possibility that will ensure compilability far into the future, even after your dependencies' maintainers have long abandoned it.) After so many decades of academia and industry alike trumpeting code reuse, I'm starting to become skeptical that perhaps King Code Reuse has invisible clothes. Dependency hell is a smell suggesting that something is fundamentally wrong with the concept. T -- That's not a bug; that's a feature!
Jan 06
parent reply berni44 <dlang d-ecke.de> writes:
On Tuesday, 7 January 2020 at 00:10:08 UTC, H. S. Teoh wrote:
 After so many decades of academia and industry alike trumpeting 
 code reuse, I'm starting to become skeptical that perhaps King 
 Code Reuse has invisible clothes. Dependency hell is a smell 
 suggesting that something is fundamentally wrong with the 
 concept.
Years ago I came to the same conclusion but in a completely different area: I was programming generators for sudoku puzzles and it's variants. This looks like being ideal for OOP, because all those variations are quit similar, but it proved not to be. The puzzles, the program produced, where not really good. And the reason was, that every variation needs it's own optimization for good results and that was almost impossible with all those code sharing. Eventually I rewrote everything with a single program for every variation. That worked much better, although it was code repetition over and over again.
Jan 07
parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Tue, Jan 07, 2020 at 12:17:57PM +0000, berni44 via Digitalmars-d wrote:
 On Tuesday, 7 January 2020 at 00:10:08 UTC, H. S. Teoh wrote:
 After so many decades of academia and industry alike trumpeting code
 reuse, I'm starting to become skeptical that perhaps King Code Reuse
 has invisible clothes.
[...]
 Years ago I came to the same conclusion but in a completely different
 area: I was programming generators for sudoku puzzles and it's
 variants. This looks like being ideal for OOP, because all those
 variations are quit similar, but it proved not to be.
Yeah, OOP is another of those things that IMNSHO has been overhyped far beyond its actual scope of usefulness. It *does* have its uses, that's undeniable, but only to certain classes (har har) of programming problems. In writing GUI components, for example, it seems to work really well. Trying to shoehorn *every* programming problem into an OO model, however, to me is a code smell. (Ironically, modern OO design advice is to avoid inheritance and embrace composition, which to me is a sign that polymorphism via inheritance, one of the foundational features of OOP, isn't as widely applicable as it's hyped to be.)
 The puzzles, the program produced, where not really good. And the
 reason was, that every variation needs it's own optimization for good
 results and that was almost impossible with all those code sharing.
 Eventually I rewrote everything with a single program for every
 variation. That worked much better, although it was code repetition
 over and over again.
Mind you, I'm not against code reuse via refactoring per se. Too much code repetition is also a code smell... but refactoring and generalizing to the point of obsession is a sign that it has gone off the deep end. Sometimes, it's just not worth the trouble, and the technical debt, to factor out the common code of two (or more) similar pieces of code. They may be similar but differ in complex ways that cannot be easily captured via a simple abstraction, and if you try to forcefully push the refactoring through anyway, you usually end up with a completely disproportionately more complex architecture that suffers from obvious symptoms of premature (or over-) generalization and over-engineering. The result usually is an order of magnitude more difficult to maintain, disproportionately harder to understand, and contains disproportionately more LOC. In such cases, the best solution may actually be copy-n-paste + modify. The code may well turn out to be simpler and more maintainable that way! T -- Why do conspiracy theories always come from the same people??
Jan 07
prev sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 1/6/20 6:38 PM, Laeeth Isharc wrote:
 On Monday, 6 January 2020 at 21:05:37 UTC, JN wrote:
 On Monday, 6 January 2020 at 19:40:10 UTC, berni44 wrote:
 b) Should we instead remove some of these? Probably std.zip is here 
 the first candidate. (I put some work in it in the last few weeks, 
 but it would be fine for me throwing this away.)

 It's on a gut level - just wondering, what you think about this.

 [1] https://code.dlang.org/packages/std-experimental-xml
IMO all except base64 could be removed. Putting everything into the standard library made a lot of sense in the times before we got a package manager. Nowadays it might be better to simplify the standard library and just have XML, JSON, ZIP, CSV as "blessed" packages.
Dependencies will be our doom.  Whereas if you use something in Phobos then you have some confidence it will still build across platforms in a couple of years. It also seems to make sense to work on making code.dlang.org ultra-reliable before making it necessary to use libraries before doing even basic things.
Except you do need to depend on it to do basic things. Have you tried using std.xml? Because something lives on code.dlang.org does not mean it is unreliable. The reliability depends on the developer, not the platform. I imagine vibe.d is going to be super-reliable for years, and it lives on code.dlang.org. H.S. Teoh also talks about arsd. This is also available on code.dlang.org (though not exactly built for it), but the mechanism of copying license compatible code into your project so you can maintain it doesn't depend on it being outside of code.dlang.org. You can do that with anything! D modules are pretty movable. One thing that is common between these two projects (and Phobos as well) -- They are both major projects with almost all their dependencies included in the project. That is, the project moves together, so you always have a cohesive "mini-standard" library. The dependency problem really shows up when your dependencies depend on dependencies that depend on dependencies that are all written and maintained by various people. A great example of how this can be a problem was the story of npm left-pad https://qz.com/646467/how-one-programmer-broke-the-internet-by-deleting-a-tiny-piece-of-code/ There's something to be said about having things outside the standard library not for the reasons of stability but for maneuverability. One can easily add/improve/change projects that are outside the standard library, but poor decisions in the standard library are sometimes stuck there (see std.xml). For file formats, or parsers, or really any kind of system that has innumerable ways of solving the problem, I really think the standard library is not where they should be. The benefits are cohesion with the library, and usage by the library. But there are many ways to skin the JSON cat. Especially when there's little reason for JSON to be included in Phobos other than a place for it to be. Oh, and I agree that reliability of code.dlang.org (or having dub more immune to the web site going down) should be a high priority. -Steve
Jan 06
next sibling parent Adam D. Ruppe <destructionator gmail.com> writes:
On Tuesday, 7 January 2020 at 01:45:21 UTC, Steven Schveighoffer 
wrote:
 H.S. Teoh also talks about arsd. This is also available on 
 code.dlang.org (though not exactly built for it), but the 
 mechanism of copying license compatible code into your project 
 so you can maintain it doesn't depend on it being outside of 
 code.dlang.org. You can do that with anything! D modules are 
 pretty movable.
In theory yes, but in practice it can be very difficult because of the size of the dependency graph. You import foo which needs bar which needs baz which needs joe and sally which need fred, tom, dick, and harry... it is very, very easy to fall down that rabbit hole and package managers make it look enticing. Heck, even I am very, very tempted to introduce a few base modules to my libs, especially now that I use dmd -i more so it would be almost transparent. And I probably would have already if I didn't type out my modules each time for so long. So my policy isn't to package dependencies, it is to *eliminate* them. So the individual files mostly stand alone. You can import one without needing the others, not even from the same repo.
Jan 06
prev sibling parent reply Laeeth Isharc <Laeeth laeeth.com> writes:
On Tuesday, 7 January 2020 at 01:45:21 UTC, Steven Schveighoffer 
wrote:
 On 1/6/20 6:38 PM, Laeeth Isharc wrote:
 On Monday, 6 January 2020 at 21:05:37 UTC, JN wrote:
 On Monday, 6 January 2020 at 19:40:10 UTC, berni44 wrote:
 b) Should we instead remove some of these? Probably std.zip 
 is here the first candidate. (I put some work in it in the 
 last few weeks, but it would be fine for me throwing this 
 away.)

 It's on a gut level - just wondering, what you think about 
 this.

 [1] https://code.dlang.org/packages/std-experimental-xml
IMO all except base64 could be removed. Putting everything into the standard library made a lot of sense in the times before we got a package manager. Nowadays it might be better to simplify the standard library and just have XML, JSON, ZIP, CSV as "blessed" packages.
Dependencies will be our doom.  Whereas if you use something in Phobos then you have some confidence it will still build across platforms in a couple of years. It also seems to make sense to work on making code.dlang.org ultra-reliable before making it necessary to use libraries before doing even basic things.
Except you do need to depend on it to do basic things. Have you tried using std.xml?
Yes, and I didn't persist for long. I wonder if sometimes there can be a tendency to let the best be the enemy of the considerably better. I think you made an argument for replacing std.xml rather than XML not being in the standard library. Though maybe that's a better argument than for getting rid of JSON and CSV.
 Because something lives on code.dlang.org does not mean it is 
 unreliable. The reliability depends on the developer, not the 
 platform.
Yes, I agree, but people are wired different ways and learnt to program in different eras and the cognitive cost of figuring out if a code.dlang.org library is any good is much greater for some people than others. You don't even know if the project will build with the current release of DMD. That mostly doesn't bother me personally, but it definitely does bother others. "The reliability depends on the developer, not the
 platform."
I don't agree with this. Phobos will probably work on BSD or smartos or whatever. And quite well-written libraries may require tweaking to build even on 32 bit Windows. And when you have a reasonable number of dependencies there is always something breaking with new releases so the cost may not be trivial for larger projects.
 I imagine vibe.d is going to be super-reliable for years, and 
 it lives on code.dlang.org.
Maybe. I am very grateful to Sonke for his contribution but vibe is one of the projects that breaks frequently and you can get into a mess where you need newer versions of the compiler for other reasons and vibe doesn't compile. Plus dub often can bring in dependencies spuriously, though that's a different problem.
 H.S. Teoh also talks about arsd. This is also available on 
 code.dlang.org (though not exactly built for it), but the 
 mechanism of copying license compatible code into your project 
 so you can maintain it doesn't depend on it being outside of 
 code.dlang.org. You can do that with anything! D modules are 
 pretty movable.
I like arsd and think we should use it more internally. It's pretty different though from having a module in Phobos, particularly if you want to get people who don't know D to use it, all the more so if they don't consider themselves programmers by trade. It can be too much to absorb in one go.
 There's something to be said about having things outside the 
 standard library not for the reasons of stability but for 
 maneuverability. One can easily add/improve/change projects 
 that are outside the standard library, but poor decisions in 
 the standard library are sometimes stuck there (see std.xml).
Yes I agree but isn't that an argument for adopting a better one as part of the standard library? Undead could also be more widely publicised and dub could be educated to ask if you want to fix old code by updating references.
 For file formats, or parsers, or really any kind of system that 
 has innumerable ways of solving the problem, I really think the 
 standard library is not where they should be. The benefits are 
 cohesion with the library, and usage by the library. But there 
 are many ways to skin the JSON cat. Especially when there's 
 little reason for JSON to be included in Phobos other than a 
 place for it to be.
A sensible default doesn't stop you from using something suited to your particular needs. It also saves time and cognitive effort to know what to expect when reading code.
 Oh, and I agree that reliability of code.dlang.org (or having 
 dub more immune to the web site going down) should be a high 
 priority.
I wonder what we could do about that. I could support if resources would help.
Jan 06
parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 1/6/20 10:48 PM, Laeeth Isharc wrote:
 On Tuesday, 7 January 2020 at 01:45:21 UTC, Steven Schveighoffer wrote:
 Except you do need to depend on it to do basic things. Have you tried 
 using std.xml?
Yes, and I didn't persist for long. I wonder if sometimes there can be a tendency to let the best be the enemy of the considerably better.  I think you made an argument for replacing std.xml rather than XML not being in the standard library. Though maybe that's a better argument than for getting rid of JSON and CSV.
Yes, for sure you can have a "basic" implementation for something, and then go elsewhere for more fancy implementations. Unit testing is a good example of this, the default works, and runs unit tests, but if you want fancy outputs, you go with one of the 3rd-party tools. The nice thing is that the runtime library allows you to swap out it's dull minimal implementation with your fancy implementation. xml and other file formats are not like that. First, there isn't a requirement in the library to have xml anywhere -- it's its own project, and simply lives in Phobos, nothing else in Phobos depends on it. Because it is of poor quality, we should replace or remove it. Of course, there are many ways to do xml, and it's hard to agree on the "right" way, which is why it sits there basically untouched for over a decade. It's also hard to make a similar mechanism like the previously mentioned unittests where you can "swap out" the implementation with what you really want. It fits better as an add-on library. It doesn't have to be part of Phobos to be maintained by a quality team. It's just kind of tacked on, an xml package, and we were doing what they did. Note that the bare minimum has been done to it, just to keep it building. I don't think that's desirable, and it's not a good look for the language. Not only that, but there is a cost to having a poor library being the "default" one. People do not go looking right away for something else, so they waste time on it, and finally go elsewhere anyway (perhaps away from D not even knowing there is something better for it). I'd say we are better off NOT having it in there.
 Because something lives on code.dlang.org does not mean it is 
 unreliable. The reliability depends on the developer, not the platform.
Yes, I agree, but people are wired different ways and learnt to program in different eras and the cognitive cost of figuring out if a code.dlang.org library is any good is much greater for some people than others.  You don't even know if the project will build with the current release of DMD.  That mostly doesn't bother me personally, but it definitely does bother others.
I definitely think we should have a repository system that separates "blessed" and maintained projects from all the ones that are added by random people. Maybe like a certified project list, and in order to be on it, the project has to be maintained by the core team (and tested along with the core CI), or there has to be a promise to fix issues within a certain time period or whatnot. Look at boost for example -- not part of the standard library, but might as well be. It lives in a space where innovation needs to move faster than standards committees. D isn't in the same boat exactly, but we have much less leeway to make significant changes to Phobos packages than we would to make changes on external ones. It's also difficult to say "this is the one way you should use xml", when nobody is passionate enough about it.
 
 "The reliability depends on the developer, not the
 platform."
I don't agree with this.  Phobos will probably work on BSD or smartos or whatever.  And quite well-written libraries may require tweaking to build even on 32 bit Windows.
This is due to the extensive CI system we have on there. This doesn't stop other projects from doing the same. What I meant was that code outside of Phobos can be just as reliable.
 And when you have a reasonable number of dependencies there is always 
 something breaking with new releases so the cost may not be trivial for 
 larger projects.
If we have the dependencies all covered, then there is not an issue. In other words, we could maintain a list of projects where all the dependencies are up to date.
 
 I imagine vibe.d is going to be super-reliable for years, and it lives 
 on code.dlang.org.
Maybe.   I am very grateful to Sonke for his contribution but vibe is one of the projects that breaks frequently and you can get into a mess where you need newer versions of the compiler for other reasons and vibe doesn't compile.  Plus dub often can bring in dependencies spuriously, though that's a different problem.
I haven't had this experience. When I've updated for my project, the vibe framework seems to be very good about warning me with deprecations rather than breaking.
 There's something to be said about having things outside the standard 
 library not for the reasons of stability but for maneuverability. One 
 can easily add/improve/change projects that are outside the standard 
 library, but poor decisions in the standard library are sometimes 
 stuck there (see std.xml).
Yes I agree but isn't that an argument for adopting a better one as part of the standard library?
But which one? Nobody can agree on what to put in there, so it stays. XML is not easy. JSON is easy (I wrote jsoniopipe in 1 week I think), and it is still in there despite other better systems existing.
 Undead could also be more widely publicised and dub could be educated to 
 ask if you want to fix old code by updating references.
You mean move it to undead? I'm fine with that, and I'm also fine with dub being proactive about projects that depend on it. But that doesn't mean we have to replace it.
 For file formats, or parsers, or really any kind of system that has 
 innumerable ways of solving the problem, I really think the standard 
 library is not where they should be. The benefits are cohesion with 
 the library, and usage by the library. But there are many ways to skin 
 the JSON cat. Especially when there's little reason for JSON to be 
 included in Phobos other than a place for it to be.
A sensible default doesn't stop you from using something suited to your particular needs.  It also saves time and cognitive effort to know what to expect when reading code.
The standard library should contain a minimal or best implementation of things that are core to the language. I don't think it needs a default implementation of everything. At one point we were looking at putting all sorts of things in there (std.database, std.io, std.kitchensink). I just think the advent of a build system that can pull in dependencies mitigates a lot of that need. It brings with it other problems, but if we can mow that grass down so you aren't wading through a forest, maybe it's not so bad. -Steve
Jan 06
next sibling parent berni44 <dlang d-ecke.de> writes:
On Tuesday, 7 January 2020 at 05:55:24 UTC, Steven Schveighoffer 
wrote:
 I definitely think we should have a repository system that 
 separates "blessed" and maintained projects from all the ones 
 that are added by random people.
The whole thing feels a little bit like an onion: The core is the core language, than the runtime, then Phobos and then, there is a skin missing (the one you describe) and then code.dlang.org and maybe next to whole internet. The closer to the core, the more reliable, but less flexible. So in Phobos there should only be essential stuff, like "writeln".
Jan 07
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
I'd be good with thinking about moving std.xml to dub. But not to unDead, as 
there is no replacement.
Jan 07
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Tuesday, January 7, 2020 7:20:05 PM MST Walter Bright via Digitalmars-d 
wrote:
 I'd be good with thinking about moving std.xml to dub. But not to unDead,
 as there is no replacement.
Why? I thought that the whole point of undead was for stuff from Phobos that we wanted to get rid of but still wanted to be available for older code would go to undead. I don't see how whether there is a replacement in Phobos is relevant. e.g. std.stream and friends were put into undead, and we don't really have a replacement for those. You can solve the problems that those modules try to solve in a different way using what's in Phobos, but we don't really have a stream solution in Phobos right now. What would be better about putting std.xml in its own project instead of undead? And honestly, from what I've seen of std.xml, I think that anyone using it should just plain seriously reconsider. I don't know how buggy it actually is, but it clearly was doing some stuff wrong such that it wouldn't be able to properly handle at least some XML files (though at the moment, I don't recall exactly what I found in there). And we do have multiple xml solutions available on code.dlang.org now. So, while we may not have a "blessed," default solution for people to switch to, there are better options available than std.xml. Also, I think that putting std.xml in its own project makes it too likely that someone else would use it in a new project, whereas if it's in undead, it's unlikely to be used by new projects, just older projects that were already using it. We've considered std.xml to be a poor solution for years now (we've even had a warning in its documentation that it was going to go away because it isn't up to our current standards), and I don't think that we should be encouraging its use. Putting it in undead makes it available without presenting it as a solution that people should be using. It's also where we've already been putting all of the modules that we decided to get rid of from Phobos, and we haven't really used the criteria of needing a replacement previously as far as whether it goes in undead or elsewhere. That's just been part of the question of whether we would remove it, and in some cases, we have removed stuff without really having a replacement in Phobos (most notably std.stream and friends). - Jonathan M Davis
Jan 07
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/7/2020 10:51 PM, Jonathan M Davis wrote:
 [...]
You raise some good points. I am not an XML user, so I don't have an informed opinion about this.
Jan 08
parent reply berni44 <dlang d-ecke.de> writes:
On Wednesday, 8 January 2020 at 08:26:33 UTC, Walter Bright wrote:
 On 1/7/2020 10:51 PM, Jonathan M Davis wrote:
 [...]
You raise some good points. I am not an XML user, so I don't have an informed opinion about this.
Filed a PR to add std.xml to undeaD: https://github.com/dlang/undeaD/pull/37 OK?
Jan 10
parent reply rikki cattermole <rikki cattermole.co.nz> writes:
On 11/01/2020 3:11 AM, berni44 wrote:
 On Wednesday, 8 January 2020 at 08:26:33 UTC, Walter Bright wrote:
 On 1/7/2020 10:51 PM, Jonathan M Davis wrote:
 [...]
You raise some good points. I am not an XML user, so I don't have an informed opinion about this.
Filed a PR to add std.xml to undeaD: https://github.com/dlang/undeaD/pull/37 OK?
This should have a vote and then yes we can deprecate it.
Jan 10
parent reply berni44 <dlang d-ecke.de> writes:
On Friday, 10 January 2020 at 14:21:44 UTC, rikki cattermole 
wrote:
 This should have a vote and then yes we can deprecate it.
How to start a vote?
Jan 10
parent rikki cattermole <rikki cattermole.co.nz> writes:
On 11/01/2020 3:43 AM, berni44 wrote:
 On Friday, 10 January 2020 at 14:21:44 UTC, rikki cattermole wrote:
 This should have a vote and then yes we can deprecate it.
How to start a vote?
Start a new thread, its not an official process. It gives a chance for people to give a compelling reason not to split it out and gets people on board.
Jan 10
prev sibling parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Monday, January 6, 2020 12:40:10 PM MST berni44 via Digitalmars-d wrote:
 In Phobos there are several toplevel modules which each are about
 one special file format. Wouldn't it be better to put them all
 into std.fileformats (or some similar place)?

 std.base64
 std.csv
 std.json
 std.xml
 std.zip

 If yes, there are some more questions poping up:

 a) Should we use the change to replace some of the modules? At
 least for std.xml I found std-experimental-xml [1], which seems
 to be thought to replace xml eventually. And there are lot's of
 packages which address json. I don't know if there is one, which
 is *the* candidate for replacement.

 b) Should we instead remove some of these? Probably std.zip is
 here the first candidate. (I put some work in it in the last few
 weeks, but it would be fine for me throwing this away.)

 It's on a gut level - just wondering, what you think about this.

 [1] https://code.dlang.org/packages/std-experimental-xml
Are you talking about putting them all in a single module or all in a single package? Putting them in a single module would be the opposite of the direction that we've been going with Phobos over the past few years. We've generally been breaking up larger modules, not merging modules. So, if we do rearrange these modules, it would definitely not be by putting them all in one module. However, even if we moved these modules into a sub-package, moving them around at this point would arguably just be unnecessary churn. It would break existing code for minimal benefit. If we replace any of them with new implementations (e.g. there's been talk of replacing std.xml and std.json for years now), then maybe they should go in a deeper package hierarchy, but I really don't think that it makes much sense to simply rearrange modules. It's also the sort of thing that Walter tends to be against. Now, personally, I don't think that anything regarding file formats should have been in the standard library in the first place. IMHO, that's not the sort of thing that belongs in a standard library, and they really should have been on code.dlang.org. However, when they were originally written, we didn't have code.dlang.org, so they ended up in Phobos. Either way, based on previous discussions on adding stuff to Phobos, I think that it's pretty clear that any new, major additions would have to be on code.dlang.org and battle-tested there before being moved into Phobos, and I don't see any of these modules being replaced any time soon even if we want to replace them. std.experimental.xml was a GSoC project that was not completed and is basically dead. It seems like the original author got too busy with school and never got back to it. For it to get anywhere, someone else would have to finish it. It was started with the intention of replacing std.xml, but like any other major additions, it would still have to go through the Phobs review process and be voted in. http://code.dlang.org/packages/dxml might end up in Phobos at some point, but that's not a fight that I want to fight right now, and the only reason that I think that it would make sense to put any XML parser/writer in Phobos at this point is because we already have std.xml, and it really needs to either be replaced or removed. http://code.dlang.org/packages/std_data_json was a candidate for replacing std.json, but IIRC, there was basically too much arguing over what the new std.json should look like, so Sonke gave up on getting it into Phobos. No one has tried to get a new JSON solution through the Phobos review process since then, and I think that for the most part, people have been happy to just put their stuff on code.dlang.org rather than trying to push anything through the Phobos review process. For the most part, I don't see any point in removing any of these modules, since that would break existing code, and while I don't think that Phobos should have been implementing parsers for standard file formats, that doesn't necessarily mean that we should be breaking existing code to remove them. The primary exception would probably be std.xml, since it arguably does more harm than good, but there's never really been a consensus on ripping it out without a replacement being added to Phobos. BTW, base64 isn't really a file format. It's an encoding. So, even if we were going to move all of these into a sub-package, std.base64 wouldn't belong with them. - Jonathan M Davis
Jan 06
parent reply berni44 <dlang d-ecke.de> writes:
On Tuesday, 7 January 2020 at 01:10:08 UTC, Jonathan M Davis 
wrote:
 Are you talking about putting them all in a single module or 
 all in a single package?
Sorry, for being unclear here. I thought of a package.
 Now, personally, I don't think that anything regarding file 
 formats should have been in the standard library in the first 
 place.
Thinking about this whole stuff, I noticed, that there are two different points of view, which should be separated: The idealist view and the pragmatic view. IMHO both are important. So when I got you right, from an idealists view, you'd say these file formats should be removed from phobos, but from a pragmatists view this looks much more difficult. I think, I share this point of view. But I'd like to get rid of them anyway.
 For the most part, I don't see any point in removing any of 
 these modules, since that would break existing code,
Well, every module, that is kept inside Phobos produces (lots of) maintainance work. From my perspective, we are missing resources here. So I prefere a controlled breaking of code (with deprecation and all) instead of having the code rosting and breaking uncontrolled sooner or later. I came up with this issue, when I looked at my own comment on issue 17709 [1]: I found the reason for this issue and I think I could fix that in a reasonable amout of time. But is it worth it doing so if that module might be removed or replaced in the near future? Wouldn't it be much better to use that time to fix a bug at a more important place? But on the other side: How does such a comment look like to someone how is using std.xml and found that issue, cause he stumbled over the same problem? Wouldn't it be better to remove std.xml completely in the first place? [1] https://issues.dlang.org/show_bug.cgi?id=17709 And than, there is something more to be thought of: The (public) perception of Phobos. If it contains parts, that are more or less broken, useless or outdated, this will be the parts where people will look at, when they judge. It won't help much to have other parts, that work well.
 BTW, base64 isn't really a file format. It's an encoding.
Really? So why isn't it in std.encoding? :-) I know, that base64 is somewhat different, maybe it's in the gray area... Or look at it the other way round: Isn't zipping also just encoding and unzipping decoding?
Jan 07
next sibling parent rikki cattermole <rikki cattermole.co.nz> writes:
On 08/01/2020 1:44 AM, berni44 wrote:
 BTW, base64 isn't really a file format. It's an encoding.
Really? So why isn't it in std.encoding? :-) I know, that base64 is somewhat different, maybe it's in the gray area... Or look at it the other way round: Isn't zipping also just encoding and unzipping decoding?
Perhaps, just perhaps std.encoding should be a package too ;) A zip file is a container, an encoding is not a container, it is a representation (just with different assumptions i.e. the base).
Jan 07
prev sibling parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Tuesday, January 7, 2020 5:44:40 AM MST berni44 via Digitalmars-d wrote:
 On Tuesday, 7 January 2020 at 01:10:08 UTC, Jonathan M Davis

 wrote:
 Now, personally, I don't think that anything regarding file
 formats should have been in the standard library in the first
 place.
Thinking about this whole stuff, I noticed, that there are two different points of view, which should be separated: The idealist view and the pragmatic view. IMHO both are important. So when I got you right, from an idealists view, you'd say these file formats should be removed from phobos, but from a pragmatists view this looks much more difficult. I think, I share this point of view. But I'd like to get rid of them anyway.
std.xml is probably the only one from your list that I'd argue should be seriously considered for being removed from Phobos and moved into undead sooner rather than later. I don't know quite what state std.json is in, so maybe it should have the same done to it, though it's had some work done on it recently to try to improve it. I see no reason to rip out stuff like std.csv or std.zip at this point though. They work and are useful. They also aren't fundamentally broken in the way that std.xml is AFAIK.
 For the most part, I don't see any point in removing any of
 these modules, since that would break existing code,
Well, every module, that is kept inside Phobos produces (lots of) maintainance work. From my perspective, we are missing resources here. So I prefere a controlled breaking of code (with deprecation and all) instead of having the code rosting and breaking uncontrolled sooner or later. I came up with this issue, when I looked at my own comment on issue 17709 [1]: I found the reason for this issue and I think I could fix that in a reasonable amout of time. But is it worth it doing so if that module might be removed or replaced in the near future? Wouldn't it be much better to use that time to fix a bug at a more important place? But on the other side: How does such a comment look like to someone how is using std.xml and found that issue, cause he stumbled over the same problem? Wouldn't it be better to remove std.xml completely in the first place? [1] https://issues.dlang.org/show_bug.cgi?id=17709
std.xml is broken. We've agreed for years now that it should go. It's just that there has been no agreement on removing it without having a replacement, which is why it's still there. I don't think that it's worth your time to work on it. However, out of the list of modules that you provided, std.json is the only other one where I recall any real discussion about replacing or removing. Certainly, spending time fixing a bug in something like std.base64 or std.zip is not a waste of time.
 BTW, base64 isn't really a file format. It's an encoding.
Really? So why isn't it in std.encoding? :-) I know, that base64 is somewhat different, maybe it's in the gray area... Or look at it the other way round: Isn't zipping also just encoding and unzipping decoding?
Encoding involves taking information and converting it into another format which contains the same information in a different manner. File formats may use encodings for some of the information that they contain, but an encoding has to do with how information is encoded. e.g. Unicode code points are encoded with UTF-8, UTF-16, or UTF-32. All three encodings contain exactly the same information, but the way that that information is encoded differs. And while UTF-8 may be used inside a file, it is in no way tied to files. It's just a way that Unicode character information is encoded. base64 is a binary encoding, whereas std.encoding deals with character encodings. zip is a file / container format. It uses different compression algorithms to encode binary information internally, but zip itself is a container format, not an encoding. It does far more than encode a string of data in a different way like an encoding does. std.encoding is also a bit of an oddball. It's an older module that probably needs to be revamped / redesigned. It has some level of support for various character encodings - including UTF-8, UTF-16, and UTF-32 - but we have std.utf for UTF handling, and std.utf is what gets used by Phobos for handling UTF encodings. std.encoding is class-based and has some range support, but it isn't really range-based aside from improvements that have been made to it over time. It does get some occasional tweaks, but largely, it's an older module with an older design that doesn't necessarily fit all that well into the rest of Phobos. I don't know what should be done with it though. Some of what it does is stuff that we really should have, but it probably needs to be redesigned. However, somebody would have to step up to do that. - Jonathan M Davis
Jan 07