www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Example code in std.logger.core doesn't even work

reply Yuxuan Shui <yshuiv7 gmail.com> writes:
Example here:

https://dlang.org/phobos/std_logger_core.html#.sharedLog

Result:

https://run.dlang.io/is/RMtF0I
Mar 12 2023
next sibling parent reply Johann Lermer <johann.lermer elvin.eu> writes:
Try this:

auto sharedLog = new FileLogger("/dev/null");
Mar 13 2023
next sibling parent Petar Kirov [ZombineDev] <petar.p.kirov gmail.com> writes:
On Monday, 13 March 2023 at 10:03:59 UTC, Johann Lermer wrote:

 Try this:

 auto sharedLog = new FileLogger("/dev/null");


This also works, but is likely thread unsafe:

```d
sharedLog = cast(shared)new FileLogger("/dev/null");
```
Mar 13 2023
prev sibling parent reply Yuxuan Shui <yshuiv7 gmail.com> writes:
On Monday, 13 March 2023 at 10:03:59 UTC, Johann Lermer wrote:

 Try this:

 auto sharedLog = new FileLogger("/dev/null");


That defines a new variable right. That's completely different.
Mar 13 2023
parent Johann Lermer <johann.lermer elvin.eu> writes:
On Monday, 13 March 2023 at 15:48:03 UTC, Yuxuan Shui wrote:

 On Monday, 13 March 2023 at 10:03:59 UTC, Johann Lermer wrote:

 Try this:

 auto sharedLog = new FileLogger("/dev/null");


 That defines a new variable right. That's completely different.


Ah, sh..t. Didn't see that.
Mar 13 2023
prev sibling next sibling parent reply WebFreak001 <d.forum webfreak.org> writes:
On Monday, 13 March 2023 at 02:53:00 UTC, Yuxuan Shui wrote:

 Example here:

 https://dlang.org/phobos/std_logger_core.html#.sharedLog

 Result:

 https://run.dlang.io/is/RMtF0I


this changed in one of the most recent releases, but you can 
always expect breaking changes in an std.experimental module
Mar 13 2023
next sibling parent Yuxuan Shui <yshuiv7 gmail.com> writes:
On Monday, 13 March 2023 at 20:32:08 UTC, WebFreak001 wrote:

 On Monday, 13 March 2023 at 02:53:00 UTC, Yuxuan Shui wrote:

 Example here:

 https://dlang.org/phobos/std_logger_core.html#.sharedLog

 Result:

 https://run.dlang.io/is/RMtF0I


 this changed in one of the most recent releases, but you can 
 always expect breaking changes in an std.experimental module


Breaking change is fine... I expect documentation to be updated 
with the change.
Mar 13 2023
prev sibling next sibling parent James Blachly <james.blachly gmail.com> writes:
On 3/13/23 4:32 PM, WebFreak001 wrote:

 On Monday, 13 March 2023 at 02:53:00 UTC, Yuxuan Shui wrote:

 Example here:

 https://dlang.org/phobos/std_logger_core.html#.sharedLog

 Result:

 https://run.dlang.io/is/RMtF0I


 
 this changed in one of the most recent releases, but you can always 
 expect breaking changes in an std.experimental module


It is no longer namespaced under std.experimental, so I would not expect 
breaking changes
Mar 17 2023
prev sibling parent reply mw <mw g.c> writes:
On Monday, 13 March 2023 at 20:32:08 UTC, WebFreak001 wrote:

 On Monday, 13 March 2023 at 02:53:00 UTC, Yuxuan Shui wrote:

 Example here:

 https://dlang.org/phobos/std_logger_core.html#.sharedLog

 Result:

 https://run.dlang.io/is/RMtF0I


 this changed in one of the most recent releases, but you can 
 always expect breaking changes in an std.experimental module


Can you please show the correct usage pattern of this sharedLog, 
in the new release?
Jul 11 2023
parent reply WebFreak001 <d.forum webfreak.org> writes:
On Wednesday, 12 July 2023 at 03:00:11 UTC, mw wrote:

 On Monday, 13 March 2023 at 20:32:08 UTC, WebFreak001 wrote:

 On Monday, 13 March 2023 at 02:53:00 UTC, Yuxuan Shui wrote:

 Example here:

 https://dlang.org/phobos/std_logger_core.html#.sharedLog

 Result:

 https://run.dlang.io/is/RMtF0I


 this changed in one of the most recent releases, but you can 
 always expect breaking changes in an std.experimental module


 Can you please show the correct usage pattern of this 
 sharedLog, in the new release?


what I do now in my code:

```d
static if (__VERSION__ < 2101)
	sharedLog = new FileLogger(io.stderr);
else
	sharedLog = (()  trusted => cast(shared) new 
FileLogger(io.stderr))();
```

not a big fan of the changes and the fact that this got made 
std.logger instead of the previous API, but it works at least
Jul 12 2023
next sibling parent reply mw <m g.c> writes:
On Wednesday, 12 July 2023 at 10:29:50 UTC, WebFreak001 wrote:

 On Wednesday, 12 July 2023 at 03:00:11 UTC, mw wrote:

 On Monday, 13 March 2023 at 20:32:08 UTC, WebFreak001 wrote:

 On Monday, 13 March 2023 at 02:53:00 UTC, Yuxuan Shui wrote:

 Example here:

 https://dlang.org/phobos/std_logger_core.html#.sharedLog

 Result:

 https://run.dlang.io/is/RMtF0I


 this changed in one of the most recent releases, but you can 
 always expect breaking changes in an std.experimental module


 Can you please show the correct usage pattern of this 
 sharedLog, in the new release?


 what I do now in my code:

 ```d
 static if (__VERSION__ < 2101)
 	sharedLog = new FileLogger(io.stderr);
 else
 	sharedLog = (()  trusted => cast(shared) new 
 FileLogger(io.stderr))();
 ```

 not a big fan of the changes and the fact that this got made 
 std.logger instead of the previous API, but it works at least


1) Is this `cast` ugly?
2) if it's a regular file, is this still thread safe to use?


Without the `cast`, I got compiler error

https://run.dlang.io/is/h3S0eb
```
import std.stdio;
import std.algorithm;
import std.logger.core, std.logger.filelogger;
import std.range;

void main()
{
     // sharedLog = new shared(FileLogger)("/dev/null");

     static if (__VERSION__ < 2101)
	sharedLog = new FileLogger(io.stderr);
else
	sharedLog = (()  trusted => new shared(FileLogger)(stderr))();

}
```


onlineapp.d(13): Error: none of the overloads of `__ctor` are 
callable using a `shared` object
/dlang/dmd/linux/bin64/../../src/phobos/std/logger/filelogger.d(40):       
Candidates are: `std.logger.filelogger.FileLogger.this(const(string) fn,
const(LogLevel) lv = LogLevel.all)`
/dlang/dmd/linux/bin64/../../src/phobos/std/logger/filelogger.d(66):           
            `std.logger.filelogger.FileLogger.this(const(string) fn,
const(LogLevel) lv, Flag createFileNameFolder)`
/dlang/dmd/linux/bin64/../../src/phobos/std/logger/filelogger.d(105):          
             `std.logger.filelogger.FileLogger.this(File file, const(LogLevel)
lv = LogLevel.all)`


Sigh, D is so broken on such basic stuff.
Jul 12 2023
parent reply Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Wednesday, July 12, 2023 10:32:22 AM MDT mw via Digitalmars-d wrote:

 On Wednesday, 12 July 2023 at 10:29:50 UTC, WebFreak001 wrote:

 On Wednesday, 12 July 2023 at 03:00:11 UTC, mw wrote:

 On Monday, 13 March 2023 at 20:32:08 UTC, WebFreak001 wrote:

 On Monday, 13 March 2023 at 02:53:00 UTC, Yuxuan Shui wrote:

 Example here:

 https://dlang.org/phobos/std_logger_core.html#.sharedLog

 Result:

 https://run.dlang.io/is/RMtF0I


 this changed in one of the most recent releases, but you can
 always expect breaking changes in an std.experimental module


 Can you please show the correct usage pattern of this
 sharedLog, in the new release?


 what I do now in my code:

 ```d
 static if (__VERSION__ < 2101)

     sharedLog = new FileLogger(io.stderr);

 else

     sharedLog = (()  trusted => cast(shared) new

 FileLogger(io.stderr))();
 ```

 not a big fan of the changes and the fact that this got made
 std.logger instead of the previous API, but it works at least


 1) Is this `cast` ugly?
 2) if it's a regular file, is this still thread safe to use?


 Without the `cast`, I got compiler error

 https://run.dlang.io/is/h3S0eb
 ```
 import std.stdio;
 import std.algorithm;
 import std.logger.core, std.logger.filelogger;
 import std.range;

 void main()
 {
      // sharedLog = new shared(FileLogger)("/dev/null");

      static if (__VERSION__ < 2101)
   sharedLog = new FileLogger(io.stderr);
 else
   sharedLog = (()  trusted => new shared(FileLogger)(stderr))();

 }
 ```


 onlineapp.d(13): Error: none of the overloads of `__ctor` are
 callable using a `shared` object
 /dlang/dmd/linux/bin64/../../src/phobos/std/logger/filelogger.d(40):
 Candidates are: `std.logger.filelogger.FileLogger.this(const(string) fn,
 const(LogLevel) lv = LogLevel.all)`
 /dlang/dmd/linux/bin64/../../src/phobos/std/logger/filelogger.d(66):
                 `std.logger.filelogger.FileLogger.this(const(string) fn,
 const(LogLevel) lv, Flag createFileNameFolder)`
 /dlang/dmd/linux/bin64/../../src/phobos/std/logger/filelogger.d(105):
                  `std.logger.filelogger.FileLogger.this(File file,
 const(LogLevel) lv = LogLevel.all)`


 Sigh, D is so broken on such basic stuff.


In general, objects are not supposed to be constructed as shared or even
really operated on as shared. A shared object is shared across threads, and
as long as that's done without protections in place, it's not thread-safe to
actually do anything with the object. So, in most cases, what you need to do
in order to operate on a shared object is to protect that section of code
with a mutex, cast away shared to get a thread-local reference to the
object, do whatever you need to do to the object while the mutex is locked,
and then before you release the mutex, you make sure that no thread-local
references to the object remain. That way, you don't ever operate on data
while it's actively been shared across threads. The compiler prevents you
from operating on the object through a shared reference precisely because
doing so would not be thread-safe. This does of course result in some
annoying casts, but it allows you to segregate the code that actually
operates on shared objects, and because the language has shared, it prevents
you from accidentally doing operations on shared objects that aren't
thread-safe as long as the code is  safe (since the casts to remove shared
have to be done in  trusted code to be callable by  safe code).

Outside of dealing with atomics, the only time that a shared object can be
operated on when it's still shared is when it's a type that's specifically
designed to work as shared. In that case, its member functions will be
marked with shared so that they can be called when the object is shared, and
instead of you casting away shared to do anything to the object, the mutexes
and casting and whatnot are all handled internally within the member
functions. But whether you're handling the mutexes and casting yourself, or
whether the object itself handles that, the data within the object can't be
operated one as long as it's shared (except for when using atomics).

Unfortunately, because most languages don't have shared in their type
systems (and don't have objects be thread-local by default), shared is often
misunderstood and misused.. And it is annoying to deal with, but that
annoyance is largely on purpose, because it's the type system attempting to
protect you from doing anything that isn't thread-safe. It would of course
be nice if the type system were sophisticated enough to be able to
automatically remove shared for you in some cases (e.g. because it knows
that you have already protected the shared object with a mutex in a
particular section of code), but it's difficult to give the compiler enough
information for it to be able to do that. So, as things stand, it's up to
the programmer to make sure that they've protected shared data appropriately
before casting away shared and operating on it as thread-local while it's
safe to do so.

Ultimately, you have to do basically the same stuff in languages like C or
C++, but in those languages, you don't have the compiler trying to prevent
you from doing anything that isn't thread-safe when it knows that data is
shared across threads. They just leave everything as shared and leave it up
to the programmer to deal with it all correctly. With D, you still have to
get it right in the sections of code where you cast away shared, but the
sections of code where that happens are then much better defined (since they
have to be  trusted if you're using  safe), and in  safe code, you can't
accidentally operate on shared data, because the compiler gives you an error
when you attempt it. So, we're arguably much better off, but there is
certainly some annoyance in the trade-off.

- Jonathan M Davis
Jul 12 2023
next sibling parent reply mw <mw g.c> writes:
On Wednesday, 12 July 2023 at 23:11:00 UTC, Jonathan M Davis 
wrote:


 Sigh, D is so broken on such basic stuff.


 In general, objects are not supposed to be constructed as 
 shared or even really operated on as shared. ...
 a shared reference precisely because doing so would not be 
 thread-safe.
 
 ...

 Ultimately, you have to do basically the same stuff in 
 languages like C or C++, but in those languages, you don't have 
 the compiler trying to prevent you from doing anything that



I think the current D confused the users:

1) why such simple program has compiler errors?


https://run.dlang.io/is/2k9Uvu
```
import std.logger;

void main() {
     std.logger.sharedLog.trace("msg");
}

```

2) it is named `std.logger.sharedLog` from the standard library, 
the users would expect it's thread safe, otherwise, why don't 
just name it `std.logger.Log`, then user would not have such 
wrongful expectations.

3) if it's not provided at all, or not named that way, the users 
will write their own synchronized thread safe loggers. (Since 
standard library does not provide a ready made one out-of-box).


That is why I said: sign, D is so broken on such basic stuff.

The users have to learn it the hard way, and each users have to 
develop their own basic thread safe logging library.
Jul 12 2023
parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Wednesday, July 12, 2023 6:29:31 PM MDT mw via Digitalmars-d wrote:

 On Wednesday, 12 July 2023 at 23:11:00 UTC, Jonathan M Davis

 wrote:

 Sigh, D is so broken on such basic stuff.


 In general, objects are not supposed to be constructed as
 shared or even really operated on as shared. ...
 a shared reference precisely because doing so would not be
 thread-safe.

 ...

 Ultimately, you have to do basically the same stuff in
 languages like C or C++, but in those languages, you don't have
 the compiler trying to prevent you from doing anything that


 I think the current D confused the users:

 1) why such simple program has compiler errors?


 https://run.dlang.io/is/2k9Uvu
 ```
 import std.logger;

 void main() {
      std.logger.sharedLog.trace("msg");
 }

 ```

 2) it is named `std.logger.sharedLog` from the standard library,
 the users would expect it's thread safe, otherwise, why don't
 just name it `std.logger.Log`, then user would not have such
 wrongful expectations.

 3) if it's not provided at all, or not named that way, the users
 will write their own synchronized thread safe loggers. (Since
 standard library does not provide a ready made one out-of-box).


 That is why I said: sign, D is so broken on such basic stuff.

 The users have to learn it the hard way, and each users have to
 develop their own basic thread safe logging library.


Presumably, it's named sharedLog, because it's shared. Being shared does not
imply thread-safety. It just means that the object is shared across threads,
and that means that the compiler is going to give you error messages when
you try to use it in a way that it can't guarantee is thread-safe (which
usually means getting an error when you try to do much of anything with it,
since very few operations are guaranteed to be thread-safe). How
thread-safety is actually achieved when using a type depends on the API of
that type. It could specifically have a shared API and handle all of the
locking internally (which realistically means that sharedLog would have to
be a different type of object from any thread-local logging objects), or it
could leave the locking mechanism up to the user. For better or worse,
std.logger has done the latter. This provides more flexibility but at the
cost of requiring that the user go to extra effort. I can't really comment
on how good or bad a decision that is, since I've never needed to use
std.logger before and don't usually work on applications that have any need
of functionality of that sort, but I can certainly see why it would be
annoying if you just want to have a logger that works across threads without
having to worry about shared. Realistically though, if you're going to be
doing much with threads in D, you really need to learn the ins and outs of
shared. So, while there's certainly an argument that the current situation
with std.logger is suboptimal, the fact that it's giving an error when
trying to use a shared object is the kind of thing that anyone dealing with
threads in D is going to need to understand.

Regardless, there is no need to create a separate logging library to use
sharedLog. It just means that you either have to provide an appropriate
mutex and deal with locking it yourself, casting away shared, etc., or you
need to put a wrapper around it which takes care of that. std.logger
probably should provide such a wrapper itself, but the fact that it doesn't
doesn't mean that you need to create a whole other solution.

- Jonathan M Davis
Jul 12 2023
prev sibling parent reply Danilo <codedan aol.com> writes:
Nice explanation. Thanks, Jonathan M Davis!

After reading https://dlang.org/phobos/std_logger.html
very exactly, we understand that `sharedLog` is the default
internal logger. `sharedLog` can be used to change the default 
logger:
```d
     //auto file = File("logFile.log", "w");
     auto file = stderr; // stdout

     sharedLog = cast(shared)new FileLogger(file);
```
After that, `sharedLog` is not used directly (maybe the docs 
could make this more clear).

Instead, we use:
- log / logf
- trace
- info
- warning / warningf
- critical
- error / errorf
- fatal

Those functions use `sharedLog` internally.

If you look at the implementations, the provided default `Logger`
and `FileLogger` are thread-safe already.
They use mutexes to manage access to the global logger.

So the correct way is to use the global functions or create
local loggers using `auto logger = new WhateverLogger(): 
logger.info("msg");`

A complete example using threads was already provided at:
- 
https://forum.dlang.org/post/lhewpbfniwozsqdvsudc forum.dlang.org
Jul 12 2023
parent Jonathan M Davis <newsgroup.d jmdavisprog.com> writes:
On Wednesday, July 12, 2023 11:13:52 PM MDT Danilo via Digitalmars-d wrote:

 Nice explanation. Thanks, Jonathan M Davis!

 After reading https://dlang.org/phobos/std_logger.html
 very exactly, we understand that `sharedLog` is the default
 internal logger. `sharedLog` can be used to change the default
 logger:
 ```d
      //auto file = File("logFile.log", "w");
      auto file = stderr; // stdout

      sharedLog = cast(shared)new FileLogger(file);
 ```
 After that, `sharedLog` is not used directly (maybe the docs
 could make this more clear).

 Instead, we use:
 - log / logf
 - trace
 - info
 - warning / warningf
 - critical
 - error / errorf
 - fatal

 Those functions use `sharedLog` internally.

 If you look at the implementations, the provided default `Logger`
 and `FileLogger` are thread-safe already.
 They use mutexes to manage access to the global logger.

 So the correct way is to use the global functions or create
 local loggers using `auto logger = new WhateverLogger():
 logger.info("msg");`

 A complete example using threads was already provided at:
 -
 https://forum.dlang.org/post/lhewpbfniwozsqdvsudc forum.dlang.org


Ah, so it uses free functions to handle the locking for sharedLog rather
than having shared member functions. I looked over the code quickly and saw
that it didn't have any shared member functions and concluded that it must
have left the locking issues up to the user. Clearly, I didn't look closely
enough.

My gut reaction to that is that maybe sharedLog should have had a setter
rather than being made public, but either way, since it's shared, no one is
going to accidentally do anything that isn't thread-safe to it.

- Jonathan M Davis
Jul 12 2023
prev sibling parent reply IchorDev <zxinsworld gmail.com> writes:
On Wednesday, 12 July 2023 at 10:29:50 UTC, WebFreak001 wrote:

 not a big fan of the changes and the fact that this got made 
 std.logger instead of the previous API, but it works at least


How would you say the old API was better? (I didn’t use it for 
myself because the stuff in std.experimental usually has an 
unreliable API)
Jul 17 2023
parent WebFreak001 <d.forum webfreak.org> writes:
On Tuesday, 18 July 2023 at 03:45:36 UTC, IchorDev wrote:

 On Wednesday, 12 July 2023 at 10:29:50 UTC, WebFreak001 wrote:

 not a big fan of the changes and the fact that this got made 
 std.logger instead of the previous API, but it works at least


 How would you say the old API was better? (I didn’t use it for 
 myself because the stuff in std.experimental usually has an 
 unreliable API)


oh I think the new API is worse here primarily just the shared 
change that is needed in absolutely _every_ project that uses 
`trace`, since otherwise `trace` isn't able to output anything 
unless you set the logger. (globalLogLevel can only reduce the 
logs, but you can't enable trace log this way)

Seeing that something this basic didn't work without the  trusted 
hack, but still getting into std.logger in the same release (!) 
that introduced these changes really doesn't seem to fit with how 
phobos works. Usually modules need to be field tested to get into 
phobos, which certainly has some upsides and downsides, but for 
this module, basically it got rewritten and immediately placed 
into standard phobos. It kind of feels that in large projects how 
the log levels work might eventually be found to be broken in a 
way that can't be fixed.

Additionally it looks like it's quite a runtime-heavy 
implementation, I think we might have been able to do a better 
implementation that can be better optimized into less runtime 
overhead. Don't know how much we can optimize the current 
std.logger implementation in the future, but the whole dynamic 
structure with like thread local logger forwarding to global 
sharedLogger, etc. make it seem to be less able to do it than 
something like e.g. passing logger objects/templates around.
Jul 20 2023
prev sibling next sibling parent =?UTF-8?Q?Christian_K=c3=b6stlin?= <christian.koestlin gmail.com> writes:
Thanks for all the explanations in the thread.

Reading again through
https://dlang.org/phobos/std_logger.html I think the chapter Basic 
Logging explains nicely how to work with sharedLog (but leaves out the 
shared detail).

Logging is something very dear to me and I think it's important to have 
that done right.

imho there are only two main usecases for logging:
1. logging to debug stuff (sometimes logging is easier than unittesting)
2. logging in production
those two usecases have quite different requirements (e.g. 1. needs to 
print as much out as it can before the program eventually crashes, 
whereas 2. needs to be as fast as possible, to not drain resources from 
the real program).

I think the current std.logger might be a good candidate as is for use 
case 1. It really tries hard to get the data out. I do not understand 
why FileLogger internally takes some more locks on the file (several 
calls to lockingTextWriter for one logmessage) (as my understanding is, 
that the locking is already done earlier on). Perhaps Robert can explain 
us why that is needed?

Also interesting is, that the sharedLog is not directly used, but with a 
thread local StdForwardLogger. For me the real thread safe locking 
happens here.

For usecase 2 I do not see how the existing loggers are a very good fit, 
as for my taste too much locking is going on. Still I think the API is 
powerfull enough to e.g. just forward the logmessages (that are not 
filtered out) from StdForwardLogger to a logger thread, that 
asynchronously outputs the log messages.

It would probably be great to also have a rotating filelogger, as this 
is something most people are used to.


Kind regards,
Christian
Jul 13 2023
prev sibling parent o3o <dlang orfeo.fastmail.com> writes:
On Monday, 13 March 2023 at 02:53:00 UTC, Yuxuan Shui wrote:

 Example here:

 https://dlang.org/phobos/std_logger_core.html#.sharedLog

 Result:

 https://run.dlang.io/is/RMtF0I


Also this https://run.dlang.io/is/NpBbgB
```
import std.logger.core;
void main() {
     log("output: no"); // no output
     info("output: yes");
}
```
result:
```
2023-07-20T12:40:07.653 [info] onlineapp.d:7:main output: yes
```
doesn't work.
Jul 20 2023