www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Suggestion/proposal regarding std.logger candidate

reply "David Osborne" <krendilboove gmail.com> writes:
Yesterday, in the livestream chat, there was some discussion about
the current std.logger proposal.  The consensus seemed to be that
it's probably fine for most use cases and it's the best proposal
we've had so far, it doesn't look suitable for heavy duty,
enterprise-grade logging.  In an ideal world, the std.logger 
module
in phobos would be flexible enough to build an enterprise-grade
logging library on top of, but it seems the current proposal isn't
there yet.  I got to thinking about how to make the logging system
more flexible, and this is what I came up with.

I identified three key parts to a logger: the filter, the format,
and the sink.

     log.trace(...) ----> Filter ----> Format ----> Sink

The filter determines which log messages to act on, and which to
ignore.  Usually this is done based on the log level, but one use
case that was brought up was automatically sending an email to the
developers only when a particular exception occurred (i.e. filter
based on Exception type).  Currently, std.logger implements the
filtering for you, based on the global log level, and the log 
level
of that particular logger.

The format determines what the output of the log message looks 
like.
Usually, this winds up being some sort of call to formattedWrite,
though there is no need to assume that the output must be a 
string,
or even human-readable.  Currently, formatting tends to be
implemented as part of Logger.writeLogMsg, and is specific to each
Logger implementation.

The sink is where the log message goes.  Popular destinations
include stderr, /dev/null, some text file somewhere, or a central
logging facility provided by your OS.  Currently, the sink is
specified when implementing Logger.writeLogMsg.

Based on these three parameters, I came up with the following 
proof
of concept implementation, built on top of the current proposal:

public class Clogger(alias format, alias filter, Sink) : Logger
     if( isLogFilter!filter && isLogFormat!format && 
isLogSink!(Sink, format) )
{

     public this(Sink sink, string newName, LogLevel lv) {
         super(newName, lv);
         this.sink = sink;
     }

     override
     public void writeLogMsg(ref LoggerPayload payload) {
         import std.algorithm : copy;

         if( this.filterFun(payload) ) {
             sink = formatFun(payload).copy(sink);
         }
     }

     private:
     Sink sink;
     alias formatFun = unaryFun!format;
     alias filterFun = unaryFun!filter;

}

The full code is at https://github.com/krendil/clogger.

As you can see, there are three template parameters, that 
correspond
to the three components identified above.

- filter is a predicate that takes a LoggerPayload and returns 
true
     if that log message should be logged.

- format takes a LoggerPayload and converts it to something
     suitable for output, in the form of a Range.

- Sink is an OutputRange that has the same element type as the
     the output of format.

I think there are a few benefits to this setup:
   - Separation of format from sink
   - Custom, arbitrary filtering
   - Simpler to implement new Loggers

With this setup, multiplexing can either be done at the Logger
level, as it is currently, or at the Sink level. Log messages can
still be converted to any sort of output. Aliases or convenience
functions could be created for common Logger types (I currently 
have
convenience functions for a multiplexer, a stdout logger and a
generic dchar range logger).

Destroy :)
May 24 2014
next sibling parent "David Osborne" <krendilboove gmail.com> writes:
Just to be clear, what I am suggesting/proposing is that the 
current Logger class in std.logger be altered so that it looks 
more like this, with filter, format and sink template arguments 
that determine its behaviour, instead of using an 
inheritance-based approach.
May 24 2014
prev sibling next sibling parent "HaraldZealot" <harald_zealot tut.by> writes:
The logger is very good and need thing. But has anybody an idea 
how to preserve purity of functions?
May 24 2014
prev sibling next sibling parent "Gary Willoughby" <Dev nomad.so> writes:
On Saturday, 24 May 2014 at 10:04:33 UTC, David Osborne wrote:
 Just to be clear, what I am suggesting/proposing is that the 
 current Logger class in std.logger be altered so that it looks 
 more like this, with filter, format and sink template arguments 
 that determine its behaviour, instead of using an 
 inheritance-based approach.

Please be aware of KISS, YAGNI and other beware of over engineering principles
May 24 2014
prev sibling next sibling parent "Tobias Pankrath" <tobias pankrath.net> writes:
On Saturday, 24 May 2014 at 11:58:35 UTC, Gary Willoughby wrote:
 On Saturday, 24 May 2014 at 10:04:33 UTC, David Osborne wrote:
 Just to be clear, what I am suggesting/proposing is that the 
 current Logger class in std.logger be altered so that it looks 
 more like this, with filter, format and sink template 
 arguments that determine its behaviour, instead of using an 
 inheritance-based approach.

Please be aware of KISS, YAGNI and other beware of over engineering principles

Why not just copy the design of boost::log?
May 24 2014
prev sibling next sibling parent Jeremy Powers via Digitalmars-d <digitalmars-d puremagic.com> writes:
--f46d043893ede2f99704fa2aafc7
Content-Type: text/plain; charset=UTF-8

On Sat, May 24, 2014 at 2:57 AM, David Osborne via Digitalmars-d <
digitalmars-d puremagic.com> wrote:

 Destroy :)

Overall, this approach seems good. Not sure I like forcing one format for all sinks though - it is often desired/necessary to have different sinks (aka appenders) format the log messages differently. Aside: I'd recommend anyone looking at logging examine the logback architecture, if they haven't already: http://logback.qos.ch/manual/architecture.html Not saying std.logger should look like this, but shows one fairly well thought out solution. --f46d043893ede2f99704fa2aafc7 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">On S= at, May 24, 2014 at 2:57 AM, David Osborne via Digitalmars-d <span dir=3D"l= tr">&lt;<a href=3D"mailto:digitalmars-d puremagic.com" target=3D"_blank">di= gitalmars-d puremagic.com</a>&gt;</span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-= left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p= adding-left:1ex">Destroy :)<br> </blockquote></div><br></div><div class=3D"gmail_extra">Overall, this appro= ach seems good.</div><div class=3D"gmail_extra"><br></div><div class=3D"gma= il_extra">Not sure I like forcing one format for all sinks though - it is o= ften desired/necessary to have different sinks (aka appenders) format the l= og messages differently.</div> <div class=3D"gmail_extra"><br></div><div class=3D"gmail_extra">Aside: I&#3= 9;d recommend anyone looking at logging examine the logback architecture, i= f they haven&#39;t already:</div><div class=3D"gmail_extra"><a href=3D"http= ://logback.qos.ch/manual/architecture.html">http://logback.qos.ch/manual/ar= chitecture.html</a><br> </div><div class=3D"gmail_extra">Not saying std.logger should look like thi= s, but shows one fairly well thought out solution.</div></div> --f46d043893ede2f99704fa2aafc7--
May 24 2014
prev sibling next sibling parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Sat, 24 May 2014 11:42:30 +0000
HaraldZealot via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 The logger is very good and need thing. But has anybody an idea
 how to preserve purity of functions?

That's impossible if you're doing I/O. By definition, that has to access mutable, global state. The only way to do it would be to go the Haskell route and use something like monads - you'd have to basically pass an object around which built up the logging messages without actually logging anything and then have an impure function higher up (which is calling these pure functions) make the function call which actually caused the queued up logging to be output. Logging is basically anti-purity. So, unless you do something like I just described (which I doubt much of anyone will), then logging anything in your code means giving up purity. - Jonathan M Davis
May 24 2014
prev sibling next sibling parent Robert BuRnEr Schadek via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 05/24/2014 02:57 AM, David Osborne via Digitalmars-d wrote:
 Yesterday, in the livestream chat, there was some discussion about
 the current std.logger proposal.  The consensus seemed to be that
 it's probably fine for most use cases and it's the best proposal
 we've had so far, it doesn't look suitable for heavy duty,
 enterprise-grade logging.  In an ideal world, the std.logger module
 in phobos would be flexible enough to build an enterprise-grade
 logging library on top of, but it seems the current proposal isn't
 there yet.  I got to thinking about how to make the logging system
 more flexible, and this is what I came up with.

 I identified three key parts to a logger: the filter, the format,
 and the sink.

     log.trace(...) ----> Filter ----> Format ----> Sink

 The filter determines which log messages to act on, and which to
 ignore.  Usually this is done based on the log level, but one use
 case that was brought up was automatically sending an email to the
 developers only when a particular exception occurred (i.e. filter
 based on Exception type).  Currently, std.logger implements the
 filtering for you, based on the global log level, and the log level
 of that particular logger.

 The format determines what the output of the log message looks like.
 Usually, this winds up being some sort of call to formattedWrite,
 though there is no need to assume that the output must be a string,
 or even human-readable.  Currently, formatting tends to be
 implemented as part of Logger.writeLogMsg, and is specific to each
 Logger implementation.

 The sink is where the log message goes.  Popular destinations
 include stderr, /dev/null, some text file somewhere, or a central
 logging facility provided by your OS.  Currently, the sink is
 specified when implementing Logger.writeLogMsg.

 Based on these three parameters, I came up with the following proof
 of concept implementation, built on top of the current proposal:

 public class Clogger(alias format, alias filter, Sink) : Logger
     if( isLogFilter!filter && isLogFormat!format && isLogSink!(Sink,
 format) )
 {

     public this(Sink sink, string newName, LogLevel lv) {
         super(newName, lv);
         this.sink = sink;
     }

     override
     public void writeLogMsg(ref LoggerPayload payload) {
         import std.algorithm : copy;

         if( this.filterFun(payload) ) {
             sink = formatFun(payload).copy(sink);
         }
     }

     private:
     Sink sink;
     alias formatFun = unaryFun!format;
     alias filterFun = unaryFun!filter;

 }

 The full code is at https://github.com/krendil/clogger.

 As you can see, there are three template parameters, that correspond
 to the three components identified above.

 - filter is a predicate that takes a LoggerPayload and returns true
     if that log message should be logged.

 - format takes a LoggerPayload and converts it to something
     suitable for output, in the form of a Range.

 - Sink is an OutputRange that has the same element type as the
     the output of format.

 I think there are a few benefits to this setup:
   - Separation of format from sink
   - Custom, arbitrary filtering
   - Simpler to implement new Loggers

 With this setup, multiplexing can either be done at the Logger
 level, as it is currently, or at the Sink level. Log messages can
 still be converted to any sort of output. Aliases or convenience
 functions could be created for common Logger types (I currently have
 convenience functions for a multiplexer, a stdout logger and a
 generic dchar range logger).

 Destroy :)

May 24 2014
prev sibling next sibling parent Robert BuRnEr Schadek via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 05/24/2014 03:04 AM, David Osborne via Digitalmars-d wrote:
 Just to be clear, what I am suggesting/proposing is that the current
 Logger class in std.logger be altered so that it looks more like this,
 with filter, format and sink template arguments that determine its
 behaviour, instead of using an inheritance-based approach.

May 24 2014
prev sibling next sibling parent Robert BuRnEr Schadek via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 05/24/2014 05:39 AM, Tobias Pankrath via Digitalmars-d wrote:
 On Saturday, 24 May 2014 at 11:58:35 UTC, Gary Willoughby wrote:
 On Saturday, 24 May 2014 at 10:04:33 UTC, David Osborne wrote:
 Just to be clear, what I am suggesting/proposing is that the current
 Logger class in std.logger be altered so that it looks more like
 this, with filter, format and sink template arguments that determine
 its behaviour, instead of using an inheritance-based approach.

Please be aware of KISS, YAGNI and other beware of over engineering principles

Why not just copy the design of boost::log?

May 24 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Sat, 24 May 2014 20:10:21 -0700, Jonathan M Davis via Digitalmars-d  
<digitalmars-d puremagic.com> wrote:

 On Sat, 24 May 2014 11:42:30 +0000
 HaraldZealot via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 The logger is very good and need thing. But has anybody an idea
 how to preserve purity of functions?

That's impossible if you're doing I/O. By definition, that has to access mutable, global state. The only way to do it would be to go the Haskell route and use something like monads - you'd have to basically pass an object around which built up the logging messages without actually logging anything and then have an impure function higher up (which is calling these pure functions) make the function call which actually caused the queued up logging to be output. Logging is basically anti-purity. So, unless you do something like I just described (which I doubt much of anyone will), then logging anything in your code means giving up purity.

IIRC, logging with 'debug' should work. -Steve
May 24 2014
prev sibling next sibling parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Sat, 24 May 2014 23:56:39 -0700
Steven Schveighoffer via Digitalmars-d <digitalmars-d puremagic.com>
wrote:

 IIRC, logging with 'debug' should work.

Yes, logging in debug blocks would work, because debug blocks ignore purity constraints (which means that they _can_ cause a function to have weird behavior if you're not careful, because the compiler is still free to do things like optimizations based on purity). It's that way to help with debugging (since not being able to print anything when debugging can be a _real_ pain), but since logging is frequently used for purposes other than simply debugging a program, putting logging statements in debug blocks doesn't really solve the problem - and abusing debug blocks by using them specifically for logging purposes could cause some weird stuff to happen if you're not careful, since you'd be doing impure stuff in a pure function and thereby violating guarantees that the compiler assumes hold. Realistically, if you're looking to do much logging in your program, you're probably going to have to give up on using pure in a lot of places. - Jonathan M Davis
May 25 2014
prev sibling parent "Tobias Pankrath" <tobias pankrath.net> writes:
On Sunday, 25 May 2014 at 06:07:04 UTC, Robert BuRnEr Schadek via 
Digitalmars-d wrote:
 On 05/24/2014 05:39 AM, Tobias Pankrath via Digitalmars-d wrote:
 On Saturday, 24 May 2014 at 11:58:35 UTC, Gary Willoughby 
 wrote:
 On Saturday, 24 May 2014 at 10:04:33 UTC, David Osborne wrote:
 Just to be clear, what I am suggesting/proposing is that the 
 current
 Logger class in std.logger be altered so that it looks more 
 like
 this, with filter, format and sink template arguments that 
 determine
 its behaviour, instead of using an inheritance-based 
 approach.

Please be aware of KISS, YAGNI and other beware of over engineering principles

Why not just copy the design of boost::log?


That's just the surface. See: http://boost-log.sourceforge.net/libs/log/doc/html/log/design.html
May 25 2014