www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Another Log implementation

reply Vincent <thornik gmail.com> writes:
Hello, guys!
I see a lot of discussions about std.Log implementation, that I couldn't  
pass along. I use logs a long time, but some ideas of opponents looks to  
me obscure.
Of course, different commands have different requirements, but I think  
that such DUMB AND SIMPLE task as logging should not transform to 'one  
fits all' dump of monster classes.
Logging must conform most commonly used usages, all other 'features' is  
the pain of 'offbeat' developers. Also my 'fresh look' at 'verbosity':  
this is obsolete point of programmers from 70-th. Modern development has  
no time for 'verbosity' like this:

log(Verbosity1, "var i greater than zero")
log(Verbosity2, "var i is "~ to!string(i))
...etc...

My second point is that libraries must benefit from used language, what  
leads to the third point: logging from other languages(libraries) must be  
used only as an 'idea source', since their logging based on capabilities  
of _their_ languages.

In this light I made this draft for logging, may be it'll cold/delight  
people with its simplicity.
Let's start from usage:
====================================================================================
version(logging) { // we benefit from D (conditional compilation)
     // Different 'severity' emulated with different log objects
     auto logWrn = new Log(new FileLogEngine(`c:\res.txt`), new  
DBLogEngine(`server=.;database=ABC`));// log can be done to many  
destinations
     auto logErr = new Log();// by default log outputs to the screen.
     auto logCrash = new Log(new EmailLogEngine(`smtp.myserver.com`);// log  
output is not only a file!
}
try {
    // some action
} catch (LightException lex) {
     version(logging)    logWrn(`Problem!`); // simple usage. Non-critical  
messages going to its own log.
} catch (Exception ex) {
     version(logging)    logCrash(`Something serious happen! %s`, ex);//  
logging with formatting. No necessaries for 'severity' - critical stuff  
gone to critical log only.
}

version(logging) {  writeln(`Log disabled`); logDbg.IsEnabled = false; }
version(logging)	logDbg(`Disabled message`);// logging is disabled at  
runtime
====================================================================================

Like it? Here is the code! http://pastebin.com/WwkrD02g
Note on recordTemplate field - customization for log records, which allow  
even XML.

Since it's 'one hour draft', be lenient in details - my goal is to show  
conception, but of course I'm happy to discuss any improvements!

Vincent.
May 28 2011
next sibling parent reply Jimmy Cao <jcao219 gmail.com> writes:
--0016363b7a4aa693f804a4586654
Content-Type: text/plain; charset=ISO-8859-1

I like this design.

Can you add an HTTPLogEngine to send data using POST/GET?  That would be
useful.
Perhaps EmailLogEngine should be renamed to SMTPLogEngine.

--0016363b7a4aa693f804a4586654
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I like this design.<div><br></div><div>Can you add an HTTPLogEngine to send=
 data using POST/GET? =A0That would be useful.</div><div>Perhaps EmailLogEn=
gine should be renamed to SMTPLogEngine.</div>

--0016363b7a4aa693f804a4586654--
May 28 2011
parent reply David Nadlinger <see klickverbot.at> writes:
On 5/29/11 9:50 PM, Vincent wrote:
 Perhaps EmailLogEngine should be renamed to SMTPLogEngine.

And next year we will add IMAPLogEngine? :) Nope, I prefer just 'Email' - more generic approach.

Uh, IMAP has nothing to do with sending mail, it's just a protocol for accessing your server-side mailbox, like POP3. David
May 29 2011
next sibling parent reply Daniel Gibson <metalcaedes gmail.com> writes:
Am 30.05.2011 00:55, schrieb Vincent:
 On Sun, 29 May 2011 22:05:23 +0200, David Nadlinger <see klickverbot.at>
 wrote:
 
 Uh, IMAP has nothing to do with sending mail

For bores: Just an example that 'mail' doesn't mean 'SMTP'. Naming functions based on low level details leads to unavoidable refactoring.

In the contrary: Assuming that an alternative to SMTP will emerge, you can't assume that anyone supports it (just like not every server that supports pop3 also supports IMAP). So if it's just "EmailLogEngine" or whatever and it's using SMTP, changing it to use the SMTP-alternative will break the logger for everyone whose mail-server only supports SMTP. So calling it SMTPServer is smarter - if you ever want to use something else for mail-based logging just implemente a FooBarLogEngine and change the parts of your code (probably it's just one part, anyway) to use the FooBarLogEngine instead of the SMTPLogEngine. Cheers, - Daniel
May 29 2011
parent Daniel Gibson <metalcaedes gmail.com> writes:
Am 30.05.2011 11:18, schrieb Vincent:
 On Mon, 30 May 2011 01:32:44 +0200, Daniel Gibson
 <metalcaedes gmail.com> wrote:
 
 So if it's just "EmailLogEngine" or whatever and it's using SMTP,
 changing it to use the SMTP-alternative....

It's just a wrong view on a whole conception. Log uses different KIND of targets: file, database(do you care which one??), mail (is it important which proto??), etc. 'Mail' is an abstraction of "something which can send message to the human". So... instead of making annoying set of classes for all known mail protocols, we use ONE mail class with appropriate constructors for all imaginable protocols. Say: enum MailProto { SMTP, IMAP } class EmailLogEngine : LogEngine { public this(MailProto type, string user, string psw, string host, int port = 0) { if (port == 0) switch(type) { case SMTP: port = 25; break; } // initialize mail server structures } override public void Write(string msg) { // send message, using appropriate proto } }

How is auto logCrash = new Log(new EmailLogEngine(MailProto.SMTP, "smtp.myserver.com")); better than auto logCrash = new Log(new SMPTLogEngine("smtp.myserver.com")); ?
May 30 2011
prev sibling parent David Nadlinger <see klickverbot.at> writes:
On 5/30/11 4:48 PM, Graham Fawcett wrote:
 On Sun, 29 May 2011 22:05:23 +0200, David Nadlinger wrote:

 On 5/29/11 9:50 PM, Vincent wrote:
 Perhaps EmailLogEngine should be renamed to SMTPLogEngine.

And next year we will add IMAPLogEngine? :) Nope, I prefer just 'Email' - more generic approach.

Uh, IMAP has nothing to do with sending mail, it's just a protocol for accessing your server-side mailbox, like POP3.

Actually, an IMAP-based logging sink would be conceivable. Unlike POP3, you can actively store documents into an IMAP mailbox. As an example, "imapfs" is a FUSE-based filesystem which stores files to an IMAP host. Deciding whether this is a good idea is left as an exercise. :)

Yeah, I know that APPENDing messages to IMAP mailboxes is possible, and an IMAP-based logging sink could certainly be implemented. But in my opinion, uploading a message to an user's mailbox is quite a different concept semantically than sending mail via SMTP – for example, you could additionally support parameters for the target mailbox, etc., that would make no sense for just sending a message via SMTP. I didn't know that there are IMAP-based FUSE filesystems out there, though, thanks for mentioning – an … uhm … interesting idea. ;) David
May 30 2011
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 05/28/2011 10:08 AM, Vincent wrote:
 Hello, guys!
 I see a lot of discussions about std.Log implementation, that I couldn't
 pass along. I use logs a long time, but some ideas of opponents looks to
 me obscure.
 Of course, different commands have different requirements, but I think
 that such DUMB AND SIMPLE task as logging should not transform to 'one
 fits all' dump of monster classes.

I salute the attitude and approach that simple things should be kept simple. Forgetting about it is what spoils logging frameworks like log4j and pantheios.
 Logging must conform most commonly used usages, all other 'features' is
 the pain of 'offbeat' developers. Also my 'fresh look' at 'verbosity':
 this is obsolete point of programmers from 70-th. Modern development has
 no time for 'verbosity' like this:

 log(Verbosity1, "var i greater than zero")
 log(Verbosity2, "var i is "~ to!string(i))
 ...etc...

Verbosity is not very often used indeed but I'd guess many people got used to it because it's present in most libraries. What I do see enjoying usage is the notion of severity levels.
 My second point is that libraries must benefit from used language, what
 leads to the third point: logging from other languages(libraries) must
 be used only as an 'idea source', since their logging based on
 capabilities of _their_ languages.

 In this light I made this draft for logging, may be it'll cold/delight
 people with its simplicity.
 Let's start from usage:

In many ways logging is the "hello, world" of library design so it is a very popular topic. There are many possible approaches that are difficult to compare. At one extreme we have people who can't even fathom the notion: "Here's my logging library: it's called writeln". At the other extreme we have monsters like pantheios (88KLOC). That being said, simple should be not confused with simplistic. An informal logging discipline based on writeln is practically inconceivable for systematic use by a team larger than a couple of persons (as everyone will choose their own format, place to log to, notion of severity and verboseness etc). Furthermore, a logging library with a "simple" implementation is not necessarily "simple" because it may actually complicate matters for its client, sometimes in surprising ways. In detail (I'm talking about the sketch at http://pastebin.com/WwkrD02g): * The library doesn't take care of capturing the origin of the log (file/module/line) so the user must pass them manually. * The library requires version(logging) to control enabling. Essentially the client would need to prefix _all_ logging lines with version(logging) in order to benefit of compile-time configurability. This is verbosity pushed onto the user, not confined to the library. * The library always evaluates arguments, even when logging is dynamically disabled. This makes the library unnecessarily slow. (The mechanism of generating one new string from a template with each call is also slow, but that can be fixed modularly.) * The library does not define a notion of a severity hierarchy, leaving it to the user to define one. But the absence of such a hierarchy also means there is no library-level notion thereof, which means the user must take care of enabling and disabling e.g. the info level when the error level is disabled. Arguably defining a hierarchy could be considered outside the scope of a "core" logging library, but that means more work for the casual user of the library to achieve a common task. Improving on such issues is possible but inevitably the simplicity of the implementation erodes, to the point that all logging libraries that have a minimum of reasonable features have the "same" complexity, with some wiggle room given by implementation particulars. So the real trick is to know where the feature list must stop. Without claiming to be exactly right, I believe yours stops too early and pantheios stops too late. Also without claiming to be right, although I find it entirely explainable that everyone believes they can write the perfect logging library (unlike everybody else), I believe something close to Jose's library (after community feedback) will become the standard library for logging, so we should at best we consolidate efforts behind it. Thanks, Andrei
May 29 2011
next sibling parent reply Vincent <thornik gmail.com> writes:
Hello, Andrei! Very appreciate you spent time on my work.
I agree with some your critique, sure it's just a question of  
improvements. My main question is acceptance of my log library as a whole  
conception, since it's simple and 10 from 10 programmers can understand  
its sources (I keep it as very serious goal).
Let's discuss doubtful points...

 Verbosity is not very often used indeed but I'd guess many people got  
 used to it because it's present in most libraries.

Habit is not the rule. You must remember those days when every DOS editor show time in menu bar. WHY? Just because everybody did it. I see nothing bad to loose this 'verbosity' as archaic feature. As I said before, 'special demands' is the pain of 'special programmers' - if they REALLY cannot live w/o 'verbosity', they can add it in 3 LOC.
 * The library doesn't take care of capturing the origin of the log  
 (file/module/line) so the user must pass them manually.

I've read posts related __LINE__s in log and don't see any reason in this feature. Why? 1. It's not direct task of 'logging'. Log doesn't care about your lines/files just because you're too negligent to control what and where you log - its task is just assemble together runtime info. 2. Nobody prevent you from doing it! Just make 'mixin' with __FILE__ and use. 3. In my practice I didn't see so low-level approach for debugging - well organised log shows enough, why people care about number of code line in editor?? Funny, heh... 4. Logging is not only "We have a problem in line 10!" - it's "trace of life" for our program.
 * The library requires version(logging) to control enabling. Essentially  
 the client would need to prefix _all_ logging lines

Agree completely. But something says to me that it's just a my little knowledge of D (and it's true). Sure there is some feature to make log calls 'invisible' without 'version' prefix - I just used feature I sure about. May be mixins will help? This is the place where I need help of community.
 * The library always evaluates arguments, even when logging is  
 dynamically disabled.

First, let's remember Knuth's "Premature Optimization" warning - say "it's slow" only when it's really slow down your program. Second, remember use case of this "runtime switch": "ALL logging is enabled (read "everywhere it's slow"), but at this place it's disabled (and who cares how slow it is?)". Third, if you use log, you NEED this info, despite how fast it was calculated. And even if you jump over with 'lazy' arguments, they HAVE to be calculated - you wait 'em anyway! So... what a benefit you have from laziness? And hell, if you're good programmer, you'll never write critical section like log("a"); log("b"); ...rather you accumulate log info and output it at once. I agree with usefullness of 'lazy' stuff, but not in this case.
 * The library does not define a notion of a severity hierarchy

If you look at use cases of this 'severity', there is no 'hierarchy' (look at usage of my library) - it's again "habits" of 70-th. Just obvious example from real life: I log all 'info' messages about client actions. But somewhere I get database exception - my fault, logged as 'critical'. Together with two levels above I have 'warning' level about problems like 'this person has no surname'. Question is: "Why I must dig in 'warning' trash, when I just need info what client did and which critical problem happen?". As you see, 'hierarchy' is just "flat" view on problem - there is independed severities and _combination_ of 'em. Again, no apocalypse happen - who need this feature, that must spent time on it.
 Arguably defining a hierarchy could be considered outside the scope of a  
 "core" logging library, but that means more work for the casual user of  
 the library to achieve a common task.

Exactly. You cannot include in simple 'string assembler' all functionality (esp. legacy point of view - time is changed!). Simple requirements must lead to simple usage - if you don't use 'verbosity', why you should pass parameters in every log call?
 I believe something close to Jose's library (after community feedback)  
 will become the standard library for logging, so we should at best we  
 consolidate efforts behind it.

What if I don't believe? :) We'll spent time on functions, used by 1% of 'old shoes' and give a bad example for next generations. I don't pretend I wrote the best library ever, but I kept on rule 'implement only necessary stuff' and I have a reason for every decision. Well, I don't want to press anybody, but before you completely give up my implementation I wanna be smashed with serious arguments how bad my design is. :) (may be it'll be good lesson for another designs!) Good luck for all community!
May 29 2011
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 05/29/2011 04:52 PM, Vincent wrote:
 Hello, Andrei! Very appreciate you spent time on my work.
 I agree with some your critique, sure it's just a question of
 improvements. My main question is acceptance of my log library as a
 whole conception, since it's simple and 10 from 10 programmers can
 understand its sources (I keep it as very serious goal).

Why? 10 out of 10 would be more interested in using the library effectively instead of reading its source. Clearly keeping the source readable is desirable, but not an overriding goal. It's probably the last thing you care about after having taken care of everything.
 Let's discuss doubtful points...

 Verbosity is not very often used indeed but I'd guess many people got
 used to it because it's present in most libraries.

Habit is not the rule. You must remember those days when every DOS editor show time in menu bar. WHY? Just because everybody did it. I see nothing bad to loose this 'verbosity' as archaic feature. As I said before, 'special demands' is the pain of 'special programmers' - if they REALLY cannot live w/o 'verbosity', they can add it in 3 LOC.

You mentioned you don't care about verbosity levels. I added that I haven't found a lot of use for them either. Next thing you know, you're presupposing your own assumption. Isn't the data set a tad small to draw conclusions from?
 * The library doesn't take care of capturing the origin of the log
 (file/module/line) so the user must pass them manually.

I've read posts related __LINE__s in log and don't see any reason in this feature. Why? 1. It's not direct task of 'logging'. Log doesn't care about your lines/files just because you're too negligent to control what and where you log - its task is just assemble together runtime info. 2. Nobody prevent you from doing it! Just make 'mixin' with __FILE__ and use. 3. In my practice I didn't see so low-level approach for debugging - well organised log shows enough, why people care about number of code line in editor?? Funny, heh... 4. Logging is not only "We have a problem in line 10!" - it's "trace of life" for our program.

Without automated headers including location of the log line and date/time, one can't even call that a logging library. It's an text streaming facility.
 * The library requires version(logging) to control enabling.
 Essentially the client would need to prefix _all_ logging lines

Agree completely. But something says to me that it's just a my little knowledge of D (and it's true). Sure there is some feature to make log calls 'invisible' without 'version' prefix - I just used feature I sure about. May be mixins will help? This is the place where I need help of community.

The help can be readily absorbed from reading the existing proposal.
 * The library always evaluates arguments, even when logging is
 dynamically disabled.

First, let's remember Knuth's "Premature Optimization" warning - say "it's slow" only when it's really slow down your program.

Evaluating arguments always vs. not at all is a matter of principles.
 Second, remember use case of this "runtime switch": "ALL logging is
 enabled (read "everywhere it's slow"), but at this place it's disabled
 (and who cares how slow it is?)".

It is very important that the program is as fast as it can when logging is disabled. I don't understand how one could argue sensibly otherwise.
 Third, if you use log, you NEED this info, despite how fast it was
 calculated. And even if you jump over with 'lazy' arguments, they HAVE
 to be calculated - you wait 'em anyway! So... what a benefit you have
 from laziness?

With lazy evaluation you can arrange that complex expressions inside the log calls are not evaluated.
 And hell, if you're good programmer, you'll never write critical section
 like
 log("a");
 log("b");
 ...rather you accumulate log info and output it at once.
 I agree with usefullness of 'lazy' stuff, but not in this case.

Sometimes you do need to insert log expressions in relatively critical places for debugging purposes.
 * The library does not define a notion of a severity hierarchy

If you look at use cases of this 'severity', there is no 'hierarchy' (look at usage of my library) - it's again "habits" of 70-th. Just obvious example from real life: I log all 'info' messages about client actions. But somewhere I get database exception - my fault, logged as 'critical'. Together with two levels above I have 'warning' level about problems like 'this person has no surname'. Question is: "Why I must dig in 'warning' trash, when I just need info what client did and which critical problem happen?". As you see, 'hierarchy' is just "flat" view on problem - there is independed severities and _combination_ of 'em. Again, no apocalypse happen - who need this feature, that must spent time on it.

Your example describes a trivial matter. Often times, the trace leading to a critical problem is indicative of the issue in more complicated applications.
 Arguably defining a hierarchy could be considered outside the scope of
 a "core" logging library, but that means more work for the casual user
 of the library to achieve a common task.

Exactly. You cannot include in simple 'string assembler' all functionality (esp. legacy point of view - time is changed!). Simple requirements must lead to simple usage - if you don't use 'verbosity', why you should pass parameters in every log call?

Who does? Did you skim the existing proposal at all?
 I believe something close to Jose's library (after community feedback)
 will become the standard library for logging, so we should at best we
 consolidate efforts behind it.

What if I don't believe? :) We'll spent time on functions, used by 1% of 'old shoes' and give a bad example for next generations. I don't pretend I wrote the best library ever, but I kept on rule 'implement only necessary stuff' and I have a reason for every decision. Well, I don't want to press anybody, but before you completely give up my implementation I wanna be smashed with serious arguments how bad my design is. :) (may be it'll be good lesson for another designs!) Good luck for all community!

Same here, and welcome. First, this seems to be the third time you are referring negatively about age. That is never in good taste, and can at best be tolerated if it comes from a position of exceptional achievement. Second, if you're looking for definitive arguments for or against your design, you're unlikely to find them. Design is a subjective endeavor, so you can always make a counter-argument to pretty much any argument (as you just did). Third, I find it tenuous to agree that you "implemented only the necessary stuff" and that you "have a reason for every decision". If you cast a candid eye over your design, you'll notice that it has virtually nothing to do with logging aside from using the "log" word here and there. It's barely a design at all as it's just a barebones streaming text out glue that assembles an external string template engine with an ad-hoc streaming interface containing only the Write primitive. So unless you set out to implement nothing but simple streaming, you didn't implement only the necessary stuff; and if you have a reason for every decision, then by necessity one motivating reason was to stay away from anything specific to logging. Andrei
May 29 2011
parent reply David Nadlinger <see klickverbot.at> writes:
On 5/30/11 12:13 PM, Vincent wrote:
 I can say more: looking at program trace I more interested not in 'give
 me more details', but 'give me details on this object'. In your case
 it's IO object - turning ON verbosity, you do it for ALL logging, while
 you need just an IO module. What you say on this? :)
 In this case I prefer snippets like this:

 // some IO logic
 version(log_io) logDbg(`SENT: ` ~ line);

First, please note that Jose turned verbose logging on just for IO as well, you might want to have a closer look at his post. Second, your proposal doesn't allow enabling/disabling verbose logging without recompiling your application, which is clearly not desirable in many contexts where a logging library would be used. David
May 30 2011
parent reply David Nadlinger <see klickverbot.at> writes:
On 5/30/11 3:48 PM, Vincent wrote:
 On Mon, 30 May 2011 14:24:10 +0200, David Nadlinger <see klickverbot.at>
 wrote:

 First, please note that Jose turned verbose logging on just for IO as
 well, you might want to have a closer look at his post.

Oh, really? This is cite from his library: vlog(0)("Verbosity 0 message"); vlog(1)("Verbosity 1 message"); Can you point me where is any logical separation between modules?

In his post you originally replied to, he wrote: »So QA reruns the app with the command line option --vmodule=io=3« – note the »io« part there. In terms of code, e.g. here: https://github.com/jsancio/phobos/blob/master/std/log.d#L864
 Second, your proposal doesn't allow enabling/disabling verbose logging...

My realisation have no any 'verbosity' at all due to helpless. But yes, to completely remove logging (from binaries) you have to recompile the program. Or you can do it at runtime, just 1 line of code. And look please at second version: http://pastebin.com/fd3UCgjv - it doesn't force you to use version(log) anymore.

I was referring to the post of you I actually cited, where you wrote: --- In this case I prefer snippets like this: // some IO logic version(log_io) logDbg(`SENT: ` ~ line); --- Clearly, you can't activate/deactivate logging at runtime here, but you proposed that directly in answer to a post where the merits of being able to enable verbose logging in an existing application were discussed. No offense intended, but could you please try to respect common newsgroup/mailing list etiquette and actually read the posts you are replying to? David
May 30 2011
parent David Nadlinger <see klickverbot.at> writes:
On 5/30/11 4:50 PM, Vincent wrote:
 I was referring to the post of you I actually cited, where you wrote:

 ---
 In this case I prefer snippets like this:

 // some IO logic
 version(log_io) logDbg(`SENT: ` ~ line);
 ---

 Clearly, you can't activate/deactivate logging at runtime here

You can, see Log.IsEnabled member.

You wrote in response to Jose: »In your case it's IO object - turning ON verbosity, you do it for ALL logging, while you need just an IO module. What you say on this? :) In this case I prefer snippets like this: […]« I don't quite see how you would be able to enable/disable logging specifically for the I/O part _at runtime_ in your snippet (the one I quoted above), assuming that logDbg is a global logging object not specific to the I/O code. David
May 30 2011
prev sibling next sibling parent reply Vincent <thornik gmail.com> writes:
Small work on issues: http://pastebin.com/fd3UCgjv

As you see, there is no more 'version' spaghetti:

auto logDbg = new Log(new FileLogEngine(`c:\res.txt`), new  
DBLogEngine(``));
auto logErr = new Log();

logDbg(`It works for %s`, `me!`);// won't be printed if '-version=log' is  
not specified
logErr(`Serious problem!`);
May 29 2011
parent Matthew Ong <ongbp yahoo.com> writes:
On 5/30/2011 6:35 AM, Vincent wrote:
 Small work on issues: http://pastebin.com/fd3UCgjv

 Great! Didn't catch it from docs, sorry. Anyway, it's not 'verbosity' as
 meaning in english - it's FILTERING and yep, this is useful. But all
 those 'verbose level 1, 2, 3....' that's ridiculous.

debugging runtime state of your network application and not concern about other issue such as security and hacking attacks. Day to day network admin person will have to analyze log generated by a server application that he/she is tasked to maintained http://www.techrepublic.com/blog/opensource/how-do-i-customize-apache-server-log-files-to-track-my-web-site/71 I am using apache server as an example, as it is one well know and widely used server. Others may have similar to such data captured. Common Log Format (CLF). However, there is no business logic trace information for developer to do debugging. Verbosity and Log level is very important to Network Admin Person http://www.securityweek.com/content/how-defend-against-ddos-attacks Please see: 3. Dump the logs If you find log files growing large quite quickly, you're faced with the choice between keeping the data and losing the server, or losing the data and keeping the server. BTW, if log library designed properly, the server can turn on-off different level of the logging activity without having to restart the entire state of the application. The Log level allow Admin person to do log filtering and analyze the damaged done if there is abnormal activity or hacking done on the application. If there is ONLY single level logging, the log file/files will grow and grow until the resource is eaten up by disk IO and CPU context switch, and also network is fully chocked because of the DDOS.
There is not so much 'log specific' stuff, some of 'em is just a 

-----
2. Nobody prevent you from doing it! Just make 'mixin' with __FILE__ 

3. In my practice I didn't see so low-level approach for debugging - 

line in editor?? Funny, heh... ------ I entirely agrees with you here. As web application developer, SMTP mail client, or protocol X consumer application, they are more concerned about what is the state of the application if the application crashed or compromised by hacker. How to roll back the last action done or track damages done? __FILE__ + __LINE__ is too little information. That is why I posted another new thread: How about adding NEW Special Tokens?? For ease and Security. *new compile time information's tokens* and *runtime special tokens* Could those token have help you? I believe it does because it is at least listed by apache web server group as 'common' item. --------------------------------------------------------------------------------- Different topic, implementing the logging could be done also...
1. Minimal logging must be as simple as one function call with one 

2. Log must support multiple destinations of different kind.

Different company has different network admin policy for different application.
3. Destinations must be easy expandable - one simple class for any kind.

4. Enable/Disable log at run/compile time. IDEALLY doesn't consume CPU 

Totally agrees. No extra comment needed.
5. Suppport for different output formats.

person and NOT only developer. :) :) :) Do help to comment about this also: How about adding NEW Special Tokens?? For ease and Security. I may have loose something out. -- Matthew Ong email: ongbp yahoo.com
Jun 04 2011
prev sibling next sibling parent Jose Armando Garcia <jsancio gmail.com> writes:
Why is verbosity not useful? Or I should ask, why is verbosity as
implemented in std.log and glog not useful? Let me motivate it with a
possible use case.

Lets say that you have a "large" non-deterministic application (in
this case it means that you accept external inputs in the form of IO)
and in the QA or a really generous customer your application keeps
hitting a bug in your program. You try really hard but you are unable
to reproduce it in your environment.

You just remember that the developer that wrote the IO layer was nice
enough to log all IO inputs at verbosity level 3. He didn't log them
at info severity because of performance implications. So you tell QA
or this generous customer to turn on verbose logging for the IO
module. Also, note that the bug is not necessarily in the IO module
but information in the IO module is useful to debug the buggy module.
So QA reruns the app with the command line option --vmodule=io=3 or if
you have a really awesome application you can change this module
filter configuration while still running (you can't do this right now
with std.log but I plan to implement this). Now you are able to see
all the non-deterministic input that came to your application.

You take this log and quickly write a load generator (in D of course)
with a similar timing (time is part of the information in std.log's
log) as that in the QA log and boom you hit the bug and are able to
debug it in your favorite debugger.

-Jose

2011/5/29 Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:
 On 05/29/2011 04:52 PM, Vincent wrote:
 Hello, Andrei! Very appreciate you spent time on my work.
 I agree with some your critique, sure it's just a question of
 improvements. My main question is acceptance of my log library as a
 whole conception, since it's simple and 10 from 10 programmers can
 understand its sources (I keep it as very serious goal).

Why? 10 out of 10 would be more interested in using the library effectively instead of reading its source. Clearly keeping the source readable is desirable, but not an overriding goal. It's probably the last thing you care about after having taken care of everything.
 Let's discuss doubtful points...

 Verbosity is not very often used indeed but I'd guess many people got
 used to it because it's present in most libraries.

Habit is not the rule. You must remember those days when every DOS editor show time in menu bar. WHY? Just because everybody did it. I see nothing bad to loose this 'verbosity' as archaic feature. As I said before, 'special demands' is the pain of 'special programmers' - if they REALLY cannot live w/o 'verbosity', they can add it in 3 LOC.

You mentioned you don't care about verbosity levels. I added that I haven't found a lot of use for them either. Next thing you know, you're presupposing your own assumption. Isn't the data set a tad small to draw conclusions from?
 * The library doesn't take care of capturing the origin of the log
 (file/module/line) so the user must pass them manually.

I've read posts related __LINE__s in log and don't see any reason in this feature. Why? 1. It's not direct task of 'logging'. Log doesn't care about your lines/files just because you're too negligent to control what and where you log - its task is just assemble together runtime info. 2. Nobody prevent you from doing it! Just make 'mixin' with __FILE__ and use. 3. In my practice I didn't see so low-level approach for debugging - well organised log shows enough, why people care about number of code line in editor?? Funny, heh... 4. Logging is not only "We have a problem in line 10!" - it's "trace of life" for our program.

Without automated headers including location of the log line and date/time, one can't even call that a logging library. It's an text streaming facility.
 * The library requires version(logging) to control enabling.
 Essentially the client would need to prefix _all_ logging lines

Agree completely. But something says to me that it's just a my little knowledge of D (and it's true). Sure there is some feature to make log calls 'invisible' without 'version' prefix - I just used feature I sure about. May be mixins will help? This is the place where I need help of community.

The help can be readily absorbed from reading the existing proposal.
 * The library always evaluates arguments, even when logging is
 dynamically disabled.

First, let's remember Knuth's "Premature Optimization" warning - say "it's slow" only when it's really slow down your program.

Evaluating arguments always vs. not at all is a matter of principles.
 Second, remember use case of this "runtime switch": "ALL logging is
 enabled (read "everywhere it's slow"), but at this place it's disabled
 (and who cares how slow it is?)".

It is very important that the program is as fast as it can when logging is disabled. I don't understand how one could argue sensibly otherwise.
 Third, if you use log, you NEED this info, despite how fast it was
 calculated. And even if you jump over with 'lazy' arguments, they HAVE
 to be calculated - you wait 'em anyway! So... what a benefit you have
 from laziness?

With lazy evaluation you can arrange that complex expressions inside the log calls are not evaluated.
 And hell, if you're good programmer, you'll never write critical section
 like
 log("a");
 log("b");
 ...rather you accumulate log info and output it at once.
 I agree with usefullness of 'lazy' stuff, but not in this case.

Sometimes you do need to insert log expressions in relatively critical places for debugging purposes.
 * The library does not define a notion of a severity hierarchy

If you look at use cases of this 'severity', there is no 'hierarchy' (look at usage of my library) - it's again "habits" of 70-th. Just obvious example from real life: I log all 'info' messages about client actions. But somewhere I get database exception - my fault, logged as 'critical'. Together with two levels above I have 'warning' level about problems like 'this person has no surname'. Question is: "Why I must dig in 'warning' trash, when I just need info what client did and which critical problem happen?". As you see, 'hierarchy' is just "flat" view on problem - there is independed severities and _combination_ of 'em. Again, no apocalypse happen - who need this feature, that must spent time on it.

Your example describes a trivial matter. Often times, the trace leading to a critical problem is indicative of the issue in more complicated applications.
 Arguably defining a hierarchy could be considered outside the scope of
 a "core" logging library, but that means more work for the casual user
 of the library to achieve a common task.

Exactly. You cannot include in simple 'string assembler' all functionality (esp. legacy point of view - time is changed!). Simple requirements must lead to simple usage - if you don't use 'verbosity', why you should pass parameters in every log call?

Who does? Did you skim the existing proposal at all?
 I believe something close to Jose's library (after community feedback)
 will become the standard library for logging, so we should at best we
 consolidate efforts behind it.

What if I don't believe? :) We'll spent time on functions, used by 1% of 'old shoes' and give a bad example for next generations. I don't pretend I wrote the best library ever, but I kept on rule 'implement only necessary stuff' and I have a reason for every decision. Well, I don't want to press anybody, but before you completely give up my implementation I wanna be smashed with serious arguments how bad my design is. :) (may be it'll be good lesson for another designs!) Good luck for all community!

Same here, and welcome. First, this seems to be the third time you are referring negatively about age. That is never in good taste, and can at best be tolerated if it comes from a position of exceptional achievement. Second, if you're looking for definitive arguments for or against your design, you're unlikely to find them. Design is a subjective endeavor, so you can always make a counter-argument to pretty much any argument (as you just did). Third, I find it tenuous to agree that you "implemented only the necessary stuff" and that you "have a reason for every decision". If you cast a candid eye over your design, you'll notice that it has virtually nothing to do with logging aside from using the "log" word here and there. It's barely a design at all as it's just a barebones streaming text out glue that assembles an external string template engine with an ad-hoc streaming interface containing only the Write primitive. So unless you set out to implement nothing but simple streaming, you didn't implement only the necessary stuff; and if you have a reason for every decision, then by necessity one motivating reason was to stay away from anything specific to logging. Andrei

May 29 2011
prev sibling next sibling parent Vincent <thornik gmail.com> writes:
On Mon, 30 May 2011 02:58:00 +0200, Jose Armando Garcia  
<jsancio gmail.com> wrote:

 Why is verbosity not useful?

Probably, because it's too expensive? :)
 You just remember that the developer that wrote the IO layer was nice
 enough to log all IO inputs at verbosity level 3.

OK, but saying this you assume we also have level 1 and 2! Can you give an example how your code will look with ALL these levels implemented? I fear on 3 LOC you'll have 10 lines just to provide 'verbosity' logic. This is why it's expensive. Problem is not only in digits - HOW you'll decide how many levels to have and what to show on every level? Don't you scared for your head if it's 1M LOC app? :) I can say more: looking at program trace I more interested not in 'give me more details', but 'give me details on this object'. In your case it's IO object - turning ON verbosity, you do it for ALL logging, while you need just an IO module. What you say on this? :) In this case I prefer snippets like this: // some IO logic version(log_io) logDbg(`SENT: ` ~ line); It allows to see only what you need and I don't give a damn if it's longer for two words. Jose, just take it right: long term log libraries are OK _for_people_who_use_it_. But I see no any reason to copy again and again old-time solutions - features are not only accumulated, but outdated too (this is why you are with D, not C :) ). So I want 'clean up' old stuff and implement just really necessary capabilities (Which Andrei named as "simple streaming" :) ). Before you(we) implement something, don't think "Oh, we have verbosity and we can use it like this!" - start from GOAL: "We have this task, how it can be implemented?" - sure, from this point you never even think about 'verbosities' (depends from mind, OK). Well, since people are conservative, I don't expect too much attention, but anyway thanks you read my ideas - hope it help.
May 30 2011
prev sibling next sibling parent Vincent <thornik gmail.com> writes:
On Mon, 30 May 2011 01:13:13 +0200, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 Without automated headers including location of the log line and  
 date/time, one can't even call that a logging library.

I repeat: if people so careless that they don't know what and from where written in their log, lines and files won't help. EVERY log place must provide high level info, which must be enough for conclusion. If in 5 places you do log(`SEND: ` ~ line) of course your log is dump of useless lines! Again: nothing prevent you from giving file and line, just include 'em in message! And log purpose is not only 'now I run line 20 and i = 3' - it can be log of user actions for manager's report - who cares which files/lines it involved?
 It's an text streaming facility.

It is! Like any GUI library is "just a graphics rectangles to click on". :) There is not so much 'log specific' stuff, some of 'em is just a 'tradition'. Log is TEXT, which help you to debug code.
 Evaluating arguments always vs. not at all is a matter of principles.

Principles are nice in MacDonalds, programming is a bit more complex stuff, where every principle must be reasonable to apply.
 * The library does not define a notion of a severity hierarchy

If you look at use cases of this 'severity', there is no 'hierarchy'

Your example describes a trivial matter. Often times, the trace leading to a critical problem is indicative of the issue in more complicated applications.

Sorry, nothing understand what you disagree with. First, I don't decline 'severity' since we can have logWarnings, logErrors, etc. Second, I just show that 'linear severity levels' helps not so much when you need selective messages - you cannot skip intermediate levels when you don't need 'em.
 First, this seems to be the third time you are referring negatively  
 about age.

Surprisingly wrong view. Not age, but OLD LIBRARIES written in 70-th, when people even didn't mention 'usability', 'maintainability' and other stuff. Nothing unusual in the fact, that programming practice made outdated some conceptions. Look at LISP - lispers sh*t with bricks proving how LISP good is. WHERE is this LISP? In the dustbin of history. What you found offensive here? Like you name spade as spade, I name old as old. *amazement*
 Design is a subjective endeavor, so you can always make a  
 counter-argument to pretty much any argument (as you just did).

Disagree. There is good conceptions, which cannot be satisfied with ugly design: extensibility, scalability, maintenance, robustness, usability ... Bad design quickly drops under good critique. I'm happy you found weak places in my draft, but some critique rely just on your imagination how logging must be used - I cannot see reason to change library by every subjective demands.
 So unless you set out to implement nothing but simple streaming....

Again, it's because of your vision "what logging is". As many messages I read, same many opinions I met - from ridiculous (to me) __FILES__ and __LINES__ to "all logging is a file output routines". Before people touch any library they must create a list of requirements, discuss it (instead of pointing "oh, in library blah-blah it made like this - let's do the same!"), test in practice and only after this they may pretend to be in std.* namespace. I didn't see any discussions except already done library with automatically generated docs. Nice development, heh! :) (kill me if I'll try to buy such software) To be more productive I can formulate my _core_ list, which can be extended with "bells and whistles", but everybody must understand where is LOG and where is I LIKE IT. 1. Minimal logging must be as simple as one function call with one string argument. 2. Log must support multiple destinations of different kind. 3. Destinations must be easy expandable - one simple class for any kind. 4. Enable/Disable log at run/compile time. IDEALLY doesn't consume CPU if completely disabled. 5. Suppport for different output formats. That's it! Even handy string formatting is not a requirement - just because it's a 'sugar', not 'logging' itself. In my practice I never used anything more complex due to useless of any other functionality - good log provides enough. WBR...
May 30 2011
prev sibling next sibling parent Vincent <thornik gmail.com> writes:
On Mon, 30 May 2011 14:24:10 +0200, David Nadlinger <see klickverbot.at>  
wrote:

 First, please note that Jose turned verbose logging on just for IO as  
 well, you might want to have a closer look at his post.

Oh, really? This is cite from his library: vlog(0)("Verbosity 0 message"); vlog(1)("Verbosity 1 message"); Can you point me where is any logical separation between modules?
 Second, your proposal doesn't allow enabling/disabling verbose logging...

My realisation have no any 'verbosity' at all due to helpless. But yes, to completely remove logging (from binaries) you have to recompile the program. Or you can do it at runtime, just 1 line of code. And look please at second version: http://pastebin.com/fd3UCgjv - it doesn't force you to use version(log) anymore.
May 30 2011
prev sibling next sibling parent Vincent <thornik gmail.com> writes:
On Mon, 30 May 2011 16:01:21 +0200, David Nadlinger <see klickverbot.at>  
wrote:

 In terms of code, e.g. here:

 https://github.com/jsancio/phobos/blob/master/std/log.d#L864

Great! Didn't catch it from docs, sorry. Anyway, it's not 'verbosity' as meaning in english - it's FILTERING and yep, this is useful. But all those 'verbose level 1, 2, 3....' that's ridiculous.
 I was referring to the post of you I actually cited, where you wrote:

 ---
 In this case I prefer snippets like this:

 // some IO logic
 version(log_io) logDbg(`SENT: ` ~ line);
 ---

 Clearly, you can't activate/deactivate logging at runtime here

You can, see Log.IsEnabled member.
 No offense intended, but could you please try to respect common  
 newsgroup/mailing list etiquette and actually read the posts you are  
 replying to?

Strange statement... are you stupid or you make me stupid? How I physically can answer on message if I didn't read it?? I read it, try to understand and reply. What's wrong you found here?
May 30 2011
prev sibling next sibling parent Jose Armando Garcia <jsancio gmail.com> writes:
2011/5/30 Vincent <thornik gmail.com>:
 On Mon, 30 May 2011 02:58:00 +0200, Jose Armando Garcia <jsancio gmail.com>
 wrote:

 Why is verbosity not useful?

Probably, because it's too expensive? :)
 You just remember that the developer that wrote the IO layer was nice
 enough to log all IO inputs at verbosity level 3.

OK, but saying this you assume we also have level 1 and 2! Can you give an example how your code will look with ALL these levels implemented? I fear on 3 LOC you'll have 10 lines just to provide 'verbosity' logic. This is why it's expensive. Problem is not only in digits - HOW you'll decide how many levels to have and what to show on every level? Don't you scared for your head if it's 1M LOC app? :) I can say more: looking at program trace I more interested not in 'give me more details', but 'give me details on this object'. In your case it's IO object - turning ON verbosity, you do it for ALL logging, while you need just an IO module. What you say on this? :) In this case I prefer snippets like this: // some IO logic version(log_io) logDbg(`SENT: ` ~ line); It allows to see only what you need and I don't give a damn if it's longer for two words. Jose, just take it right: long term log libraries are OK _for_people_who_use_it_. But I see no any reason to copy again and again old-time solutions - features are not only accumulated, but outdated too (this is why you are with D, not C :) ). So I want 'clean up' old stuff and implement just really necessary capabilities (Which Andrei named as "simple streaming" :) ). Before you(we) implement something, don't think "Oh, we have verbosity and we can use it like this!" - start from GOAL: "We have this task, how it can be implemented?" - sure, from this point you never even think about 'verbosities' (depends from mind, OK). Well, since people are conservative, I don't expect too much attention, but anyway thanks you read my ideas - hope it help.

Your solution (using version(log_io)) doesn't solve the problem/use case I presented. I recommend you study the domain before suggesting solutions.
May 30 2011
prev sibling parent Jose Armando Garcia <jsancio gmail.com> writes:
On Mon, May 30, 2011 at 11:50 AM, Vincent <thornik gmail.com> wrote:
 On Mon, 30 May 2011 16:01:21 +0200, David Nadlinger <see klickverbot.at>
 wrote:

 In terms of code, e.g. here:

 https://github.com/jsancio/phobos/blob/master/std/log.d#L864

Great! Didn't catch it from docs, sorry. Anyway, it's not 'verbosity' as meaning in english - it's FILTERING and yep, this is useful. But all those 'verbose level 1, 2, 3....' that's ridiculous.

Actually the sample space for verbose logging is [short.min, short.max] (using set notation). You don't have to use them if you don't want to. You can always use verbose level 1 and do filtering only based on the module. You can even create your own logical module that doesn't map to source files as follow. --- string specialModule; vlog(1, specialModule)(...) --- I should state though that, this is not recommended. Use at your own risk! ;)
 I was referring to the post of you I actually cited, where you wrote:

 ---
 In this case I prefer snippets like this:

 // some IO logic
 version(log_io) logDbg(`SENT: ` ~ line);
 ---

 Clearly, you can't activate/deactivate logging at runtime here

You can, see Log.IsEnabled member.
 No offense intended, but could you please try to respect common
 newsgroup/mailing list etiquette and actually read the posts you are
 replying to?

Strange statement... are you stupid or you make me stupid? How I physically can answer on message if I didn't read it?? I read it, try to understand and reply. What's wrong you found here?

May 30 2011
prev sibling next sibling parent Vincent <thornik gmail.com> writes:
On Sat, 28 May 2011 18:17:53 +0200, Jimmy Cao <jcao219 gmail.com> wrote:

 Can you add an HTTPLogEngine to send data using POST/GET?

Not a problem, my source is just an academic example of overloading :) Anybody can extend it with any logging destination. Problem is community - how fast we will agree with minimal and optimal set of features.
 Perhaps EmailLogEngine should be renamed to SMTPLogEngine.

And next year we will add IMAPLogEngine? :) Nope, I prefer just 'Email' - more generic approach.
May 29 2011
prev sibling next sibling parent Vincent <thornik gmail.com> writes:
On Sun, 29 May 2011 22:05:23 +0200, David Nadlinger <see klickverbot.at>  
wrote:

 Uh, IMAP has nothing to do with sending mail

For bores: Just an example that 'mail' doesn't mean 'SMTP'. Naming functions based on low level details leads to unavoidable refactoring.
May 29 2011
prev sibling next sibling parent Vincent <thornik gmail.com> writes:
On Mon, 30 May 2011 01:32:44 +0200, Daniel Gibson <metalcaedes gmail.com>  
wrote:

 So if it's just "EmailLogEngine" or whatever and it's using SMTP,
 changing it to use the SMTP-alternative....

It's just a wrong view on a whole conception. Log uses different KIND of targets: file, database(do you care which one??), mail (is it important which proto??), etc. 'Mail' is an abstraction of "something which can send message to the human". So... instead of making annoying set of classes for all known mail protocols, we use ONE mail class with appropriate constructors for all imaginable protocols. Say: enum MailProto { SMTP, IMAP } class EmailLogEngine : LogEngine { public this(MailProto type, string user, string psw, string host, int port = 0) { if (port == 0) switch(type) { case SMTP: port = 25; break; } // initialize mail server structures } override public void Write(string msg) { // send message, using appropriate proto } }
May 30 2011
prev sibling next sibling parent Vincent <thornik gmail.com> writes:
On Mon, 30 May 2011 11:22:39 +0200, Daniel Gibson <metalcaedes gmail.com>  
wrote:

 How is
 auto logCrash = new Log(new EmailLogEngine(MailProto.SMTP,
 "smtp.myserver.com"));
 better than
 auto logCrash = new Log(new SMPTLogEngine("smtp.myserver.com")); ?

Well, my answer was too fast. Look at this: enum MailProto { SMTP, IMAP, MSEXCH } class EmailLogEngine : LogEngine { public this(string host, string user, string psw, MailProto type = MailProto.SMTP, int port = 0)// most common defaults { if (port == 0) switch(type) { case SMTP: port = 25; break; } // initialize mail server structures } } In this case auto logCrash = new Log(new EmailLogEngine("smtp.myserver.com", `user`, `psw`)); is enough for most usages. Even better: auto logCrash = new Log(new EmailLogEngine(Settings.MailHost, Settings.MailUser, Settings.MailUserPsw, Settings.MailType)); This line even doesn't require recompilation, using params from settings. What you'll do with your SMTPLogEngine if tomorrow we switch to MS Exchange? Jumping around with find/replace? And you don't catch the main point: lower you go in implementation - harder you use it again, so 'be generic' is a good advice here.
May 30 2011
prev sibling parent Graham Fawcett <fawcett uwindsor.ca> writes:
On Sun, 29 May 2011 22:05:23 +0200, David Nadlinger wrote:

 On 5/29/11 9:50 PM, Vincent wrote:
 Perhaps EmailLogEngine should be renamed to SMTPLogEngine.

And next year we will add IMAPLogEngine? :) Nope, I prefer just 'Email' - more generic approach.

Uh, IMAP has nothing to do with sending mail, it's just a protocol for accessing your server-side mailbox, like POP3.

Actually, an IMAP-based logging sink would be conceivable. Unlike POP3, you can actively store documents into an IMAP mailbox. As an example, "imapfs" is a FUSE-based filesystem which stores files to an IMAP host. Deciding whether this is a good idea is left as an exercise. :) Graham
May 30 2011