www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Resolution of core.time.Duration...

reply Alexander <aldem+dmars nk7.net> writes:
...why it is in hnsec? I know that this resolution is used in Win32 API (file
time), but since TickDuration may be 1 ns resolution, wouldn't it be better to
make Duration to be stored with maximum (defined so far) resolution?

Especially because Duration may not hold long intervals (> months) - so there
is no problem with overflow.

Thread.sleep() accepts Duration (or hnsec) as an argument, while system
resolution is higher, and on some systems it is even possible that it can sleep
less than 100ns.

SysTime is also kept in hnsecs, while resolution of system time (on Linux at
least) is 1ns. Sure, in case of SysTime it is all bound to overflow, but it
depends how value is stored - if we split seconds and nanoseconds, it will be
fine.

Additionally, when accepting long values as an argument for duration it is more
logically to use SI units :)

/Alexander
May 17 2011
next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On 2011-05-17 03:15, Alexander wrote:
 ...why it is in hnsec? I know that this resolution is used in Win32 API
 (file time), but since TickDuration may be 1 ns resolution, wouldn't it be
 better to make Duration to be stored with maximum (defined so far)
 resolution?
 
 Especially because Duration may not hold long intervals (> months) - so
 there is no problem with overflow.
 
 Thread.sleep() accepts Duration (or hnsec) as an argument, while system
 resolution is higher, and on some systems it is even possible that it can
 sleep less than 100ns.
 
 SysTime is also kept in hnsecs, while resolution of system time (on Linux
 at least) is 1ns. Sure, in case of SysTime it is all bound to overflow,
 but it depends how value is stored - if we split seconds and nanoseconds,
 it will be fine.
 
 Additionally, when accepting long values as an argument for duration it is
 more logically to use SI units :)
 
 /Alexander

hnsecs is the highest resolution that you can hold in a 64-bit integer and still have it cover a reasonable range. And remember that a Duration is what's used as the result of subtracting one SysTime from another, so making it more precise would be bad. As it is, it can't actually hold the difference of SysTime.max - SysTime.min. Because it's signed, it can only hold half that range. Also, the system clock definitely does not reach hnsecs on any system that I've seen. It generally maxes out at microseconds or a bit better than that. So, in most cases, using a precision greater than hnsecs doesn't gain you anything. I'd love to have greater precision, but it would require something larger than a 64-bit integer, which we don't have. Or it would require splitting up the seconds and sub-seconds which makes the types take up more space and makes the math that much worse. During the development and review process, it was decided that hnsecs was ideal. Sure, higher precision might be nice, but we don't have anything bigger than a 64-bit integer, and odds are that you couldn't really take advantage of the higher precision anyway. - Jonathan M Davis
May 17 2011
parent reply Alexander <aldem+dmars nk7.net> writes:
On 17.05.2011 12:34, Jonathan M Davis wrote:

 Also, the system clock definitely does not reach hnsecs on any system that
I've seen.

Duration (unlike TickDuration) is not tied to system clock only, AFAIK (at least, this is not mentioned in the documentation). Using rdtsc it is possible to obtain even more precision than 1ns. Also, Duration can be used to store intervals which are not really related to system clock at all.
 Or it would require splitting up the seconds and sub-seconds which makes the
types take up 
 more space and makes the math that much worse.

OK, though I don't think that 4 or 8 bytes more will really make any noticeable change (many Phobos/druntime structures are over-sized anyway, IMHO) - time values are not used in big arrays, usually. As to math - for most operations involving calendar computations only seconds are relevant, and math itself takes literally nanoseconds to execute - so only functions which actually require better precision may use more complex math. What could be done, though - is to hide the internal format (hnsecs), i.e. do not expose it to functions like Thread.sleep() with ulong argument. /Alexander
May 17 2011
parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
On 17.05.2011 15:44, Alexander wrote:
 On 17.05.2011 12:34, Jonathan M Davis wrote:

 Also, the system clock definitely does not reach hnsecs on any system that
I've seen.

Using rdtsc it is possible to obtain even more precision than 1ns.

and has it's fare share of caveats: http://msdn.microsoft.com/en-us/library/ee417693.aspx or more practical: http://www.strchr.com/performance_measurements_with_rdtsc (also there are processors with dynamic frequency...) And like you said there are no APIs that use such precision.
    Also, Duration can be used to store intervals which are not really related
to system clock at all.

 Or it would require splitting up the seconds and sub-seconds which makes the
types take up
 more space and makes the math that much worse.

As to math - for most operations involving calendar computations only seconds are relevant, and math itself takes literally nanoseconds to execute - so only functions which actually require better precision may use more complex math.

we are talking about small scale CPU performance measures.
    What could be done, though - is to hide the internal format (hnsecs), i.e.
do not expose it to functions like Thread.sleep() with ulong argument.

 /Alexander

-- Dmitry Olshansky
May 17 2011
prev sibling next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 17 May 2011 06:15:54 -0400, Alexander <aldem+dmars nk7.net> wrote:

 ...why it is in hnsec? I know that this resolution is used in Win32 API  
 (file time), but since TickDuration may be 1 ns resolution, wouldn't it  
 be better to make Duration to be stored with maximum (defined so far)  
 resolution?

if you use hnsecs, then you get a range of SysTime of -30k to 30k years. That might seem overkill, but consider that even going to dnsecs (10 nanoseconds) reduces your range to -3k to +3k years. The problem is that nobody is likely to care about that extra 10 intervals, but losing out on 27,000 years x 2 is pretty significant. It seems like a no-brainer to me. straight nanoseconds are not possible, because we couldn't even represent our current date with it. Plus, we have good precedence, both Microsoft and Tango use that tick duration. It's a natural conclusion.
 Especially because Duration may not hold long intervals (> months) - so  
 there is no problem with overflow.

A Duration is the result of subtracting two SysTime's, which uses hnsecs as its tick, so yeah, there is a problem with overflow if you use a finer resolution.
 Thread.sleep() accepts Duration (or hnsec) as an argument, while system  
 resolution is higher, and on some systems it is even possible that it  
 can sleep less than 100ns.

The minimum sleep time for a thread is one clock period. If your OS is context switching more than once per 100ns, your OS is going to be doing nothing but context switching. Processors just aren't fast enough to deal with that (and likely won't ever be). 100 ns is a reasonable resolution for that. Real time applications may require more precise timing, but you would likely need a separate API for that.
 SysTime is also kept in hnsecs, while resolution of system time (on  
 Linux at least) is 1ns. Sure, in case of SysTime it is all bound to  
 overflow, but it depends how value is stored - if we split seconds and  
 nanoseconds, it will be fine.

Again, the resolution of the *structure* may be nsecs, but the actual intervals you have access to is about every 4ms on linux ( see http://en.wikipedia.org/wiki/Jiffy_(time) ). If it makes you feel better to use higher resolution timing, the facilities are there, just use the C system calls.
 Additionally, when accepting long values as an argument for duration it  
 is more logically to use SI units :)

I agree that accepting a long as an alternative to Duration, it makes sense to use a more normal tick resolution. The chances of someone wanting to have their process sleep for more than 300 years (e.g. for nanosecond resolution) is pretty small. This might be a worthwhile change. I'm not sure how much code this might affect, though. It would be plenty disturbing if your code started sleeping for 100ms instead of the 10s you thought you requested. What might be a good path is to disable those functions that accept a long for a few releases, then re-instate them with a new meaning. -Steve
May 17 2011
next sibling parent reply Daniel Gibson <metalcaedes gmail.com> writes:
Am 17.05.2011 15:25, schrieb Steven Schveighoffer:
 
 I agree that accepting a long as an alternative to Duration, it makes
 sense to use a more normal tick resolution.  The chances of someone
 wanting to have their process sleep for more than 300 years (e.g. for
 nanosecond resolution) is pretty small.  This might be a worthwhile change.
 
 I'm not sure how much code this might affect, though.  It would be
 plenty disturbing if your code started sleeping for 100ms instead of the
 10s you thought you requested.  What might be a good path is to disable
 those functions that accept a long for a few releases, then re-instate
 them with a new meaning.
 

Or just add nanoSleep() or something like that. Cheers, - Daniel
May 17 2011
next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
 On Tue, 17 May 2011 09:42:27 -0400, Daniel Gibson <metalcaedes gmail.com>
 
 wrote:
 Am 17.05.2011 15:25, schrieb Steven Schveighoffer:
 I agree that accepting a long as an alternative to Duration, it makes
 sense to use a more normal tick resolution. The chances of someone
 wanting to have their process sleep for more than 300 years (e.g. for
 nanosecond resolution) is pretty small. This might be a worthwhile
 change.
 
 I'm not sure how much code this might affect, though. It would be
 plenty disturbing if your code started sleeping for 100ms instead of the
 10s you thought you requested. What might be a good path is to disable
 those functions that accept a long for a few releases, then re-instate
 them with a new meaning.

Or just add nanoSleep() or something like that.

Probably a good idea, and deprecate Thread.sleep(long). It's more self-documenting.

I very much support the idea of deprecating all of the functions in druntime and phobos which take naked time values. And then if a function that sleeps at nanosecond resolution is considered to be of real value (which I question), then we can add that as something like nanoSleep. I believe that at present, most - if not all - of the function in druntime in Phobos which take a naked time value (aside from std.date) also take a Duration now, but I don't think that any of the old versions which take naked values are scheduled for deprecation. If there is a valid reason for wanting to keep any of them around, then we should consider it, but I'm definitely inclined to get rid of them in favor of the greater safety and maintainability of functions which take a Duration. - Jonathan M Davis
May 17 2011
prev sibling parent Sean Kelly <sean invisibleduck.org> writes:
On May 17, 2011, at 11:06 AM, Jonathan M Davis wrote:
=20
 I very much support the idea of deprecating all of the functions in =

 and phobos which take naked time values. And then if a function that =

 nanosecond resolution is considered to be of real value (which I =

 then we can add that as something like nanoSleep.

That was the plan. I just wanted a few releases with the new Duration = routines before deprecating the old ones. Now sounds like a good time.=
May 17 2011
prev sibling parent reply Alexander <aldem+dmars nk7.net> writes:
On 17.05.2011 15:25, Steven Schveighoffer wrote:

 if you use hnsecs, then you get a range of SysTime of -30k to 30k years.  That
might seem overkill, but consider that even going to dnsecs (10 nanoseconds)
reduces your range to -3k to +3k years.  The problem is that nobody is likely
to care about
 that extra 10 intervals, but losing out on 27,000 years x 2 is pretty
significant.  It seems like a no-brainer to me.

Well, if you put it this way, it seems reasonable.
 Plus, we have good precedence, both Microsoft and Tango use that tick
duration.  It's a natural conclusion.

Linux/Posix are using ns (clock_gettime(), nanosleep() etc - timespec) - I guess there is a reason for this.
 Again, the resolution of the *structure* may be nsecs, but the actual
intervals you have access to is about every 4ms on linux ( see
http://en.wikipedia.org/wiki/Jiffy_(time) ).

Not really. Take a difference from two consecutive get_clock() calls on Linux, and you will see that it is far below 1 µs (depends on CPU, though). Not all systems use timer interrupt for timekeeping.
 I agree that accepting a long as an alternative to Duration, it makes sense to
use a more normal tick resolution.  The chances of someone wanting to have
their process sleep for more than 300 years (e.g. for nanosecond resolution) is
pretty small. 
 This might be a worthwhile change.

Well, this is, actually, the whole reason of my post :) While 100ns resolution seems reasonable (after your explanations), accepting 100ns intervals as values directly seems not really good idea. /Alexander
May 17 2011
parent reply Alexander <aldem+dmars nk7.net> writes:
On 17.05.2011 16:45, Steven Schveighoffer wrote:

 The structures have used nanoseconds for over 10 years (I think gettimeofday
used it back in the 90s!), so the reason for using it was likely for future
compatibility (clearly nanosecond timing wasn't possible back then).  It looks
like the future is
 now, so it's good to have that resolution.

As to gettimeofday() - it is using timeval, which has 1µs resolution - still quite good for most applications.
 As for measuring time, yes, it would be good to use a higher precision timer. 
And in fact, std.datetime.StopWatch does just that.

Just in case - StopWatch is used in benchmarking functions while measuring wall-clock time, and this may produce incorrect results on busy systems when benchmarking CPU-intensive code. /Alexander
May 17 2011
parent reply Alexander <aldem+dmars nk7.net> writes:
On 17.05.2011 20:06, Jonathan M Davis wrote:

 StopWatch uses a mononotonic clock.

Monotonic clock is not CPU-usage-bound - it is in sync with wall-time, so problem on busy systems remains. /Alexander
May 17 2011
parent reply Alexander <aldem+dmars nk7.net> writes:
On 18.05.2011 01:18, Jonathan M Davis wrote:

 A monotonic clock is as good as you're going to get for accurate stopwatch 
 functionality. The system cannot possibly do any better than that. Context 
 switching can always get in the way. Increasing precision doesn't help that. 

Probably, you have misunderstood me - I wasn't talking about precision. I was talking about the usage of real-time clock for benchmarking, which may be incorrect when you use real-time clock for measuring performance of CPU-bound tasks. Say, you have to benchmark something that heavily is using CPU - and this takes 10s. But when the system is doing something else - the real-time clock may differ significantly to CPU-clock. So, in my example, 10s of CPU-intensive work may take 20s of real-time, if another CPU-bound task is running at the same time, thus, benchmark results will be incorrect. /Alexander
May 18 2011
parent Alexander <aldem+dmars nk7.net> writes:
On 18.05.2011 12:12, Jonathan M Davis wrote:

 To my knowledge, using the system's monotonic clock is the absolute best that
you're going to get for stuff like benchmarking.

It depends. If system is not busy, then it doesn't really matter - which clock to use, especially if you take average over several runs. If you benchmark I/O performance, then you need real-time clock and idle system - no way around. But if you benchmark CPU performance, the you have to use CPU clock - i.e. the clock which is increasing only when application is actually using CPU.
 Are you suggesting that there is an alternative which is better?

Sure, there is - clock_gettime with CLOCK_THREAD_CPUTIME_ID or CLOCK_PROCESS_CPUTIME_ID as clock id. This way, you will get exactly the amount of time that was spent while specific thread was using CPU.
 As far as I know, the issues of context  switching and whatnot are just a part
of life and that you can't do anything 
 about them except restrict what you're running on your computer when you run
benchmarks.

The way I've described it (CLOCK_THREAD_CPUTIME_ID or CLOCK_PROCESS_CPUTIME_ID), or using simple clock() function (less precise, though) you can do this. It only doesn't work well with measuring I/O performance, as CPU is mostly waiting, so CPU time is useless. I would propose extension of StopWatch - an option to specify which clock to use, CPU or real-time (monotonic is real-time). /Alexander
May 18 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 17 May 2011 09:55:49 -0400, Alexander <aldem+dmars nk7.net> wrote:

 On 17.05.2011 15:25, Steven Schveighoffer wrote:

 Plus, we have good precedence, both Microsoft and Tango use that tick  
 duration.  It's a natural conclusion.

Linux/Posix are using ns (clock_gettime(), nanosleep() etc - timespec) - I guess there is a reason for this.

I just read the man page on time (page 7) and indeed, there is support for sub-jiffy timing. And Duration will support that notion quite well. If you need sub-hnsec timing, you can call these functions directly. The structures have used nanoseconds for over 10 years (I think gettimeofday used it back in the 90s!), so the reason for using it was likely for future compatibility (clearly nanosecond timing wasn't possible back then). It looks like the future is now, so it's good to have that resolution. I still maintain that for things like sleeping, it is pointless to sleep less than 100ns since it likely takes longer than that to do a context switch, and your sleep time is only guaranteed to be *greater* than what you requested. The only time this is useful is for very specific real-time applications. I could be wrong, there have obviously been improvements to the timing mechanisms of Linux since I last looked at it. As for measuring time, yes, it would be good to use a higher precision timer. And in fact, std.datetime.StopWatch does just that. http://www.digitalmars.com/d/2.0/phobos/std_datetime.html#StopWatch The core.time.Duration type is for low-level timing facilities, such as waiting on a condition or sleeping a thread. It is also beneficial to re-use the structure in std.datetime for timing facilities that are generically useful for most cases. This allows one to fluently do date calculations, timing calculations and pass them directly to low level facilities. There will always be those cases where the time resolution is not enough, and for those cases, you will need to use a more specialized API. Consider that Java, C#, and many other languages get by just fine with coarser resolution timing. IMO dealing with times less than 1ms is pretty specialized.
 Again, the resolution of the *structure* may be nsecs, but the actual  
 intervals you have access to is about every 4ms on linux ( see  
 http://en.wikipedia.org/wiki/Jiffy_(time) ).

Not really. Take a difference from two consecutive get_clock() calls on Linux, and you will see that it is far below 1 µs (depends on CPU, though). Not all systems use timer interrupt for timekeeping.

I stand corrected, I was not aware of these timing facilities.
 I agree that accepting a long as an alternative to Duration, it makes  
 sense to use a more normal tick resolution.  The chances of someone  
 wanting to have their process sleep for more than 300 years (e.g. for  
 nanosecond resolution) is pretty small.
 This might be a worthwhile change.

Well, this is, actually, the whole reason of my post :) While 100ns resolution seems reasonable (after your explanations), accepting 100ns intervals as values directly seems not really good idea.

Yes, hnsecs is not a typical concept one has to deal with. -Steve
May 17 2011
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 17 May 2011 09:42:27 -0400, Daniel Gibson <metalcaedes gmail.com>  
wrote:

 Am 17.05.2011 15:25, schrieb Steven Schveighoffer:
 I agree that accepting a long as an alternative to Duration, it makes
 sense to use a more normal tick resolution.  The chances of someone
 wanting to have their process sleep for more than 300 years (e.g. for
 nanosecond resolution) is pretty small.  This might be a worthwhile  
 change.

 I'm not sure how much code this might affect, though.  It would be
 plenty disturbing if your code started sleeping for 100ms instead of the
 10s you thought you requested.  What might be a good path is to disable
 those functions that accept a long for a few releases, then re-instate
 them with a new meaning.

Or just add nanoSleep() or something like that.

Probably a good idea, and deprecate Thread.sleep(long). It's more self-documenting. -Steve
May 17 2011
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
 On 17.05.2011 16:45, Steven Schveighoffer wrote:
 The structures have used nanoseconds for over 10 years (I think
 gettimeofday used it back in the 90s!), so the reason for using it was
 likely for future compatibility (clearly nanosecond timing wasn't
 possible back then). It looks like the future is now, so it's good to
 have that resolution.

As to gettimeofday() - it is using timeval, which has 1µs resolution - still quite good for most applications.
 As for measuring time, yes, it would be good to use a higher precision
 timer. And in fact, std.datetime.StopWatch does just that.

Just in case - StopWatch is used in benchmarking functions while measuring wall-clock time, and this may produce incorrect results on busy systems when benchmarking CPU-intensive code.

StopWatch uses a mononotonic clock. - Jonathan M Davis
May 17 2011
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On 2011-05-17 12:34, Sean Kelly wrote:
 On May 17, 2011, at 11:06 AM, Jonathan M Davis wrote:
 I very much support the idea of deprecating all of the functions in
 druntime and phobos which take naked time values. And then if a function
 that sleeps at nanosecond resolution is considered to be of real value
 (which I question), then we can add that as something like nanoSleep.

That was the plan. I just wanted a few releases with the new Duration routines before deprecating the old ones. Now sounds like a good time.

As I understand it, deprecation is supposed to be a 3 stage process. 1. Mark the item to be deprecated as scheduled to be deprecated in the documentation, and if possible, give it a pragma which says it as well (though that generally involves turning the function in question into a template function if it isn't already so that the pragma will only kick in if the function is actually used). 2. Mark it with deprecated so that -d is required to use it. 3. Fully remove it from the code. None of those functions have actually had #1 happen to them yet, so for the most part, people have likely kept on using them as they have been. So, they should probably be marked as scheduled for deprecation for at least a release before they're actually deprecated. Unfortunately, we haven't actually decided on how long each phase of deprecation is supposed to be. It's been brought up a time or two, but no decision has ever been made. I keep intending to bring it up again, since the last couple of releases have seen several items enter phase 1, but I keep forgetting. - Jonathan M Davis
May 17 2011
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
 On 17.05.2011 20:06, Jonathan M Davis wrote:
 StopWatch uses a mononotonic clock.

Monotonic clock is not CPU-usage-bound - it is in sync with wall-time, so problem on busy systems remains.

A monotonic clock is as good as you're going to get for accurate stopwatch functionality. The system cannot possibly do any better than that. Context switching can always get in the way. Increasing precision doesn't help that. And since StopWatch uses TickDuration, it has the highest precision that the the sytem has anyway (whatever its clock tick is) rather than hnsecs. I don't understand what your concern is. StopWatch is using the highest precision, most accurate clock that it can, and there's no way to stop issues with context switching. What more do you expect it to do? You're talking like there's something wrong with std.datetime's benchmarking functionality, and as far as I know, it has the best that is possible as far as timing accuracy goes. - Jonathan M Davis
May 17 2011
prev sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On 2011-05-18 02:13, Alexander wrote:
 On 18.05.2011 01:18, Jonathan M Davis wrote:
 A monotonic clock is as good as you're going to get for accurate
 stopwatch functionality. The system cannot possibly do any better than
 that. Context switching can always get in the way. Increasing precision
 doesn't help that.

Probably, you have misunderstood me - I wasn't talking about precision. I was talking about the usage of real-time clock for benchmarking, which may be incorrect when you use real-time clock for measuring performance of CPU-bound tasks. Say, you have to benchmark something that heavily is using CPU - and this takes 10s. But when the system is doing something else - the real-time clock may differ significantly to CPU-clock. So, in my example, 10s of CPU-intensive work may take 20s of real-time, if another CPU-bound task is running at the same time, thus, benchmark results will be incorrect.

To my knowledge, using the system's monotonic clock is the absolute best that you're going to get for stuff like benchmarking. Are you suggesting that there is an alternative which is better? As far as I know, the issues of context switching and whatnot are just a part of life and that you can't do anything about them except restrict what you're running on your computer when you run benchmarks. - Jonathan M Davis
May 18 2011