D - coroutine?
- nobody yahoo.com Aug 09 2003
- "Walter" <walter digitalmars.com> Aug 10 2003
- "Mike Wynn" <mike.wynn l8night.co.uk> Aug 10 2003
- "Walter" <walter digitalmars.com> Aug 10 2003
- "Sean L. Palmer" <palmer.sean verizon.net> Aug 10 2003
- "Matthew Wilson" <matthew stlsoft.org> Aug 10 2003
- "Mike Wynn" <mike.wynn l8night.co.uk> Aug 10 2003
- "Matthew Wilson" <matthew stlsoft.org> Aug 10 2003
- "Mike Wynn" <mike.wynn l8night.co.uk> Aug 10 2003
- "Matthew Wilson" <matthew stlsoft.org> Aug 10 2003
- "Matthew Wilson" <matthew stlsoft.org> Aug 10 2003
- "Walter" <walter digitalmars.com> Aug 11 2003
- "Sean L. Palmer" <palmer.sean verizon.net> Aug 12 2003
- "Walter" <walter digitalmars.com> Aug 12 2003
- Ilya Minkov <midiclub 8ung.at> Aug 12 2003
- "Walter" <walter digitalmars.com> Aug 12 2003
- Bill Cox <bill viasic.com> Aug 12 2003
- nobody yahoo.com Aug 12 2003
- nobody yahoo.com Aug 12 2003
- "Mike Wynn" <mike.wynn l8night.co.uk> Aug 12 2003
- "Mike Wynn" <mike.wynn l8night.co.uk> Aug 11 2003
- "Matthew Wilson" <matthew stlsoft.org> Aug 11 2003
- "Mike Wynn" <mike.wynn l8night.co.uk> Aug 11 2003
- "Walter" <walter digitalmars.com> Aug 10 2003
- "Mike Wynn" <mike.wynn l8night.co.uk> Aug 10 2003
- "Walter" <walter digitalmars.com> Aug 10 2003
Has coroutine (sather style iterator) been discussed before on this list? Will D has something similar? C# has it: http://gary.burd.info/space/entry-7.html Sather has it: http://www.icsi.berkeley.edu/~sather/Publications/toplas.html Python has it: http://www.python.org/doc/current/whatsnew/section-generators.html some discussion on comp.lang.eiffel: http://groups.google.com/groups?dq=&hl=en&lr=&ie=UTF-8&threadm=uy8zjfnmc.fsf%40wanadoo.fr&prev=/groups%3Fdq%3D%26num%3D25%26hl%3Den%26lr%3D%26ie%3DUTF-8%26group%3Dcomp.lang.eiffel%26start%3D25 http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&frame=right&th=4cd646634db7588c&seekm=b0f57d2d.0110101547.78d69f50%40posting.google.com#link21
Aug 09 2003
Coroutines are pretty cool. They aren't in D now, but they could be added in the future.
Aug 10 2003
"Walter" <walter digitalmars.com> wrote in message news:bh617r$1vst$1 digitaldaemon.com...Coroutines are pretty cool. They aren't in D now, but they could be added
the future.
extern(Windows) { LPVOID CreateFiber(DWORD dwStackSize,LPFIBER_START_ROUTINE lpStartAddress,LPVOID lpParameter); VOID DeleteFiber(LPVOID lpFiber); LPVOID ConvertThreadToFiber(LPVOID lpParameter); VOID SwitchToFiber(LPVOID lpFiber); BOOL SwitchToThread(); } not sure if linux/pthreads have similar facilities. I know they do work I've tried a few basic tests with fibers, if someone posts an example of what they would like I'll see if I can produce a version in current D that uses fibers as an example.
Aug 10 2003
I've been reading about fibers on google, but I am not yet convinced they are the right solutions for coroutines because: 1) seems like a lot of overhead to manage them 2) not portable "Mike Wynn" <mike.wynn l8night.co.uk> wrote in message news:bh63a0$21m6$1 digitaldaemon.com..."Walter" <walter digitalmars.com> wrote in message news:bh617r$1vst$1 digitaldaemon.com...Coroutines are pretty cool. They aren't in D now, but they could be
inthe future.
extern(Windows) { LPVOID CreateFiber(DWORD dwStackSize,LPFIBER_START_ROUTINE lpStartAddress,LPVOID lpParameter); VOID DeleteFiber(LPVOID lpFiber); LPVOID ConvertThreadToFiber(LPVOID lpParameter); VOID SwitchToFiber(LPVOID lpFiber); BOOL SwitchToThread(); } not sure if linux/pthreads have similar facilities. I know they do work I've tried a few basic tests with fibers, if someone posts an example of what they would like I'll see if I can produce a
in current D that uses fibers as an example.
Aug 10 2003
Fibers are as lightweight as you want to make them. They have almost zero overhead compared with threads. All it is is swapping to a different stack; very similar to cooperative multitasking. You can implement them yourself using nothing more than setjmp/longjmp (so long as you're able to figure out what the stack pointer is in the registers structure, which is platform dependent). Or you can use Windows fibers, but you gain nothing by it except avoiding the hairy details of switching stacks and registers. The compiler can internally implement fibers however it wants; it's really quite simple and I can give you advice on implementation if you'd like. You would want a custom implementation for each platform to get the most performance. My coworkers used to call this "stack twiddling". ;) It can sure clear up the logic behind a state machine like you wouldn't believe. Coroutines are just language syntax that enables transparent utilization of fibers. "Yield" keyword just calls out to the fiber manager and saves the state of this fiber and lets the next fiber run, or switches back to the main fiber. Calling a coroutine starts a new fiber or switches to one that's already running, resuming at just past where the previous yield left off. You don't need language support to do this, but it would be cleaner, and the portability hassle of switching stacks would be best done in the compiler or standard library. Sean "Walter" <walter digitalmars.com> wrote in message news:bh667t$24js$1 digitaldaemon.com...I've been reading about fibers on google, but I am not yet convinced they are the right solutions for coroutines because: 1) seems like a lot of overhead to manage them 2) not portable "Mike Wynn" <mike.wynn l8night.co.uk> wrote in message news:bh63a0$21m6$1 digitaldaemon.com..."Walter" <walter digitalmars.com> wrote in message news:bh617r$1vst$1 digitaldaemon.com...Coroutines are pretty cool. They aren't in D now, but they could be
inthe future.
extern(Windows) { LPVOID CreateFiber(DWORD dwStackSize,LPFIBER_START_ROUTINE lpStartAddress,LPVOID lpParameter); VOID DeleteFiber(LPVOID lpFiber); LPVOID ConvertThreadToFiber(LPVOID lpParameter); VOID SwitchToFiber(LPVOID lpFiber); BOOL SwitchToThread(); } not sure if linux/pthreads have similar facilities. I know they do work I've tried a few basic tests with fibers, if someone posts an example of what they would like I'll see if I can produce a
in current D that uses fibers as an example.
Aug 10 2003
Win32 fibers have various problems, such as not being able to de-fiberise a thread, nor to have multiple fibers within a thread, or the prosaic one of not being implemented in 9x. I've wanted to do a platform-independent fibers API for ages, but just have never got round to it. I'd love to help do that for D, but alas would probably be more hindrance than help, asm not being my speciality ... <blushes>. They definitely should be in D, as they're very useful things to have, but should be D's own, not layered on top of Win32. "Sean L. Palmer" <palmer.sean verizon.net> wrote in message news:bh6euu$2ci7$1 digitaldaemon.com...Fibers are as lightweight as you want to make them. They have almost zero overhead compared with threads. All it is is swapping to a different
very similar to cooperative multitasking. You can implement them yourself using nothing more than setjmp/longjmp (so long as you're able to figure out what the stack pointer is in the
structure, which is platform dependent). Or you can use Windows fibers,
you gain nothing by it except avoiding the hairy details of switching
and registers. The compiler can internally implement fibers however it wants; it's
quite simple and I can give you advice on implementation if you'd like.
would want a custom implementation for each platform to get the most performance. My coworkers used to call this "stack twiddling". ;) It can sure clear
the logic behind a state machine like you wouldn't believe. Coroutines are just language syntax that enables transparent utilization
fibers. "Yield" keyword just calls out to the fiber manager and saves the state of this fiber and lets the next fiber run, or switches back to the main fiber. Calling a coroutine starts a new fiber or switches to one that's already running, resuming at just past where the previous yield
off. You don't need language support to do this, but it would be cleaner, and
portability hassle of switching stacks would be best done in the compiler
standard library. Sean "Walter" <walter digitalmars.com> wrote in message news:bh667t$24js$1 digitaldaemon.com...I've been reading about fibers on google, but I am not yet convinced
are the right solutions for coroutines because: 1) seems like a lot of overhead to manage them 2) not portable "Mike Wynn" <mike.wynn l8night.co.uk> wrote in message news:bh63a0$21m6$1 digitaldaemon.com..."Walter" <walter digitalmars.com> wrote in message news:bh617r$1vst$1 digitaldaemon.com...Coroutines are pretty cool. They aren't in D now, but they could be
inthe future.
extern(Windows) { LPVOID CreateFiber(DWORD dwStackSize,LPFIBER_START_ROUTINE lpStartAddress,LPVOID lpParameter); VOID DeleteFiber(LPVOID lpFiber); LPVOID ConvertThreadToFiber(LPVOID lpParameter); VOID SwitchToFiber(LPVOID lpFiber); BOOL SwitchToThread(); } not sure if linux/pthreads have similar facilities. I know they do work I've tried a few basic tests with fibers, if
posts an example of what they would like I'll see if I can produce a
in current D that uses fibers as an example.
Aug 10 2003
"Matthew Wilson" <matthew stlsoft.org> wrote in message news:bh6g92$2dpv$1 digitaldaemon.com...Win32 fibers have various problems, such as not being able to de-fiberise
thread, nor to have multiple fibers within a thread, or the prosaic one of not being implemented in 9x. I've wanted to do a platform-independent fibers API for ages, but just
never got round to it. I'd love to help do that for D, but alas would probably be more hindrance than help, asm not being my speciality ... <blushes>. They definitely should be in D, as they're very useful things to have, but should be D's own, not layered on top of Win32.
you don't need to de-fiber, they are cooperative so once you finish with a fiber (other than main) you delete it the only reason to "fiberise" main is to allow you to get back to it. the OS could make all threads single fibers initially, but why when fiber use is rare. (same might be said for a process why have a process with a thread, why not have a process, that then has to do ConvertProcessToThread is you want threading) fibers are a lot better than trying to do coroutines with setjump/longjump no manual manipulation of the stack from my MS docs fibers are supported on win98+ and NT3.51 sp 3 you can have multipul fibers in a thread (my D test code had 3) there are 2 ways to do coroutines (that I know of) (threads and semaphores or monitors (rendezvous are better)) should work cross platform or abuse the stack and use setjump/longjump to get between then but you can either rely on the stack not getting shrunk by the OS or if is checks ESP as task switch and free unused stack pages you'll have to use alloca or similar to get your stack back and run into issues there. and I believe there are OS's that have non contiguious stacks (you can think of the stack as a singly linked list of frames) usual function epilog is `get my old sp back and jump to the instruction after the one that called me` so on most arch's you can alloc a block of memory, store the sp params and anything else in the block as required change the SP so the pop in the epilog will get it and the back, and jump. (on risc arch's its very easy to do (they have link registers so the return address is stored by the callee (and usually no in leaf funcs) not the caller). getting OT, but as I see it use fibers on win32, on other platform you'll have to do something else. to get fiber like behavoiur you need : a way to stop the current and switch to an existing thread/process (I belive that on freebsd threads are just processes that share data space, from the OS each threads is a process rather than a process containing several threads)"Sean L. Palmer" <palmer.sean verizon.net> wrote in message news:bh6euu$2ci7$1 digitaldaemon.com...Fibers are as lightweight as you want to make them. They have almost
overhead compared with threads. All it is is swapping to a different
very similar to cooperative multitasking.
which is prementively threaded just (don't you just love the random priority boosting)).
Aug 10 2003
you don't need to de-fiber, they are cooperative so once you finish with a fiber (other than main) you delete it the only reason to "fiberise" main is to allow you to get back to it. the
could make all threads single fibers initially, but why when fiber use is rare. (same might be said for a process why have a process with a thread, why not have a process, that then has to do ConvertProcessToThread is you want threading)
This is not true. (Don't ask me to explain, as this is all in an article that's yet-to-be-published, and I want to keep my powder dry. I'm know that's an *incredibly* lame thing to say, so I'm not going to attempt to press any points on this thread further.)fibers are a lot better than trying to do coroutines with setjump/longjump no manual manipulation of the stack
Indeed. Fibers are great, and have lots of unforeseen uses. If only they were implemented a bit better in Win32, or, better, were a standard part of C. I started a SourceForge project last year - http://fiberspp.sourceforge.net/ - but have yet to do anything. I'm looking for talented people who know more about asm stuff than me to help me out ...from my MS docs fibers are supported on win98+ and NT3.51 sp 3
True, I forgot that.you can have multipul fibers in a thread (my D test code had 3)
Hmm. This does not correspond with my understanding and experience of them. Can you post your code?there are 2 ways to do coroutines (that I know of) (threads and semaphores or monitors (rendezvous are better)) should work cross platform or abuse
stack and use setjump/longjump to get between then but you can either rely on the stack not getting shrunk by the OS or if is checks ESP as task
and free unused stack pages you'll have to use alloca or similar to get
stack back and run into issues there. and I believe there are OS's that have non contiguious stacks (you can
of the stack as a singly linked list of frames) usual function epilog is `get my old sp back and jump to the instruction after the one that called me` so on most arch's you can alloc a block of memory, store the sp params and anything else in the block as required change the SP so the pop in the epilog will get it and the back, and jump. (on risc arch's its very easy
do (they have link registers so the return address is stored by the callee (and usually no in leaf funcs) not the caller). getting OT, but as I see it use fibers on win32, on other platform you'll have to do something else. to get fiber like behavoiur you need : a way to stop the current and switch to an existing thread/process (I belive that on freebsd threads are just processes that share data space, from the OS each threads is a process rather than a process containing several threads)
Aug 10 2003
"Matthew Wilson" <matthew stlsoft.org> wrote in message news:bh6rqi$2o4e$1 digitaldaemon.com...you don't need to de-fiber, they are cooperative so once you finish with
fiber (other than main) you delete it the only reason to "fiberise" main is to allow you to get back to it.
OScould make all threads single fibers initially, but why when fiber use
rare. (same might be said for a process why have a process with a
why not have a process, that then has to do ConvertProcessToThread is
want threading)
This is not true. (Don't ask me to explain, as this is all in an article that's yet-to-be-published, and I want to keep my powder dry. I'm know that's an *incredibly* lame thing to say, so I'm not going to attempt to press any points on this thread further.)
my conceptual overview of fiber, threads, process and sessions is wrong!!fibers are a lot better than trying to do coroutines with
no manual manipulation of the stack
Indeed. Fibers are great, and have lots of unforeseen uses. If only they were implemented a bit better in Win32, or, better, were a standard part
C.
with win32 fibers you can have several and jump between them :) see the example pulled from MS's own docs on the subjectI started a SourceForge project last year - http://fiberspp.sourceforge.net/ - but have yet to do anything. I'm
for talented people who know more about asm stuff than me to help me out
from my MS docs fibers are supported on win98+ and NT3.51 sp 3
True, I forgot that.you can have multipul fibers in a thread (my D test code had 3)
Hmm. This does not correspond with my understanding and experience of
Can you post your code?
import c.stdio; import windows; extern(Windows) { alias void function( LPVOID ) LPFIBER_START_ROUTINE; LPVOID CreateFiber(DWORD dwStackSize,LPFIBER_START_ROUTINE lpStartAddress,LPVOID lpParameter); VOID DeleteFiber(LPVOID lpFiber); LPVOID ConvertThreadToFiber(LPVOID lpParameter); VOID SwitchToFiber(LPVOID lpFiber); BOOL SwitchToThread(); } LPVOID lpFiber1; // Addresses of each fiber that's created LPVOID lpFiber2; LPVOID lpFiber3; LPVOID lpFiberMain; extern (Windows) void FiberRoutine( LPVOID param ) { printf( " In fiber %u\n", param ); // Tell the world who we are if ( 1 == cast(DWORD)param ) { printf( " changing fiber 1->2\n" ); SwitchToFiber( lpFiber2 ); // fiber1 -> fiber2 } else if ( 2 == cast(DWORD)param ) { printf( " changing fiber 2->3\n" ); SwitchToFiber( lpFiber3 ); // fiber2 -> fiber3 } else if ( 3 == cast(DWORD)param ) { printf( " changing fber 3->main\n" ); SwitchToFiber( lpFiberMain ); // fiber3->main fiber (original thread) } printf( " leaving fiber %u\n", param ); // Tell the world who we are printf( " returning to main fiber\n" ); // Tell the world who we are SwitchToFiber( lpFiberMain ); // fiber->main fiber (original) // if you don't end with this then you code will exit. } int main( char[][] argv ) { // Create 3 fibers, with starting addresses of the above routine lpFiber1 = CreateFiber( 16384, &FiberRoutine, cast(PVOID)1 ); lpFiber2 = CreateFiber( 16384, &FiberRoutine, cast(PVOID)2 ); lpFiber3 = CreateFiber( 16384, &FiberRoutine, cast(PVOID)3 ); if ( !lpFiber1 || !lpFiber2 || !lpFiber3 ) // Make sure the fibers were return 0; // created // Make this thread a fiber, so that it can force the other fibers to run lpFiberMain = ConvertThreadToFiber( cast(LPVOID)0 ); // Sleep for 1 second, just to prove that the other printf( "Sleeping for 1 second\n" ); Sleep( 1000 ); printf( "Switching to first fiber\n" ); SwitchToFiber( lpFiber1 ); // Switch to fiber1 printf( "Returning from SwitchToFiber ( fiber-1 )\n" ); printf( "Switching to first fiber again\n" ); SwitchToFiber( lpFiber1 ); // Switch to fiber1 printf( "Returning from (again) SwitchToFiber( fiber-1)\n" ); DeleteFiber( lpFiber1 ); // Clean up the 3 fibers we created earlier DeleteFiber( lpFiber2 ); DeleteFiber( lpFiber3 ); return 0; } --------------------------------------------- straight port (+ a bit) from the MSDN
Aug 10 2003
This is not true. (Don't ask me to explain, as this is all in an article that's yet-to-be-published, and I want to keep my powder dry. I'm know that's an *incredibly* lame thing to say, so I'm not going to attempt to press any points on this thread further.)
my conceptual overview of fiber, threads, process and sessions is wrong!!
Oh no, I knew I'd come accross sounding like a tool. <blush>. I'm not trying to be a princess, but once you've signed over the rights to your article, you're not at liberty to go splashing the details in public until it comes out. I wish I'd never said anything now .... <glum>. I certainly don't wish to imply you are ignorant of the subject. In fact, apart from the few areas in which I've used them, and found the pitfalls, I would state that I'm pretty ignorant of them. (btw, I think I was using the wrong terminology earlier. I was using fiber for fiber-group. Naturally one can have many fibers; to only have one would be a bit pointless.)Indeed. Fibers are great, and have lots of unforeseen uses. If only they were implemented a bit better in Win32, or, better, were a standard part
C.
with win32 fibers you can have several and jump between them :) see the example pulled from MS's own docs on the subject
Your example is, with respect, a little trivial. What if I want to create a fiber group with a certain peice of information, and use the (cooperative) multi-tasking nature to carry out some operation that must be done that way, and then later I want to do the same again. There's no SetFiberData(), so there's no way to start off a new fiber "task" (i.e a job involving a fiber group), unless I kick off a new thread (hardly the solution). One option would be to have a fiber-data indirection, and then you can change the task information at some later stage. But in that case we would have to ensure that all fibers in our "group" would have finished with their use of the fiber data, and were back at an initialisable position. Not impossible, but not necessarily trivial. Finally, all that is perhaps feasible when you're totally in control of all the code, but what about if it's in a library (either header, or via some API). You cannot turn a thread into a fiber twice, so if you have two libs making use of fibers the second one's going to die. (I can't remember whether the problem comes not with the second call to ConvertThreadToFiber() or on the next context switch, but I can tell you it does die.)I started a SourceForge project last year - http://fiberspp.sourceforge.net/ - but have yet to do anything. I'm
for talented people who know more about asm stuff than me to help me out
Anyway, as I said, I'm not an asm-head, but am happy to learn from/cooperate with someone who is. If anyone wants to create a cross-platform multi-fiber-group API (if such a thing is possible, of course, there're all those nasty stack guard page issues on Win32), then please shout up. It doesn't have to be on SourceForge, it could be a DMC/DMD thing. Matthew
Aug 10 2003
Just found the bits out of my article (it was written quite a long while back): "Unfortunately there is no way to close all the fibers without closing the thread (i.e. there is no CloseThreadFiber() function), in other words one cannot "de-fiber-ise" the thread. It is possible (at least on Windows 2000) to ignore a previous initialisation and call ConvertThreadToFiber() again, but the previously allocated fiber management block remains allocated. Although the MSDN knowledge base article Q185231 implies that one can simply call LocalFree() on this block, this is sailing far too close to the wind for me to recommend as a serious technique to others (since it is not part of the documented API, and therefore could be subject to breaking changes), and is certainly not suitable for a robust and flexible library. Neither is it possible to call GetFiberData() in order to determine whether fiber support has been initialised, as this raises an access violation if it has not (and I don't like the hack of catching that exception as the indicator), rather than returning NULL; even if it were, the problems of interposing one fiber sub-system on another would be inhibitive. The hitch is, of course, what if some other part of the process is already using fibers, or starts to use them during/after the initialisation of our use of them? The simple and unappealing answer is that it will crash. If two fiber sub-systems run concurrently (within the scope defined by ConvertThreadToFiber() and the exit from the calling thread) then the second one to be initialised will destroy the execution contexts of the first. If they run consecutively, then there will be a memory leak (although it is only experience, as opposed to explicit documentation on the subject, that lends this understanding)" Make sense? (I hope so, because that's what actually happens on all the Win32 OSs that I've tested it on.) Matthew "Matthew Wilson" <matthew stlsoft.org> wrote in message news:bh7cqu$7v3$1 digitaldaemon.com...This is not true. (Don't ask me to explain, as this is all in an
that's yet-to-be-published, and I want to keep my powder dry. I'm know that's an *incredibly* lame thing to say, so I'm not going to attempt
press any points on this thread further.)
my conceptual overview of fiber, threads, process and sessions is
Oh no, I knew I'd come accross sounding like a tool. <blush>. I'm not
to be a princess, but once you've signed over the rights to your article, you're not at liberty to go splashing the details in public until it comes out. I wish I'd never said anything now .... <glum>. I certainly don't wish to imply you are ignorant of the subject. In fact, apart from the few areas in which I've used them, and found the pitfalls,
would state that I'm pretty ignorant of them. (btw, I think I was using the wrong terminology earlier. I was using fiber for fiber-group. Naturally one can have many fibers; to only have one
be a bit pointless.)Indeed. Fibers are great, and have lots of unforeseen uses. If only
were implemented a bit better in Win32, or, better, were a standard
ofC.
with win32 fibers you can have several and jump between them :) see the example pulled from MS's own docs on the subject
Your example is, with respect, a little trivial. What if I want to create
fiber group with a certain peice of information, and use the (cooperative) multi-tasking nature to carry out some operation that must be done that
and then later I want to do the same again. There's no SetFiberData(), so there's no way to start off a new fiber "task" (i.e a job involving a
group), unless I kick off a new thread (hardly the solution). One option would be to have a fiber-data indirection, and then you can change the task information at some later stage. But in that case we would have to ensure that all fibers in our "group" would have finished with
use of the fiber data, and were back at an initialisable position. Not impossible, but not necessarily trivial. Finally, all that is perhaps feasible when you're totally in control of
the code, but what about if it's in a library (either header, or via some API). You cannot turn a thread into a fiber twice, so if you have two libs making use of fibers the second one's going to die. (I can't remember whether the problem comes not with the second call to
or on the next context switch, but I can tell you it does die.)I started a SourceForge project last year - http://fiberspp.sourceforge.net/ - but have yet to do anything. I'm
for talented people who know more about asm stuff than me to help me
Anyway, as I said, I'm not an asm-head, but am happy to learn
with someone who is. If anyone wants to create a cross-platform multi-fiber-group API (if such
thing is possible, of course, there're all those nasty stack guard page issues on Win32), then please shout up. It doesn't have to be on SourceForge, it could be a DMC/DMD thing. Matthew
Aug 10 2003
It sounds like fibers have an ill-designed implementation, and are not ready for prime time. Besides, I may have found a neato way to do iterators without need of coroutines. "Matthew Wilson" <matthew stlsoft.org> wrote in message news:bh7d5m$8ab$1 digitaldaemon.com...Just found the bits out of my article (it was written quite a long while back): "Unfortunately there is no way to close all the fibers without closing the thread (i.e. there is no CloseThreadFiber() function), in other words one cannot "de-fiber-ise" the thread. It is possible (at least on Windows 2000) to ignore a previous initialisation and call ConvertThreadToFiber() again, but the previously allocated fiber management block remains allocated. Although the MSDN knowledge base article Q185231 implies that one can
call LocalFree() on this block, this is sailing far too close to the wind for me to recommend as a serious
to others (since it is not part of the documented API, and therefore could be subject to breaking changes), and
certainly not suitable for a robust and flexible library. Neither is it possible to call GetFiberData() in order to determine
fiber support has been initialised, as this raises an access violation if it has not (and I don't like the hack of catching that exception as the indicator), rather than returning NULL; even if it were, the problems of interposing one fiber sub-system on another would be inhibitive. The hitch is, of course, what if some other part of the process is already using fibers, or starts to use them during/after the initialisation of our use of them? The simple and unappealing answer
that it will crash. If two fiber sub-systems run concurrently (within the scope defined by ConvertThreadToFiber() and
exit from the calling thread) then the second one to be initialised will destroy the execution contexts of the first. If they run consecutively, then there will be a memory leak (although it is only experience, as opposed to explicit documentation on the subject, that lends this understanding)" Make sense? (I hope so, because that's what actually happens on all the Win32 OSs that I've tested it on.) Matthew "Matthew Wilson" <matthew stlsoft.org> wrote in message news:bh7cqu$7v3$1 digitaldaemon.com...This is not true. (Don't ask me to explain, as this is all in an
that's yet-to-be-published, and I want to keep my powder dry. I'm
that's an *incredibly* lame thing to say, so I'm not going to
topress any points on this thread further.)
ofmy conceptual overview of fiber, threads, process and sessions is
Oh no, I knew I'd come accross sounding like a tool. <blush>. I'm not
to be a princess, but once you've signed over the rights to your
you're not at liberty to go splashing the details in public until it
out. I wish I'd never said anything now .... <glum>. I certainly don't wish to imply you are ignorant of the subject. In
apart from the few areas in which I've used them, and found the
Iwould state that I'm pretty ignorant of them. (btw, I think I was using the wrong terminology earlier. I was using
for fiber-group. Naturally one can have many fibers; to only have one
be a bit pointless.)Indeed. Fibers are great, and have lots of unforeseen uses. If only
were implemented a bit better in Win32, or, better, were a standard
ofC.
with win32 fibers you can have several and jump between them :) see
example pulled from MS's own docs on the subject
Your example is, with respect, a little trivial. What if I want to
afiber group with a certain peice of information, and use the
multi-tasking nature to carry out some operation that must be done that
and then later I want to do the same again. There's no SetFiberData(),
there's no way to start off a new fiber "task" (i.e a job involving a
group), unless I kick off a new thread (hardly the solution). One option would be to have a fiber-data indirection, and then you can change the task information at some later stage. But in that case we
have to ensure that all fibers in our "group" would have finished with
use of the fiber data, and were back at an initialisable position. Not impossible, but not necessarily trivial. Finally, all that is perhaps feasible when you're totally in control of
the code, but what about if it's in a library (either header, or via
API). You cannot turn a thread into a fiber twice, so if you have two
making use of fibers the second one's going to die. (I can't remember whether the problem comes not with the second call to
or on the next context switch, but I can tell you it does die.)I started a SourceForge project last year - http://fiberspp.sourceforge.net/ - but have yet to do anything. I'm
for talented people who know more about asm stuff than me to help me
Anyway, as I said, I'm not an asm-head, but am happy to learn
with someone who is. If anyone wants to create a cross-platform multi-fiber-group API (if
athing is possible, of course, there're all those nasty stack guard page issues on Win32), then please shout up. It doesn't have to be on SourceForge, it could be a DMC/DMD thing. Matthew
Aug 11 2003
I'll bite. Sean "Walter" <walter digitalmars.com> wrote in message news:bha2fs$5dq$1 digitaldaemon.com...It sounds like fibers have an ill-designed implementation, and are not
for prime time. Besides, I may have found a neato way to do iterators without need of coroutines.
Aug 12 2003
I want to try and implement it first to see if it works. I have my best ideas when out jogging, including this one. What I like to do is 'load up on context', which means do all my research at the computer, with this newsgroup, google, my books, etc. Then out the door for a run. I think all the pounding seems to throw all the bits of data around in the air, shaking them around until they fit together <g>. "Sean L. Palmer" <palmer.sean verizon.net> wrote in message news:bha9su$dpk$1 digitaldaemon.com...I'll bite. Sean "Walter" <walter digitalmars.com> wrote in message news:bha2fs$5dq$1 digitaldaemon.com...It sounds like fibers have an ill-designed implementation, and are not
for prime time. Besides, I may have found a neato way to do iterators without need of coroutines.
Aug 12 2003
Walter wrote:I want to try and implement it first to see if it works. I have my best ideas when out jogging, including this one. What I like to do is 'load up on context', which means do all my research at the computer, with this newsgroup, google, my books, etc. Then out the door for a run. I think all the pounding seems to throw all the bits of data around in the air, shaking them around until they fit together <g>.
This is also one of the key symptomes of addiction - when a person keeps thinking about it when away from it. :) i.e. Internet addicts are said to have a "dense mindstream" as soon as they disconnect. -i.
Aug 12 2003
"Ilya Minkov" <midiclub 8ung.at> wrote in message news:bhbrfg$1vr0$1 digitaldaemon.com...Walter wrote:I want to try and implement it first to see if it works. I have my best ideas when out jogging, including this one. What I like to do is 'load
context', which means do all my research at the computer, with this newsgroup, google, my books, etc. Then out the door for a run. I think
the pounding seems to throw all the bits of data around in the air,
them around until they fit together <g>.
thinking about it when away from it. :) i.e. Internet addicts are said to have a "dense mindstream" as soon as they disconnect.
What else is there to think about?
Aug 12 2003
Walter wrote:It sounds like fibers have an ill-designed implementation, and are not ready for prime time. Besides, I may have found a neato way to do iterators without need of coroutines.
I'm all ears. I'd love to here about D's iterators. Bill
Aug 12 2003
Coroutine is more than just iterators: http://www.cs.utexas.edu/users/wilson/schintro/schintro_75.html " ... is far more powerful than normal procedure calling and returning, and allows you to implement advanced control structures such as backtracking, cooperative multitasking, and custom exception-handling." In article <bharr3$vcu$2 digitaldaemon.com>, Bill Cox says...Walter wrote:It sounds like fibers have an ill-designed implementation, and are not ready for prime time. Besides, I may have found a neato way to do iterators without need of coroutines.
I'm all ears. I'd love to here about D's iterators. Bill
Aug 12 2003
Coroutine is more than just iterators: http://www.cs.utexas.edu/users/wilson/schintro/schintro_75.html " ... is far more powerful than normal procedure calling and returning, and allows you to implement advanced control structures such as backtracking, cooperative multitasking, and custom exception-handling." In article <bharr3$vcu$2 digitaldaemon.com>, Bill Cox says...Walter wrote:It sounds like fibers have an ill-designed implementation, and are not ready for prime time. Besides, I may have found a neato way to do iterators without need of coroutines.
I'm all ears. I'd love to here about D's iterators. Bill
Aug 12 2003
yes, but it sounds like Walters worked out how to do iterators without coroutines iterators don't need the ability to unwind the stack back beyond the original point of creation iterators can be implemented as an object with a single member function which has no locals or params but instead all locals are member of the object (kind of like the innerfunction model which considers a function to be a member function of a stack frame). then either a switch on state or a way to save the location of the last yeild. (yeild either updates a state int, or saves the address of the next instruction in the "start from here" pointer) although this does not limit the iterated function to be only leaf function it does limit the yeild point to only being within that function (you can not yeild from a function that the itterating function calls). <nobody yahoo.com> wrote in message news:bhbeht$1k53$1 digitaldaemon.com...Coroutine is more than just iterators: http://www.cs.utexas.edu/users/wilson/schintro/schintro_75.html " ... is far more powerful than normal procedure calling and returning,
allows you to implement advanced control structures such as backtracking, cooperative multitasking, and custom exception-handling." In article <bharr3$vcu$2 digitaldaemon.com>, Bill Cox says...Walter wrote:It sounds like fibers have an ill-designed implementation, and are not
for prime time. Besides, I may have found a neato way to do iterators without need of coroutines.
I'm all ears. I'd love to here about D's iterators. Bill
Aug 12 2003
Your example is, with respect, a little trivial. What if I want to create
fiber group with a certain peice of information, and use the (cooperative) multi-tasking nature to carry out some operation that must be done that
and then later I want to do the same again. There's no SetFiberData(), so there's no way to start off a new fiber "task" (i.e a job involving a
group), unless I kick off a new thread (hardly the solution). One option would be to have a fiber-data indirection, and then you can change the task information at some later stage. But in that case we would have to ensure that all fibers in our "group" would have finished with
use of the fiber data, and were back at an initialisable position. Not impossible, but not necessarily trivial. Finally, all that is perhaps feasible when you're totally in control of
the code, but what about if it's in a library (either header, or via some API). You cannot turn a thread into a fiber twice, so if you have two libs making use of fibers the second one's going to die. (I can't remember whether the problem comes not with the second call to
or on the next context switch, but I can tell you it does die.)
that the thread has been fibered), and everyone use the same fiber library. by fiber group, I assume you mean a fiber that creates other fibers that "yield" to it rather than back to main ? (easy to do).
Aug 11 2003
Finally, all that is perhaps feasible when you're totally in control of
the code, but what about if it's in a library (either header, or via
API). You cannot turn a thread into a fiber twice, so if you have two
making use of fibers the second one's going to die. (I can't remember whether the problem comes not with the second call to
or on the next context switch, but I can tell you it does die.)
that the thread has been fibered), and everyone use the same fiber
So you're saying the answer to the problem of the Win32 fiber implementation not supporting multiple "fiber-groups" is to not use multiple "fiber-groups". Well, yeah! That solves the problem, but means that the usefulness of Win32 fibers is dramatically curtailed. Isn't that just a clearer statement of my proposition that Win32 fibers are not all that useful?by fiber group, I assume you mean a fiber that creates other fibers that "yield" to it rather than back to main ? (easy to do).
No, I mean that in your code you call some API function - we assume 3rd party - which merrily does some fiber stuff. The code then returns to you, and you carry on merrily, unaware that your thread has been "fiberised". At a later time you call another function - perhaps one of your own - that also makes use of fibers. The minimum harm here is that you will have a memory leak (at least as far as current versions of Win32 is concerned; future ones may well crash at this point). But what if the call to the 3rd party fiber-using API occurs after your fiber initialisation, and within the time that you are switching between the fibers. Crash! Are you telling me that this is desirable functionality?
Aug 11 2003
"Matthew Wilson" <matthew stlsoft.org> wrote in message news:bh84p7$11vh$1 digitaldaemon.com...by fiber group, I assume you mean a fiber that creates other fibers that "yield" to it rather than back to main ? (easy to do).
No, I mean that in your code you call some API function - we assume 3rd party - which merrily does some fiber stuff. The code then returns to you, and you carry on merrily, unaware that your thread has been "fiberised".
a later time you call another function - perhaps one of your own - that
makes use of fibers.
program that uses a library that uses it, you can't in your main app etc.. it does seem that GetCurrentFiber() is o.k. to call before you've converted your thread to a fiber ..... strangly I get the value 0x00001E00 had expected null. I see XP has `ConvertFiberToThread` and server 2003 has ConvertThreadToFiberEx to allow the x86 fpu state to be saved as the NT fibers don't store FPU state (now that might be an issue, but I would have though you would have flushed any FPU op's before yielding anyway) server 2003 also has Fiber local storage. and from the win32 headers I deduced that FS:0x10 was the current fiber data pointer and that the first enty is a pointer to the FiberData so wrote SetFiberData and it works on Win2K (see test code) I could see nothing in the Docs that implied that you can not call ConvertThreadToFiber twice but I made the assumption that the fiber state is only stored when you call SwitchToFiber and I was right, you can call convertThreadToFiber twice, and switch between then ... I've even tried manually setting the current fiber and the switching you can create some fun loops :) the docs do mention that switching to current fiber is not advisable but it does work as a way to set the return position of a fiber. see the test doc. I felt like an afternoon of being abusive to win2K, better than reading exchange docs on how write code to backup a server over a network. it would seem that if you call a library that uses fibers, and when it returns the current fiber is not the same as it was, you can either manually convert back to your main fiber and delete the one it was changed to, or delete your created fiber and continue with the one you have now. thus as I see it you can protect yourself for a lib that does not check that the thread has been fiberised first the docs mention that you can switch to a fiber of a different thread ... now that opens up some fun doors. I personally don't see anything that implies co-routines should not be written using fibers, just like any other interaction with the OS there are issuee for the user to consider.
Aug 11 2003
"Sean L. Palmer" <palmer.sean verizon.net> wrote in message news:bh6euu$2ci7$1 digitaldaemon.com...Fibers are as lightweight as you want to make them. They have almost zero overhead compared with threads. All it is is swapping to a different
very similar to cooperative multitasking. You can implement them yourself using nothing more than setjmp/longjmp (so long as you're able to figure out what the stack pointer is in the
structure, which is platform dependent). Or you can use Windows fibers,
you gain nothing by it except avoiding the hairy details of switching
and registers. The compiler can internally implement fibers however it wants; it's
quite simple and I can give you advice on implementation if you'd like.
would want a custom implementation for each platform to get the most performance. My coworkers used to call this "stack twiddling". ;) It can sure clear
the logic behind a state machine like you wouldn't believe. Coroutines are just language syntax that enables transparent utilization
fibers. "Yield" keyword just calls out to the fiber manager and saves the state of this fiber and lets the next fiber run, or switches back to the main fiber. Calling a coroutine starts a new fiber or switches to one that's already running, resuming at just past where the previous yield
off. You don't need language support to do this, but it would be cleaner, and
portability hassle of switching stacks would be best done in the compiler
standard library.
I'd like to eventually add coroutines to D, but there are other things that need to be done first. If you'd like to do a coroutine class, sort of like some of the C++ ones out there, that could become part of Phobos. It could hide the portability issues.
Aug 10 2003
import c.stdio;
import windows;
import object;
extern(Windows) {
alias void function( LPVOID ) LPFIBER_START_ROUTINE;
LPVOID CreateFiber(DWORD dwStackSize,LPFIBER_START_ROUTINE
lpStartAddress,LPVOID lpParameter);
VOID DeleteFiber(LPVOID lpFiber);
LPVOID ConvertThreadToFiber(LPVOID lpParameter);
VOID SwitchToFiber(LPVOID lpFiber);
}
// T is the return type from the co-routine
// U is the user context for the co-routine
class CoManagerBase {
static LPVOID mainFiber = null;
Exception throw_me = null;
LPVOID this_fiber = null;
bit delete_me = false;
static void initForCoRoutines() {
if ( mainFiber === null ) {
mainFiber = ConvertThreadToFiber( null );
}
}
~this() {
if ( this_fiber ) { endCoRoutine(); }
}
void createCoRoutine( LPFIBER_START_ROUTINE start_up ) {
assert( this_fiber === null );
this_fiber = CreateFiber( 16384, start_up, cast(LPVOID)this );
}
void yeildCoRoutine() {
assert( CoManagerBase.mainFiber );
SwitchToFiber( CoManagerBase.mainFiber ); // fiber->main fiber
(original)
}
void deleteCoRoutine() {
assert( this_fiber );
DeleteFiber( this_fiber );
this_fiber = null;
delete_me = false;
}
void endCoRoutine() {
delete_me = true;
SwitchToFiber( CoManagerBase.mainFiber );
}
void processCoRoutine() {
assert( this_fiber );
SwitchToFiber( this_fiber );
}
bit isActive() { return (this_fiber !== null); }
}
template coroutine( T, U ) {
alias void (*user_func)( CoManagerBase cm, U uservalue );
class CoManager : CoManagerBase {
user_func func;
U value;
T return_value;
public:
this() {
initForCoRoutines();
}
this( user_func func0, U user_value ) {
this();
begin( func0, user_value );
}
void begin( user_func func0, U user_value ) {
assert( isActive() == false );
func = func0;
value = user_value;
createCoRoutine( fiberCoRoutine );
}
void yield( T rv ) {
return_value = rv;
yeildCoRoutine();
}
void end( T rv ) {
return_value = rv;
endCoRoutine();
}
T process() {
processCoRoutine();
if ( throw_me ) { throw throw_me; }
if ( delete_me ) { deleteCoRoutine(); }
return return_value;
}
}
extern (Windows) void
fiberCoRoutine( LPVOID param )
{
CoManager cm = cast(CoManager)param;
cm.func( cm, cm.value );
while( true ){
cm.throw_me = new Exception( "CoRoutine overrun" );
SwitchToFiber( CoManagerBase.mainFiber ); // fiber->main fiber
(original)
}
}
// alias void function( CoManager mgr, U uservalue ) user_func;
}
struct Range { int start, end; }
instance coroutine( int, Range ) cr_I_R;
instance coroutine( int, int ) cr_I_I;
void inc_by( CoManagerBase mgr, int incr ) {
int i = 0;
while(true) {
i += incr;
(cast(cr_I_I.CoManager)mgr).yield( i );
}
}
void counter( CoManagerBase mgr, Range r ) {
int i = r.start;
while( i < r.end ) {
(cast(cr_I_R.CoManager)mgr).yield( i );
i++;
}
(cast(cr_I_R.CoManager)mgr).end( i );
}
int main( char[][] argv )
{
static Range r = { start:4, end:9 };
cr_I_I.CoManager coI = new cr_I_I.CoManager( cast(cr_I_I.user_func)&inc_by,
11 );
cr_I_R.CoManager coR = new cr_I_R.CoManager( cast(cr_I_R.user_func)counter,
r );
printf( "being some corountines\n" );
while( coR.isActive() ) {
printf( " incrementer : %d\n", coI.process() );
printf( "counter[range] : %d\n", coR.process() );
}
printf( "*done*\n" );
return 0;
}
----------------
how's that for proof that fibers are A easy and B do the job.
I'd like to eventually add coroutines to D, but there are other things
need to be done first. If you'd like to do a coroutine class, sort of like
some of the C++ ones out there, that could become part of Phobos. It could
hide the portability issues.
I've attached the versions I'd rather use (less casting) but form some
reason (looks like a forward ref problem)
I cant define two mutually dependant types within a template :(
Aug 10 2003
I'm going to have to study it a bit. Can you make it work on linux? Write a doc file for it? "Mike Wynn" <mike.wynn l8night.co.uk> wrote in message news:bh711f$2td5$1 digitaldaemon.com...import c.stdio; import windows; import object; extern(Windows) { alias void function( LPVOID ) LPFIBER_START_ROUTINE; LPVOID CreateFiber(DWORD dwStackSize,LPFIBER_START_ROUTINE lpStartAddress,LPVOID lpParameter); VOID DeleteFiber(LPVOID lpFiber); LPVOID ConvertThreadToFiber(LPVOID lpParameter); VOID SwitchToFiber(LPVOID lpFiber); } // T is the return type from the co-routine // U is the user context for the co-routine class CoManagerBase { static LPVOID mainFiber = null; Exception throw_me = null; LPVOID this_fiber = null; bit delete_me = false; static void initForCoRoutines() { if ( mainFiber === null ) { mainFiber = ConvertThreadToFiber( null ); } } ~this() { if ( this_fiber ) { endCoRoutine(); } } void createCoRoutine( LPFIBER_START_ROUTINE start_up ) { assert( this_fiber === null ); this_fiber = CreateFiber( 16384, start_up, cast(LPVOID)this ); } void yeildCoRoutine() { assert( CoManagerBase.mainFiber ); SwitchToFiber( CoManagerBase.mainFiber ); // fiber->main fiber (original) } void deleteCoRoutine() { assert( this_fiber ); DeleteFiber( this_fiber ); this_fiber = null; delete_me = false; } void endCoRoutine() { delete_me = true; SwitchToFiber( CoManagerBase.mainFiber ); } void processCoRoutine() { assert( this_fiber ); SwitchToFiber( this_fiber ); } bit isActive() { return (this_fiber !== null); } } template coroutine( T, U ) { alias void (*user_func)( CoManagerBase cm, U uservalue ); class CoManager : CoManagerBase { user_func func; U value; T return_value; public: this() { initForCoRoutines(); } this( user_func func0, U user_value ) { this(); begin( func0, user_value ); } void begin( user_func func0, U user_value ) { assert( isActive() == false ); func = func0; value = user_value; createCoRoutine( fiberCoRoutine ); } void yield( T rv ) { return_value = rv; yeildCoRoutine(); } void end( T rv ) { return_value = rv; endCoRoutine(); } T process() { processCoRoutine(); if ( throw_me ) { throw throw_me; } if ( delete_me ) { deleteCoRoutine(); } return return_value; } } extern (Windows) void fiberCoRoutine( LPVOID param ) { CoManager cm = cast(CoManager)param; cm.func( cm, cm.value ); while( true ){ cm.throw_me = new Exception( "CoRoutine overrun" ); SwitchToFiber( CoManagerBase.mainFiber ); // fiber->main fiber (original) } } // alias void function( CoManager mgr, U uservalue ) user_func; } struct Range { int start, end; } instance coroutine( int, Range ) cr_I_R; instance coroutine( int, int ) cr_I_I; void inc_by( CoManagerBase mgr, int incr ) { int i = 0; while(true) { i += incr; (cast(cr_I_I.CoManager)mgr).yield( i ); } } void counter( CoManagerBase mgr, Range r ) { int i = r.start; while( i < r.end ) { (cast(cr_I_R.CoManager)mgr).yield( i ); i++; } (cast(cr_I_R.CoManager)mgr).end( i ); } int main( char[][] argv ) { static Range r = { start:4, end:9 }; cr_I_I.CoManager coI = new cr_I_I.CoManager(
11 ); cr_I_R.CoManager coR = new cr_I_R.CoManager(
r ); printf( "being some corountines\n" ); while( coR.isActive() ) { printf( " incrementer : %d\n", coI.process() ); printf( "counter[range] : %d\n", coR.process() ); } printf( "*done*\n" ); return 0; } ---------------- how's that for proof that fibers are A easy and B do the job.I'd like to eventually add coroutines to D, but there are other things
need to be done first. If you'd like to do a coroutine class, sort of
some of the C++ ones out there, that could become part of Phobos. It
hide the portability issues.
I've attached the versions I'd rather use (less casting) but form some reason (looks like a forward ref problem) I cant define two mutually dependant types within a template :(
Aug 10 2003









"Walter" <walter digitalmars.com> 