www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - Doost status update

reply Marcin Kuszczak <aarti_please_no spam_interia.pl> writes:
Dear D community!

I would like to give you some information about status of doost library, 
as it got few substantial updates recently.

*NEW MODULES*
*doost.storage.Storage/FileStorage*
I want to start from this one as this module as it introduces concept 
which is used in other modules. Basically Storage is continuous array of 
elements of same type. String and array of ints are also kinds of storage.
The difference is that Storage can be much bigger than computer memory. 
Storage can be also mapped to real physical devices such as File, 
Network connection etc. In fact it is very similar concept to Stream, 
but it has additional requirement that all elements of Storage are of 
the same type. Storage have narrower set of operations than arrays. 
Supported operations are:
- checking if element with given index is in storage or outside of 
stream of elements - eos
- consuming first or few first elements from storage - get
- peeking into any element in storage (only forward) - peek
- it is also possible to operate directly on 'working window' - frame

You can move across storage in one direction and operate on 'working 
window'. First element of storage have always index of '0'. Below simple 
visualization of storage:
sink -----------|O--working window--O|----------source
O - optional buffers for input and output

It might look complicated in a first look but, it's not in fact. Using 
this module it is possible to write all operations on arrays in a such a 
way that it will be possible to use any other Storage than string and 
have it read data from e.g. file.

In storage module there are definitions of above operations for strings, 
so you can use:
char[] text;
text.put("blabla");
char c= text.get()
if (text.eos(10)) throw new Exception("Error");

When using in you program you can template it in such a way that it will 
work nevertheless user will put as Storage string or e.g. FileStorage, 
which will read data from file.
When using for strings there should be no performance penalty as all 
functions should be inlined Unfortunately DMD does not optimize 
functions which takes arguments by ref. (See: bugzilla issue: 2008).
There is one concrete Storage implemented for files in package
doost.storage.

This module is already quite important in my library, but I still think 
it needs some more conceptual work.

*doost.text.Scanner*
Module Scanner is intended to be used for parsing texts. It consists from
set of functions, which uses internally only one operation: scan. Every
function scan returnes struct Matcher, which keeps information about
results of scanning: status of scanning, error message, matched position,
index of match (in case of matching against sets of sequences or sets of
characters).

position - position in input
repeatno - how many times repeat scanning; for uint.max functions scan 
for more than zero matches
All below functions works also with instances of Storages (see above).

Following template functions are available:
Matcher scan(input, CharClass, position, repeatno)  //CharClass -> 
basically checks if input from 'position' begins with one of characters 
defined in CharClass
Matcher scan(input, SequenceSet, position, repeatno) //SequenceSet -> 
same as CharClass, but for sequences
Matcher scan(input, RegExp, position, repeatno)
Matcher scan(input, array, position, repeatno)
Matcher scan(input, element_of_array, position, repeatno)

For below functions condition can be any of types for scan  + Matcher;
uint skip(ref T input, S condition, uint repeatno=1) // returns number 
of skipped characters
StorageType!(T) munch(T, S)(ref T input, S condition, uint repeatno=1) 
// consumes characters/sequences/etc. returning them from function

Additional high level functions:
Matcher scanEscapeSequence - scans for escape sequence
Matcher scanString - scans for string
Matcher scanRegion - scans for region; region is delimited by matching 
characters e.g. [5,5,4]
Matcher scanNumber - scans for number; needs some more work
Matcher scanDate - scans for number; needs some more work

Scanner uses internally among others CharClass by Jascha Wetzel from Tango
ported to Phobos. Module needs some additional work to make it feature
complete.

*doost.text.Escaper*
This is quite simple module, which should be used for 
escaping/unescaping strings. It should work for every escape sequence 
defined by D. It has only two functions:

escape(input)
unescape(input)

Escaper module should work also for Storage (see above).

*doost.util.serializer*
This is automatic serialization library. Currently it has two compile 
time 'backends': TextArchive and JsonArchive.
Some of most important features:
- can work with arrays and as well with storages
- highly customizable
- thread/exception safe
- tracking of references (classes/pointers)
- user defined types versioning
- importing of old classes
- user defined constructors
- user defined loaders/dumpers
- different archive classes

There is quite big suite of unit tests for serialization. You can see
serializer features in usage there. Serializer works also with storages
(see above).

*doost.api.Common/doost.core*
- doost.api is meant as something similar to tango.groups; Common is for 
commonly used definitions/imports etc.
- doost.core: Any (not many changes here), Traits (templates mostly 
taken from Phobos/Tango)


*UPDATED MODULES*
*doost.util.DUnit (doost.DUnit)*
This module was improved to make tests visualization in console more
readable and usefull. Currently it shows nice table with all necessary
informations. You can set width of this table, you can say if you want to
get timing or just result of unit test, you can also disable one or more
tests. It is possible to define groups of unittests. There are printed some
useful statistics at the end of table. Traces for tests are shown only in
case of failures. 

*doost.util.config.ProgramOptions*
Program Options now internally uses serializer to store information. It
means that every type you will use to configure your program can be
automatically stored as text. I think that this module needs some redesign.
It is especially visible after introducing serializer, which obsolete few
features and force to change implementation. Currently few unit tests (not
most important) doesn't pass, but making them work needs rethinking
architecture and redesign. I will keep it for later :-)

*WORK IN PROGRESS*
Currently I am working on database package. Idea is to make operation on 
database (almost) completely type-safe. I want to achieve this using Sql 
Object pattern. Thanks to this it will be possible to send such an 
object to database layer and get recordset as result.

*HOW CAN YOU HELP (if you feel it is worthy)*
- comment about design - especially about interfaces and use-cases.
- prepare patches and attach them as tickets
- documentation: it is rather sparse. I will be happy to get some help here
- adding unit tests - fixing bugs
- adding support for Tango. In this I would hope for some help from 
Tango team especially with Storage concept.
- improving build system. Currently I am using CodeBlocks, but its far from
perfect solution. There is a need for better/more universal builder. Right
now it should be dsss, later maybe something other.
- making Sqlite database from DDBI project working in latest revision 
:-) It's terribly broken right now.

Few modules of doost have dependencies on ddbi and std2, so you will need to
play a little bit with your compiler configuration. I used ddbi revision
62, as trunk version is broken for Sqlite. I successfully run doost
libraries on Tango with Tangobos (I had to add string aliases to
object.di), but doost itself doesn't contain any dependencies on Tango.
CodeBlocks project files might not work in your environment, so your
probably will have to adjust them to.

If you think, you can help with some tasks regarding library please let me
know e.g. by e-mail.

-- 
Regards
Marcin Kuszczak (Aarti_pl)
-------------------------------------
Ask me why I believe in Jesus - http://www.zapytajmnie.com (en/pl)
Doost (port of few Boost libraries) - http://www.dsource.org/projects/doost/
-------------------------------------
Apr 30 2008
next sibling parent reply Daniel Giddings <daniel.giddings gmail.com> writes:
Marcin Kuszczak wrote:
 - making Sqlite database from DDBI project working in latest revision 
 :-) It's terribly broken right now.

StatementBinder. Removing the prepared statements code will let it compile and run (without prepared statements), or you could implement the StatementBinder and it may be ok I guess (not sure where that is up to). For tango you also need to rename fromUtf8z to fromStringz and toUtf8z to toStringz
Apr 30 2008
parent Marcin Kuszczak <aarti_please_no spam_interia.pl> writes:
Daniel Giddings wrote:

 Marcin Kuszczak wrote:
 - making Sqlite database from DDBI project working in latest revision
 :-) It's terribly broken right now.

StatementBinder. Removing the prepared statements code will let it compile and run (without prepared statements), or you could implement the StatementBinder and it may be ok I guess (not sure where that is up to). For tango you also need to rename fromUtf8z to fromStringz and toUtf8z to toStringz

Yes, probably it's not difficult, but the problem is that you have to distract yourself from your goal to fix some other library. Currently AFAIK ddbi has no active maintainer. And it is really pity as it's important project for D. -- Regards Marcin Kuszczak (Aarti_pl) ------------------------------------- Ask me why I believe in Jesus - http://www.zapytajmnie.com (en/pl) Doost (port of few Boost libraries) - http://www.dsource.org/projects/doost/ -------------------------------------
May 01 2008
prev sibling parent reply Jason House <jason.james.house gmail.com> writes:
Since doost is based on boost, it'd be nice to see a simple list of which
boost modules have been ported.  This helps those of us already familiar
with various boost libraries...

PS: I'm glad to see doost is alive and well!

Marcin Kuszczak wrote:

 Dear D community!
 
 I would like to give you some information about status of doost library,
 as it got few substantial updates recently.
 
 *NEW MODULES*
 *doost.storage.Storage/FileStorage*
 I want to start from this one as this module as it introduces concept
 which is used in other modules. Basically Storage is continuous array of
 elements of same type. String and array of ints are also kinds of storage.
 The difference is that Storage can be much bigger than computer memory.
 Storage can be also mapped to real physical devices such as File,
 Network connection etc. In fact it is very similar concept to Stream,
 but it has additional requirement that all elements of Storage are of
 the same type. Storage have narrower set of operations than arrays.
 Supported operations are:
 - checking if element with given index is in storage or outside of
 stream of elements - eos
 - consuming first or few first elements from storage - get
 - peeking into any element in storage (only forward) - peek
 - it is also possible to operate directly on 'working window' - frame
 
 You can move across storage in one direction and operate on 'working
 window'. First element of storage have always index of '0'. Below simple
 visualization of storage:
 sink -----------|O--working window--O|----------source
 O - optional buffers for input and output
 
 It might look complicated in a first look but, it's not in fact. Using
 this module it is possible to write all operations on arrays in a such a
 way that it will be possible to use any other Storage than string and
 have it read data from e.g. file.
 
 In storage module there are definitions of above operations for strings,
 so you can use:
 char[] text;
 text.put("blabla");
 char c= text.get()
 if (text.eos(10)) throw new Exception("Error");
 
 When using in you program you can template it in such a way that it will
 work nevertheless user will put as Storage string or e.g. FileStorage,
 which will read data from file.
 When using for strings there should be no performance penalty as all
 functions should be inlined Unfortunately DMD does not optimize
 functions which takes arguments by ref. (See: bugzilla issue: 2008).
 There is one concrete Storage implemented for files in package
 doost.storage.
 
 This module is already quite important in my library, but I still think
 it needs some more conceptual work.
 
 *doost.text.Scanner*
 Module Scanner is intended to be used for parsing texts. It consists from
 set of functions, which uses internally only one operation: scan. Every
 function scan returnes struct Matcher, which keeps information about
 results of scanning: status of scanning, error message, matched position,
 index of match (in case of matching against sets of sequences or sets of
 characters).
 
 position - position in input
 repeatno - how many times repeat scanning; for uint.max functions scan
 for more than zero matches
 All below functions works also with instances of Storages (see above).
 
 Following template functions are available:
 Matcher scan(input, CharClass, position, repeatno)  //CharClass ->
 basically checks if input from 'position' begins with one of characters
 defined in CharClass
 Matcher scan(input, SequenceSet, position, repeatno) //SequenceSet ->
 same as CharClass, but for sequences
 Matcher scan(input, RegExp, position, repeatno)
 Matcher scan(input, array, position, repeatno)
 Matcher scan(input, element_of_array, position, repeatno)
 
 For below functions condition can be any of types for scan  + Matcher;
 uint skip(ref T input, S condition, uint repeatno=1) // returns number
 of skipped characters
 StorageType!(T) munch(T, S)(ref T input, S condition, uint repeatno=1)
 // consumes characters/sequences/etc. returning them from function
 
 Additional high level functions:
 Matcher scanEscapeSequence - scans for escape sequence
 Matcher scanString - scans for string
 Matcher scanRegion - scans for region; region is delimited by matching
 characters e.g. [5,5,4]
 Matcher scanNumber - scans for number; needs some more work
 Matcher scanDate - scans for number; needs some more work
 
 Scanner uses internally among others CharClass by Jascha Wetzel from Tango
 ported to Phobos. Module needs some additional work to make it feature
 complete.
 
 *doost.text.Escaper*
 This is quite simple module, which should be used for
 escaping/unescaping strings. It should work for every escape sequence
 defined by D. It has only two functions:
 
 escape(input)
 unescape(input)
 
 Escaper module should work also for Storage (see above).
 
 *doost.util.serializer*
 This is automatic serialization library. Currently it has two compile
 time 'backends': TextArchive and JsonArchive.
 Some of most important features:
 - can work with arrays and as well with storages
 - highly customizable
 - thread/exception safe
 - tracking of references (classes/pointers)
 - user defined types versioning
 - importing of old classes
 - user defined constructors
 - user defined loaders/dumpers
 - different archive classes
 
 There is quite big suite of unit tests for serialization. You can see
 serializer features in usage there. Serializer works also with storages
 (see above).
 
 *doost.api.Common/doost.core*
 - doost.api is meant as something similar to tango.groups; Common is for
 commonly used definitions/imports etc.
 - doost.core: Any (not many changes here), Traits (templates mostly
 taken from Phobos/Tango)
 
 
 *UPDATED MODULES*
 *doost.util.DUnit (doost.DUnit)*
 This module was improved to make tests visualization in console more
 readable and usefull. Currently it shows nice table with all necessary
 informations. You can set width of this table, you can say if you want to
 get timing or just result of unit test, you can also disable one or more
 tests. It is possible to define groups of unittests. There are printed
 some useful statistics at the end of table. Traces for tests are shown
 only in case of failures.
 
 *doost.util.config.ProgramOptions*
 Program Options now internally uses serializer to store information. It
 means that every type you will use to configure your program can be
 automatically stored as text. I think that this module needs some
 redesign. It is especially visible after introducing serializer, which
 obsolete few features and force to change implementation. Currently few
 unit tests (not most important) doesn't pass, but making them work needs
 rethinking architecture and redesign. I will keep it for later :-)
 
 *WORK IN PROGRESS*
 Currently I am working on database package. Idea is to make operation on
 database (almost) completely type-safe. I want to achieve this using Sql
 Object pattern. Thanks to this it will be possible to send such an
 object to database layer and get recordset as result.
 
 *HOW CAN YOU HELP (if you feel it is worthy)*
 - comment about design - especially about interfaces and use-cases.
 - prepare patches and attach them as tickets
 - documentation: it is rather sparse. I will be happy to get some help
 here - adding unit tests - fixing bugs
 - adding support for Tango. In this I would hope for some help from
 Tango team especially with Storage concept.
 - improving build system. Currently I am using CodeBlocks, but its far
 from perfect solution. There is a need for better/more universal builder.
 Right now it should be dsss, later maybe something other.
 - making Sqlite database from DDBI project working in latest revision
 :-) It's terribly broken right now.
 
 Few modules of doost have dependencies on ddbi and std2, so you will need
 to play a little bit with your compiler configuration. I used ddbi
 revision 62, as trunk version is broken for Sqlite. I successfully run
 doost libraries on Tango with Tangobos (I had to add string aliases to
 object.di), but doost itself doesn't contain any dependencies on Tango.
 CodeBlocks project files might not work in your environment, so your
 probably will have to adjust them to.
 
 If you think, you can help with some tasks regarding library please let me
 know e.g. by e-mail.
 

Apr 30 2008
parent Marcin Kuszczak <aarti_please_no spam_interia.pl> writes:
Jason House wrote:

 Since doost is based on boost, it'd be nice to see a simple list of which
 boost modules have been ported.  This helps those of us already familiar
 with various boost libraries...
 
 PS: I'm glad to see doost is alive and well!

Well, currently scope of library is much wider than only libraries ported from Boost or inspired by Boost. While I was porting program_options I noticed that originall library can be easily extended with e.g. writing options back into physicall storage. D just allows to do much more, still having implementation quite simple. Now I am working on D version of part of U++ library, which is intended to allow type safe database queries. (http://www.ultimatepp.org/src$Sql$SqlExp$en-us.html) Imagine that you put database schema into file: http://www.dsource.org/projects/doost/browser/trunk/examples/database/TestSchema.schema and then you use in your program typesafe versions of sql: http://www.dsource.org/projects/doost/browser/trunk/examples/database/PackageTest.d And simple cases work already quite well :-) I just see problem with sql expressions. In U++ it is done like below: Select(BOOK.AUTHOR).WHERE((BOOK.ID == 100) && (BOOK.TITLE == "Dot")); In D it is not possible (unfortunately) and must be written like below: Select(BOOK.AUTHOR).WHERE(land(leq(BOOK.ID, 100), leq(BOOK.TITLE, "Dot"))); I hope that maybe Walter someday will reconsider adding extended operators overloading as addition to the current system. -- Regards Marcin Kuszczak (Aarti_pl) ------------------------------------- Ask me why I believe in Jesus - http://www.zapytajmnie.com (en/pl) Doost (port of few Boost libraries) - http://www.dsource.org/projects/doost/ -------------------------------------
May 01 2008