www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.learn - LuaD: creating a flexible data filter system

reply yawniek <dlang srtnwz.com> writes:
hi,

i'm reading in a stream of data that is deserialized into 
individual frames.
a frame is either of:
a)  a specific D datastructure ( struct with a few 
ulong,string,string[string] etc members), known at compile time
b) json (preferably stdx.data.json)

i now want to implement something where i can dynamically
add lua filters that then get data out of these frames, create a 
new lua object
send it back to D code where its either sent to another lua 
filter or at last
  being serialized again to json und then being processed further.

ideally i would not like to copy all the data into a lua object 
but directly access it
from lua.
is there an elegant approach to do this and support both a and b 
cases?

so far i did some benchmarks, mainly with string comparisons and 
it turned out
that luaD is about 10x faster than mruby and python D bridges.
Oct 16 2015
next sibling parent reply Chris <wendlec tcd.ie> writes:
On Friday, 16 October 2015 at 09:01:57 UTC, yawniek wrote:
 hi,

 i'm reading in a stream of data that is deserialized into 
 individual frames.
 a frame is either of:
 a)  a specific D datastructure ( struct with a few 
 ulong,string,string[string] etc members), known at compile time
 b) json (preferably stdx.data.json)

 i now want to implement something where i can dynamically
 add lua filters that then get data out of these frames, create 
 a new lua object
 send it back to D code where its either sent to another lua 
 filter or at last
  being serialized again to json und then being processed 
 further.

 ideally i would not like to copy all the data into a lua object 
 but directly access it
 from lua.
 is there an elegant approach to do this and support both a and 
 b cases?

 so far i did some benchmarks, mainly with string comparisons 
 and it turned out
 that luaD is about 10x faster than mruby and python D bridges.
You could use the Lua C API directly. I have a small project where I use DerelictLua[1] and access the Lua C API directly for fast processing (cf. http://www.lua.org/manual/5.3/ or http://www.lua.org/manual/5.2/ "C API"). Another performance trick is to compile lua functions and store them in memory, rather than having lua read and execute everything. lua_State* L = luaL_newstate(); auto func = luaL_loadstring(L, "Lua code as C-string\0"); // Lua file compiled as a function Later you call the function with the Lua C API like "lua_pcall(L, 0, 1, 0);". It's a bit tricky to move things around on the Lua stack, but you'll get there! ;) [1] https://github.com/DerelictOrg/DerelictLua
Oct 16 2015
parent reply Jakob Ovrum <jakobovrum gmail.com> writes:
On Friday, 16 October 2015 at 10:45:52 UTC, Chris wrote:
 Later you call the function with the Lua C API like 
 "lua_pcall(L, 0, 1, 0);". It's a bit tricky to move things 
 around on the Lua stack, but you'll get there! ;)
Or you could use LuaD which doesn't require you to mess around with the relatively unproductive, bug-prone C API :)
Oct 16 2015
parent reply Chris <wendlec tcd.ie> writes:
On Saturday, 17 October 2015 at 02:02:16 UTC, Jakob Ovrum wrote:
 On Friday, 16 October 2015 at 10:45:52 UTC, Chris wrote:
 Later you call the function with the Lua C API like 
 "lua_pcall(L, 0, 1, 0);". It's a bit tricky to move things 
 around on the Lua stack, but you'll get there! ;)
Or you could use LuaD which doesn't require you to mess around with the relatively unproductive, bug-prone C API :)
I've used both, LuaD and DerelictLua + my own smaller D library that wraps all these nasty Lua stack operations (e.g. creating, accessing and adding to tables). The better I got to know the Lua C API while writing my own wrappers, the more I came to appreciate LuaD :-) However, LuaD is still 5.1 and I didn't want to be stuck with 5.1. So I rolled my own. It's not as comprehensive as LuaD but did the trick. The purpose was to experiment with Lua server pages + vibe.d which worked fine. Rikki gave me the idea to compile each page as a Lua function into memory for faster execution (etLua style[1]). It works fine with both LuaD and my own wrappers. In my own version I use the Lua C API directly in some places, though I don't know, if that's really a big speedup. If I set up my own homepage, I'd probably give vibe.d + Lua a shot. No more PHP and sh*t like that. [1] https://github.com/leafo/etlua
Oct 17 2015
parent reply yawniek <dlang srtnwz.com> writes:
many thanks for the valuable insights.
so far i made a simple prototype with LuaD and classes, works 
nicely for when my niput

what so far is not 100% clear is if there is a way to have a 
parsed
msgpack or json documents being exposed in my lua code in a way 
so it behaves
like a lua object.
Ideally in a RW fashion so that changed then again can be 
processed by D code.
Oct 17 2015
parent Laeeth Isharc <Laeeth.nospam nospam-laeeth.com> writes:
On Saturday, 17 October 2015 at 13:15:17 UTC, yawniek wrote:
 many thanks for the valuable insights.
 so far i made a simple prototype with LuaD and classes, works 
 nicely for when my niput

 what so far is not 100% clear is if there is a way to have a 
 parsed
 msgpack or json documents being exposed in my lua code in a way 
 so it behaves
 like a lua object.
 Ideally in a RW fashion so that changed then again can be 
 processed by D code.
http://luajit.org/ext_ffi_tutorial.html C meta methods
Oct 17 2015
prev sibling next sibling parent Laeeth Isharc <Laeeth.nospam nospam-laeeth.com> writes:
On Friday, 16 October 2015 at 09:01:57 UTC, yawniek wrote:
 hi,

 i'm reading in a stream of data that is deserialized into 
 individual frames.
 a frame is either of:
 a)  a specific D datastructure ( struct with a few 
 ulong,string,string[string] etc members), known at compile time
 b) json (preferably stdx.data.json)

 i now want to implement something where i can dynamically
 add lua filters that then get data out of these frames, create 
 a new lua object
 send it back to D code where its either sent to another lua 
 filter or at last
  being serialized again to json und then being processed 
 further.

 ideally i would not like to copy all the data into a lua object 
 but directly access it
 from lua.
 is there an elegant approach to do this and support both a and 
 b cases?

 so far i did some benchmarks, mainly with string comparisons 
 and it turned out
 that luaD is about 10x faster than mruby and python D bridges.
I haven't done more than play with it so far, but see LuaJIT FFI. Obv LuaJIT is fast too. From memory I think you can access C structs directly. In addition you can write meta methods that provide nice sugar for accessing, constructors, etc.
Oct 16 2015
prev sibling parent Jakob Ovrum <jakobovrum gmail.com> writes:
On Friday, 16 October 2015 at 09:01:57 UTC, yawniek wrote:
 hi,

 i'm reading in a stream of data that is deserialized into 
 individual frames.
 a frame is either of:
 a)  a specific D datastructure ( struct with a few 
 ulong,string,string[string] etc members), known at compile time
 b) json (preferably stdx.data.json)

 i now want to implement something where i can dynamically
 add lua filters that then get data out of these frames, create 
 a new lua object
 send it back to D code where its either sent to another lua 
 filter or at last
  being serialized again to json und then being processed 
 further.

 ideally i would not like to copy all the data into a lua object 
 but directly access it
 from lua.
 is there an elegant approach to do this and support both a and 
 b cases?

 so far i did some benchmarks, mainly with string comparisons 
 and it turned out
 that luaD is about 10x faster than mruby and python D bridges.
The class support in LuaD supports the kind of indirection you want, but it's probably not finished enough for your use case. Once closer to completion, it will be possible to use it with structs as well.
Oct 16 2015