Erlang-Lua:
How to write an Erlang C Node

Problem: I want to integrate my Erlang program with that other existing technology over there.

Several ways of getting Erlang to interact with foreign systems:

• Port programs
  
• C / Java / Python / ... Nodes
  
• Linked-in Drivers (aka Port Driver)
  
• NIFs (Natively Implemented Functions)
  

Port programs

• Easy and simplistic
  
• Safe
  
• For any decent work, you start inventing a protocol
  

Natively Implemented Functions

• Relatively easy
  
• Unsafe
  
• Potentially quite difficult
  (e.g., scheduling, see further http://www.codemesh.io/codemesh2014/steve-vinoski)
  

C/Java Nodes

• A bit harder to get going initially
  
• Safe
  
• Uses standard Erlang distributed messaging
  
• Can use Port to start/stop and monitor
  

My example: Allow calling out to Lua from Erlang

https://github.com/rtraschke/erlang-lua

Stick to something manageable

• C Node is started from Erlang
  
• Managed through a Port
  
• Assume that C program is on path
  
• C program is single threaded
  
• C program uses EI library (http://www.erlang.org/doc/man/ei_connect.html and http://www.erlang.org/doc/man/ei.html)
  
• Simple interface from Erlang: a gen_server with API
  
• start_link(Id)
• lua(Id, Binary_String)
• stop(Id)

What I want

(rtr@127.0.0.1)1> erlang_lua:start_link(foo).

{ok,<0.47.0>}

(rtr@127.0.0.1)2> erlang_lua:lua(foo, <<"return {x=1, y='foo'}">>).

{lua,[[{y,<<"foo">>},{x,1}]]}

(rtr@127.0.0.1)3> erlang_lua:lua(foo, <<" x = 42 + 'fourty two' ">>).

{error,"[string \" x = 42 + 'fourty two' \"]:1: attempt to perform
 arithmetic on a string value"}

(rtr@127.0.0.1)4> erlang_lua:stop(foo).

ok

(rtr@127.0.0.1)5>

Starting up (Erlang side)

• Start the gen_server
  
• Build a command line (includes arguments: desired C Node name, hostname, Erlang Node name and cookie)
  
• Spawn the command using open_port/2:
• text line streaming across stdin/stdout
• redirect stderr to stdout
• exit status reporting enabled
• The server state holds at least port reference and the mailbox designator used later on to communicate with the C Node using Erlang messaging
  
• Wait for the C program to finish initialising itself
  

Starting up (C side)

• Check the arguments
  
• Set up stdin and stdout as unbuffered and binary (i.e., no cr/lf mangling)
  
• Set up buffers used for the messaging (ei_x_new(ei_x_buff* x))
  
• Fetch our IP address (struct hostent *gethostbyname(char *hostname)) and build the C Node name
  
• Initialise the C Node (ei_connect_xinit(...))
  
• Connect it to the Erlang Node (ei_connect(...))
  
• Initialise the Lua State
  
• Notify our monitoring gen_server that all is ready
  
• Start the message loop
  

Monitoring (Erlang side)

• The C Node writes to stdout; the gen_server receives:
• {Port, {data, {noeol, S}}}
• {Port, {data, {eol, S}}}

• The C Node exits with an exit code:
• {Port, {exit_status, N}}
  
• The C Node crashes in some other way:
• {'EXIT', Port, Reason}
  
• The C Node ready signal is just a print on stdout

Executing some Lua code

• A message is sent to the C Node with the code to execute
• A message is received back from the C Node with the result
• Simple, right?

handle_call({exec, Code}, From, #state{mbox=Mbox} = State) ->
    Mbox ! {exec, self(), Code},
    Result = receive
        {error, _Reason} = Error ->
            Error;
        {lua, _Result} = Reply
            Reply
    end,
    {reply, Result, State};

Executing some Lua code (cont.)

Not quite, remember that we are in a gen_server, and port messages could arrive while we are waiting for our result. For example, an exit of the C Node, due to an error.

handle_call({exec, Code}, From, #state{mbox=Mbox} = State) ->
    Mbox ! {exec, self(), Code},
    Result = receive
        {error, _Reason} = Error ->
            Error;
        {lua, _Result} = Reply
            Reply
    end,
    {reply, Result, State};

Executing some Lua code (cont.)

Use two-part gen_server calls:

• Initial call sends message to C Node and returns a noreply
  
• Reply from C Node is handled as a message and uses gen_server:reply/2

handle_call({exec, Code}, From,
        #state{mbox=Mbox, from=undefined} = State) ->
    Mbox ! {exec, self(), Code},
    {noreply, State#state{from=From}};

handle_info({error, _Reason} = Error, #state{from=From} = State)
        when From =/= undefined ->
    gen_server:reply(From, Error),
    {noreply, State#state{from=undefined}};
handle_info({lua, _Result} = Reply, #state{from=From} = State)
        when From =/= undefined ->
    gen_server:reply(From, Reply),
    {noreply, State#state{from=undefined}};

Message Loop (C side)

• Wait to receive a message (int ei_xreceive_msg(int fd, erlang_msg* msg, ei_x_buff* x))
• Return codes:
• ERL_TICK - The Erlang Node sends these regularly to see if the C Node is still alive
• ERL_ERROR - Some kind of error happened while waiting for a message
• ERL_MSG - A real message arrived, metadata is in the erlang_msg structure, and the payload is in the ei_x_buff

Message Loop (C side) (cont.)

• The three main message types we are interested in are
• ERL_EXIT and ERL_UNLINK - The Erlang Node is disconnecting, in our case this means shut down the C node
• ERL_SEND - The actual message sent from the gen_server to the C Node, e.g., {exec, Pid, Code} and {stop, Pid, []}
• When a gen_server message arrives, handle it and then return the result using int ei_send(int fd, erlang_pid* to, char* buf, int len)
• The fd in the two calls (ei_xreceive_msg() and ei_send()) was returned to us by the ei_connect() call earlier and is the descriptor of the connection to the Erlang Node

Handling the Payload (C side)

• Use the various ei_decode_...() functions to disassemble the expected terms
• Having all messages be the same length tuple with the initial atom signifying the operation makes this quite straightforward
• Malformed messages must come from somewhere else, so they simply get ignored

if (ei_decode_version(x_in->buff, &x_in->index, &version) < 0) {
    print("WARNING: Ignoring malformed message (bad version: %d).", version); return -1;
}
if (ei_decode_tuple_header(x_in->buff, &x_in->index, &arity) < 0) {
    print("WARNING: Ignoring malformed message (not tuple)."); return -1;
}
if (arity != 3) {
    print("WARNING: Ignoring malformed message (not 3-arity tuple)."); return -1;
}
if (ei_decode_atom(x_in->buff, &x_in->index, lua_atom) < 0) {
    print("WARNING: Ignoring malformed message (first tuple element not atom)."); return -1;
}
if (ei_decode_pid(x_in->buff, &x_in->index, pid) < 0) {
    print("WARNING: Ignoring malformed message (second tuple element not pid)."); return -1;
}

Handling the Payload (C side) (cont.)

if (strcmp(lua_atom, "stop") == 0) {
    print("DEBUG: Lua Erlang Node stopping normally.");
    x_out->index = 0;
    return 0;
}
if (strcmp(lua_atom, "exec") == 0) {
    ei_get_type(x_in->buff, &x_in->index, &type, &len);
    code = (char *) calloc(len+1, sizeof(char));
    if (ei_decode_binary(x_in->buff, &x_in->index, code, NULL) < 0) {
        free(code);
        set_error_msg(x_out, "Third tuple element is not a binary.");
        return 1; /* send the error message back to our gen_server */
    }
}
x_out->index = 0;
ei_x_encode_version(x_out);
ei_x_encode_tuple_header(x_out, 2);
ei_x_encode_atom(x_out, "lua");
execute_code(EI_LUA_STATE.L, x_out, code);

Handling the Payload (C side) (cont.)

• The supplied Lua code is handed over to luaL_dostring()
• Any error during evaluation is converted into an {error, Reason} reply
• If there was no return from the Lua eval, a {lua, ok} reply is sent back
• Any regular returned values from the Lua code evaluation is shipped back as the {lua, Result} reply
• Such Lua result tree is walked and translated into an Erlang term using the ei_x_encode_...() functions
• ( This might make a nice follow-up presentation one day ☺ )

So, How Does This Look (Starting Up)

(rtr@127.0.0.1)1> erlang_lua:start_link(foo).                      
=INFO REPORT==== 26-Nov-2014::10:50:04 ===
2014-11-26 10:50:04.621 UTC INFO ELua 'foo' starting using command:
lua5.1/bin/lua_enode 'foo' '127.0.0.1' 'rtr@127.0.0.1'
    'BXYEUYSCJRYFVTWCEXAO' '0'

=INFO REPORT==== 26-Nov-2014::10:50:04 ===
2014-11-26 10:50:04.627 UTC DEBUG ELua 'foo' startup message:
Lua Erlang Node 'foo@127.0.0.1' starting.

=INFO REPORT==== 26-Nov-2014::10:50:04 ===
2014-11-26 10:50:04.628 UTC DEBUG ELua 'foo' startup message:
Lua Erlang Node started.

=INFO REPORT==== 26-Nov-2014::10:50:04 ===
2014-11-26 10:50:04.628 UTC INFO ELua 'foo' is ready to accept Lua code.

{ok,<0.47.0>}
(rtr@127.0.0.1)2> 

So, How Does This Look (Run Something)

(rtr@127.0.0.1)2> erlang_lua:lua(foo, <<"return {x=1, y='foo'}">>).
=INFO REPORT==== 26-Nov-2014::10:50:11 ===
2014-11-26 10:50:11.730 UTC DEBUG ELua 'foo' executing:
return {x=1, y='foo'}

{lua,[[{y,<<"foo">>},{x,1}]]}
(rtr@127.0.0.1)3>

So, How Does This Look (Stop It)

(rtr@127.0.0.1)3> erlang_lua:stop(foo).
=INFO REPORT==== 26-Nov-2014::11:08:11 ===
2014-11-26 11:08:11.179 UTC DEBUG ELua 'foo' is being asked to stop.

=INFO REPORT==== 26-Nov-2014::11:08:11 ===
2014-11-26 11:08:11.179 UTC INFO ELua 'foo' terminating: normal

=INFO REPORT==== 26-Nov-2014::11:08:11 ===
2014-11-26 11:08:11.180 UTC DEBUG ELua 'foo':
Lua Erlang Node stopping normally.

=INFO REPORT==== 26-Nov-2014::11:08:11 ===
2014-11-26 11:08:11.180 UTC INFO ELua 'foo':
Lua Erlang Node stopped.

=INFO REPORT==== 26-Nov-2014::11:08:11 ===
2014-11-26 11:08:11.180 UTC INFO ELua 'foo' stopped normally.

ok
(rtr@127.0.0.1)4>

So, How Does This Look (Oops)

(rtr@127.0.0.1)2> erlang_lua:lua(foo,
        <<"os.execute('sleep 300') return {x=1, y='foo'}">>).

=INFO REPORT==== 26-Nov-2014::11:12:43 ===
2014-11-26 11:12:43.472 UTC DEBUG ELua 'foo' executing:
os.execute('sleep 300') return {x=1, y='foo'}

=ERROR REPORT==== 26-Nov-2014::11:13:32 ===
** Node 'foo@127.0.0.1' not responding **
** Removing (timedout) connection **

=INFO REPORT==== 26-Nov-2014::11:17:43 ===
2014-11-26 11:17:43.448 UTC DEBUG ELua 'foo':
Lua Erlang Node error in receive: 5 (Input/output error)

=INFO REPORT==== 26-Nov-2014::11:17:43 ===
2014-11-26 11:17:43.448 UTC INFO ELua 'foo':
Lua Erlang Node 'foo@127.0.0.1' reconnecting.

=INFO REPORT==== 26-Nov-2014::11:17:43 ===
2014-11-26 11:17:43.449 UTC INFO ELua 'foo':
Lua Erlang Node reconnected.

NOTE: The call is now stuck without a return!

Oops, What Happened There?

• Erlang pings connected Nodes regularly to see if the connections are still alive
• This is controlled by the kernel net_ticktime config setting
• If enough pings have not been answered, Erlang consideres the connection severed
• A reply from our C Node will then fail to reach the Erlang Node
• A subsequent call to ei_xreceive_msg() in the C node produces the I/O error
• The C Node can reconnect, but the reply message was lost forever

Oops, What Can Be Done?

• A Java Node (Jinterface) does not have this problem, as the internal message handling is run in a separate thread
• Adding a thread to C sounds like quite a bit of work
• A more pragmatic approach is needed
• A reconnection is possible, so all that is needed, is to figure out that the C Node has been disconnected
• Before sending the reply back to the gen_server, try calling something on the Erlang Node
• If that fails, the C Node needs to reconnect

Oops, What Can Be Done?

Call to erlang:is_alive() before sending the reply in the C main_message_loop():

case ERL_SEND:
    x_in->index = 0;
    running = handle_msg(&pid);
    if (running == -1) {
        running = 1; /* Ignore messages without a return pid! */
    } else {
        x_rpc_in->index = x_rpc_out->index = 0;
        ei_x_encode_empty_list(x_rpc_in); /*empty param list for erlang:is_alive()*/
        if (ei_rpc(&EI_LUA_STATE.ec, EI_LUA_STATE.fd, "erlang", "is_alive",
                x_rpc_in->buff, x_rpc_in->index, x_rpc_out) < 0) {
            reconnect();
        }
        if (x_out->index > 0 && ei_send(EI_LUA_STATE.fd, &pid,
                x_out->buff, x_out->index) < 0) {
            print("FATAL: Lua Erlang Node error in send to '%s'.", pid.node);
            exit(8);
        }
    }
    break;

Oops Resolved

(rtr@127.0.0.1)2> erlang_lua:lua(foo,
        <<"os.execute('sleep 300') return {x=1, y='foo'}">>).
=INFO REPORT==== 26-Nov-2014::10:52:45 ===
2014-11-26 10:52:45.791 UTC DEBUG ELua 'foo' executing:
os.execute('sleep 300') return {x=1, y='foo'}

=ERROR REPORT==== 26-Nov-2014::10:53:32 ===
** Node 'foo@127.0.0.1' not responding **
** Removing (timedout) connection **

=INFO REPORT==== 26-Nov-2014::10:57:45 ===
2014-11-26 10:57:45.769 UTC DEBUG ELua 'foo':
Lua Erlang Node error in 'is alive?' rpc to 'rtr@127.0.0.1'.

=INFO REPORT==== 26-Nov-2014::10:57:45 ===
2014-11-26 10:57:45.769 UTC INFO ELua 'foo':
Lua Erlang Node 'foo@127.0.0.1' reconnecting.

=INFO REPORT==== 26-Nov-2014::10:57:45 ===
2014-11-26 10:57:45.770 UTC INFO ELua 'foo':
Lua Erlang Node reconnected.

{lua,[[{y,<<"foo">>},{x,1}]]}
(rtr@127.0.0.1)3>

Summing Up

C / Java / Python / ... Nodes

• Safer than NIFs and Linked Drivers
• More powerful than Port programs
• More boilerplate
• Single threaded C net tick timeout gotcha
• Can be easily monitored when started as a Port program
• Can use simple C printf() for logging when started as a Port program

I would use a NIF

• Only when the communication cost between Erlang and C outweighs the computation cost
• And even then, only if I think the added risk is worth it

Available on Github

https://github.com/rtraschke/erlang-lua

Future Plans

• An extension to Erlang-LuaJava is in the works
• As is a load balancer for multiple Erlang-Lua Nodes using Rabbit MQ RPC
• A presentation on how to use the encode/decode functions for Erlang terms might be nice

Thank You!

Robby Raschke
Erlang Solutions Ltd.
@rtraschke