Cisco Unified Application Environment Developer Forums

« Back to Etch

Re: [Etch] Etch Wire protocol spec ?

Combination View Flat View Tree View
Threads [ Previous | Next ]
[redirecting back to the list, we want to start capturing an dsharing
this sort of
discussion..]

comments inline...

Shahed Moolji wrote:
Just a quick question / clarification.

For my client to be able to talk to my seed server (Option B, till
step 4), I assume this would work only
if there are no messages directed to the client (callbacks)

(This is obvious if you are refering to the simple add service,
where the server does not talk back to the client)
Just wanted to confirm I understood correctly.
actually, to implement add you need to have messages in both directions.
that's
the test, add( 2, 3 ) has to return the right result, 5, at every step
along the way.

certainly as you proceed you want to be as live as possible. so, where
FooHelper
(indirectly) creates a DeliveryService and returns it (all wrapped up in the
RemoteFooServer) to the client, you should do the same, doing by hand what
TransportHelper does, and just use a fake implementation of DeliveryService
with nothing underneath. when its
begincall method is called, it can return a fake mailbox stuffed with a
result of the
appropriate type. the point is to keep your client happy, not be general
purpose.
on the next pass, trash your fake DeliveryService and replace it with a
port of the
real one which uses some lower level fake-ness to do the same thing.
pretty soon,
all the fake stuff is down at the packet level and then data level and
then gone.
yippee!

my watch phrase has been "fastest time to result". as soon as you can
completely
handle the add message end to end, you can look back and see if you have any
fundamental problems to implement the breadth of what etch is. along the
path you
will take short cuts and leave lots of debris. that is ok. get down and
dirty with add
and make it happen. there is a huge moral boost that comes from seeing
it work with
itself and the other bindings on that very simple service.

this is also a good time to start running performance tests. the perf
example can be
used as an example, and necessary parts ported over. if you can catch
any performance
problems early, less code needs to be fixed. use java as your baseline,
there's no reason
you shouldn't be able to get close to it (but you can run some non-etch
tests of simple
operations to get a good idea how java and objective c compare
performance-wise).

you will also then understand a lot more about how etch works, which
will nicely
inform your more careful second phase to implement the rest and clean up
your
first phase messes.

we will start work very soon on an interop test suite which should also
be helpful.
the unit tests for binary tagged data, along with other unit tests,
cover the same
territory in a piece-wise fashion, but we'd like to see a more complete
end-to-end
demonstration.

good luck and keep us posted!

scott out
Once again thaks a lot.... I am about to start the process of diving
deep into the java code....

Regards
Shahed.

2008/8/14 scott comer (sccomer) <sccomer@cisco.com>:

hi shahed,

you are correct in the two items that need doing. we've not written the
porting guide yet
but we have ported etch enough times to have a good idea about the proper
ordering
and technique.

step 0 is to pick a binding whose nature is very close to your chosen
language. let's call
this the seed binding. design a very simple etch service and implement it in
the seed binding.
i followed a just in time and a test driven development methodology. using
this methodology
will get you interesting results much faster than a breadth first scheme.
the plan is to
implement only what you need to get the current task done. depth first
programming.

your simple etch service should have one method: int add(int x, int y). your
task at this
point is to implement this service fully and then study the code the
compiler produced
for you. note that the generated add methods can take normal integer values
such as 2
but also null. note the use of Type, Field, Validator, and Message. become
one with the
generated files and understand their purpose. here is a sample etch service:

module foo
service Foo { int add( int x, int y ) }

your server implementation should look something like this:

public Integer add( Integer x, Integer y )
{
if (x == null || y == null) return null;
return x + y;
}

your client implementation should look something like this:

assertEquals( 5, server.add( 2, 3 ) );
assertNull( server.add( null, 3 ) );
assertNull( server.add( 2, null ) );
assertNull( server.add( null, null ) );

when you are happy that this works and that you understand how the elements
of the generated files are working together, you may proceed. don't worry
and
don't try to understand what is going on under the hood yet.

step 1 is to copy the seed binding, say java, to start your new one. so,
copy
all the files from etch/binding-java to etch/binding-objc. you will use
these files and
their internal documentation to guide you along the way. (you could start
with
etch/binding-csharp, too, though it is less well documented.)

step 2 is to ignore the compiler. the compiler is a finishing touch which is
best left for
last.

step 3 get oriented

the runtime is the first order of business. the runtime is loosely organized
into 4 parts. the
best approach is to port them in a certain order by starting with the
existing code (i'll
assume that is java).

etch\binding-java\runtime contains the source and the build.xml. ignore the
ant file and
let's look at src. src is composed, in the maven pattern, of main and test.
there are many
unit tests in test which will help you with your port. their structure
mimics main. diving
deep in main we get to
etch\binding-java\runtime\src\main\java\etch\bindings\java:

msg - message model
support - the glue between messages and transport
transport - realization of support interfaces tied to actual protocols
util - general purpose utility stuff

msg is the foundation of the message model of etch. these are the primary
classes
used to model messages for the support and transport layers. the compiler
mainly
generates code which uses these classes. they are all very simple, and
porting them
should be fairly straightforward in a modern object oriented language with a
garbage
collector. i'm told by rene that objective c fits the bill nicely. however,
his initial investigation
yielded a number of naming conflicts which have to be resolved. it is
beneficial to keep
class and method names as similar as possible to other languages to
facilitate patches,
etc.

when the compiler generates code, it uses the classes from msg and support
to express
the implementation of the service.

step 4 dive in

from here it is, currently, a feel your way along operation. different paths
are possible:

option A:
1) port msg and its unit tests. don't stop until all the unit tests work.
2) repeat for support and transport.
3) port classes and their unit tests (if any) from util as you need them.
there is a lot of stuff
there, much of which you won't need.
4) this technique has been used successfully to bring along csharp, c, ruby,
and python
bindings from java seed (though csharp is the only binding which is
finished). this works
if the target binding is largely similar to the seed binding.
5) test your service against an existing binding.

option B:
1) rewrite your compiler generated files from your Foo etch service into
your target
language. start with the client and then only generated files needed by the
client. ignore
the listener / server components.
2) port the classes from the seed binding to the target binding needed to
make the
sample client compile and run. port in layers (msg, then support, then
transport), stubbing
out stuff below the current layer in order to facilitate testing. (so, port
msg while stubbing
anything from support and transport; only port stuff from util as needed.)
3) as above, port support, then transport, again only pulling stuff from
util as you need it.
4) you should be able to use your client implementation against the seed
binding
implementation of your server.
5) repeat this process with your service listener until you have your entire
service implemented
in the target language.
6) you should be able to use the seed version of your client to talk to the
target version of your
listener. of course, target versions of your client and listener should be
able to talk.
7) pat yourself on the back, proof of concept complete.
8) now finish per option A.

I suggest option B.

the unit tests present at each level will be a big help. do not ignore them.
eventually you will
get to the part about tcp sockets and wire formats. those have tests, too.
binary tagged data
tests include injecting actual packets and testing that they decode
properly, and vice versa.
there is also a document describing binary tagged data which is not yet
posted. that's on my
todo list, but you won't need it for a week or so...

certainly check back here for help or advice or just to let us know how it
is going. i personally
want to see an objective c binding so that iphone developers can start
playing with etch...

scott out

Shahed Moolji wrote:

Hi,

I am wondering what would be involved, if I want to implement Etch in
a new language (say Objective-C)

I guess the 2 parts are :-

1. The language bindings

I believe this can be done by customizing the current compiler
with velocity templates for the target language

2. The runtime implementation.

This would reqire an indepth knowledge of the RPC messages and
data serialization for each supported transport (TCP / TLS).

If these two steps are correct approach, then are there any docs
available that document the protocol spec ?
Also, are there any test tools available to check interoparability ?

Thanks
Shahed.
_______________________________________________
Etch mailing list
Etch@developer.cisco.com

_______________________________________________
Etch mailing list
Etch@developer.cisco.com

Attachment not added (content type not allowed): "sccomer.vcf"

[redirecting back to the list, we want to start capturing an dsharing
this sort of
discussion..]

comments inline...

Shahed Moolji wrote:
Just a quick question / clarification.

For my client to be able to talk to my seed server (Option B, till
step 4), I assume this would work only
if there are no messages directed to the client (callbacks)

(This is obvious if you are refering to the simple add service,
where the server does not talk back to the client)
Just wanted to confirm I understood correctly.
actually, to implement add you need to have messages in both directions.
that's
the test, add( 2, 3 ) has to return the right result, 5, at every step
along the way.

certainly as you proceed you want to be as live as possible. so, where
FooHelper
(indirectly) creates a DeliveryService and returns it (all wrapped up in the
RemoteFooServer) to the client, you should do the same, doing by hand what
TransportHelper does, and just use a fake implementation of DeliveryService
with nothing underneath. when its
begincall method is called, it can return a fake mailbox stuffed with a
result of the
appropriate type. the point is to keep your client happy, not be general
purpose.
on the next pass, trash your fake DeliveryService and replace it with a
port of the
real one which uses some lower level fake-ness to do the same thing.
pretty soon,
all the fake stuff is down at the packet level and then data level and
then gone.
yippee!

my watch phrase has been "fastest time to result". as soon as you can
completely
handle the add message end to end, you can look back and see if you have any
fundamental problems to implement the breadth of what etch is. along the
path you
will take short cuts and leave lots of debris. that is ok. get down and
dirty with add
and make it happen. there is a huge moral boost that comes from seeing
it work with
itself and the other bindings on that very simple service.

this is also a good time to start running performance tests. the perf
example can be
used as an example, and necessary parts ported over. if you can catch
any performance
problems early, less code needs to be fixed. use java as your baseline,
there's no reason
you shouldn't be able to get close to it (but you can run some non-etch
tests of simple
operations to get a good idea how java and objective c compare
performance-wise).

you will also then understand a lot more about how etch works, which
will nicely
inform your more careful second phase to implement the rest and clean up
your
first phase messes.

we will start work very soon on an interop test suite which should also
be helpful.
the unit tests for binary tagged data, along with other unit tests,
cover the same
territory in a piece-wise fashion, but we'd like to see a more complete
end-to-end
demonstration.

good luck and keep us posted!

scott out
Once again thaks a lot.... I am about to start the process of diving
deep into the java code....

Regards
Shahed.

2008/8/14 scott comer (sccomer) <sccomer@cisco.com>:

hi shahed,

you are correct in the two items that need doing. we've not written the
porting guide yet
but we have ported etch enough times to have a good idea about the proper
ordering
and technique.

step 0 is to pick a binding whose nature is very close to your chosen
language. let's call
this the seed binding. design a very simple etch service and implement it in
the seed binding.
i followed a just in time and a test driven development methodology. using
this methodology
will get you interesting results much faster than a breadth first scheme.
the plan is to
implement only what you need to get the current task done. depth first
programming.

your simple etch service should have one method: int add(int x, int y). your
task at this
point is to implement this service fully and then study the code the
compiler produced
for you. note that the generated add methods can take normal integer values
such as 2
but also null. note the use of Type, Field, Validator, and Message. become
one with the
generated files and understand their purpose. here is a sample etch service:

module foo
service Foo { int add( int x, int y ) }

your server implementation should look something like this:

public Integer add( Integer x, Integer y )
{
if (x == null || y == null) return null;
return x + y;
}

your client implementation should look something like this:

assertEquals( 5, server.add( 2, 3 ) );
assertNull( server.add( null, 3 ) );
assertNull( server.add( 2, null ) );
assertNull( server.add( null, null ) );

when you are happy that this works and that you understand how the elements
of the generated files are working together, you may proceed. don't worry
and
don't try to understand what is going on under the hood yet.

step 1 is to copy the seed binding, say java, to start your new one. so,
copy
all the files from etch/binding-java to etch/binding-objc. you will use
these files and
their internal documentation to guide you along the way. (you could start
with
etch/binding-csharp, too, though it is less well documented.)

step 2 is to ignore the compiler. the compiler is a finishing touch which is
best left for
last.

step 3 get oriented

the runtime is the first order of business. the runtime is loosely organized
into 4 parts. the
best approach is to port them in a certain order by starting with the
existing code (i'll
assume that is java).

etch\binding-java\runtime contains the source and the build.xml. ignore the
ant file and
let's look at src. src is composed, in the maven pattern, of main and test.
there are many
unit tests in test which will help you with your port. their structure
mimics main. diving
deep in main we get to
etch\binding-java\runtime\src\main\java\etch\bindings\java:

msg - message model
support - the glue between messages and transport
transport - realization of support interfaces tied to actual protocols
util - general purpose utility stuff

msg is the foundation of the message model of etch. these are the primary
classes
used to model messages for the support and transport layers. the compiler
mainly
generates code which uses these classes. they are all very simple, and
porting them
should be fairly straightforward in a modern object oriented language with a
garbage
collector. i'm told by rene that objective c fits the bill nicely. however,
his initial investigation
yielded a number of naming conflicts which have to be resolved. it is
beneficial to keep
class and method names as similar as possible to other languages to
facilitate patches,
etc.

when the compiler generates code, it uses the classes from msg and support
to express
the implementation of the service.

step 4 dive in

from here it is, currently, a feel your way along operation. different paths
are possible:

option A:
1) port msg and its unit tests. don't stop until all the unit tests work.
2) repeat for support and transport.
3) port classes and their unit tests (if any) from util as you need them.
there is a lot of stuff
there, much of which you won't need.
4) this technique has been used successfully to bring along csharp, c, ruby,
and python
bindings from java seed (though csharp is the only binding which is
finished). this works
if the target binding is largely similar to the seed binding.
5) test your service against an existing binding.

option B:
1) rewrite your compiler generated files from your Foo etch service into
your target
language. start with the client and then only generated files needed by the
client. ignore
the listener / server components.
2) port the classes from the seed binding to the target binding needed to
make the
sample client compile and run. port in layers (msg, then support, then
transport), stubbing
out stuff below the current layer in order to facilitate testing. (so, port
msg while stubbing
anything from support and transport; only port stuff from util as needed.)
3) as above, port support, then transport, again only pulling stuff from
util as you need it.
4) you should be able to use your client implementation against the seed
binding
implementation of your server.
5) repeat this process with your service listener until you have your entire
service implemented
in the target language.
6) you should be able to use the seed version of your client to talk to the
target version of your
listener. of course, target versions of your client and listener should be
able to talk.
7) pat yourself on the back, proof of concept complete.
8) now finish per option A.

I suggest option B.

the unit tests present at each level will be a big help. do not ignore them.
eventually you will
get to the part about tcp sockets and wire formats. those have tests, too.
binary tagged data
tests include injecting actual packets and testing that they decode
properly, and vice versa.
there is also a document describing binary tagged data which is not yet
posted. that's on my
todo list, but you won't need it for a week or so...

certainly check back here for help or advice or just to let us know how it
is going. i personally
want to see an objective c binding so that iphone developers can start
playing with etch...

scott out

Shahed Moolji wrote:

Hi,

I am wondering what would be involved, if I want to implement Etch in
a new language (say Objective-C)

I guess the 2 parts are :-

1. The language bindings

I believe this can be done by customizing the current compiler
with velocity templates for the target language

2. The runtime implementation.

This would reqire an indepth knowledge of the RPC messages and
data serialization for each supported transport (TCP / TLS).

If these two steps are correct approach, then are there any docs
available that document the protocol spec ?
Also, are there any test tools available to check interoparability ?

Thanks
Shahed.
_______________________________________________
Etch mailing list
Etch@developer.cisco.com

_______________________________________________
Etch mailing list
Etch@developer.cisco.com

Attachment not added (content type not allowed): "sccomer.vcf"
_______________________________________________
Etch mailing list
Etch@developer.cisco.com