An essay about expressiveness, behavior and conciseness in code

This week I came across an interesting tweet, which stated that a code is read more often than it is written.

It can only be true, as any tautology can be. It's like a book, right? Written once, read many... until errata come in the game, or new editions.

That was my point when I replied, by trolling (I admit) that Java is becoming a pain in this area... because of its verbosity.
Hopefully, Nicolas Frankel is clever enough to continue on the discussion tone rather than entering an infinite war where none can win...
So we took the decision to express our point in a cross-referenced blog where you'll find your own way of coding.

Reading a code

Before starting, I would like first to wonder why would we have to read code?

From my point of view, there are several relevant cases:

• learning a new library, code express more than doc that's why it must be opened
• improving it by adding new features, in that case, we need to have the big picture before starting any hacks
• debugging

So what I'll claim based on these three (more? poke me!) cases, is that

• the two firsts can be eased by a clean code describing a behavior using the right level of conciseness but keeping the expressiveness at its best.
• the later can be a real pain when the conciseness of the code has been badly chosen

Some words about conciseness and Java

Here I'll brush quickly some additions that Java received over the last couple of versions, a lot involved conciseness.

Let's take List and the for-loop.

See? Everything is about making the code more concise without removing any logic nor features.

However, everything is not that shiny, mostly when you need to add more logic, or let say when one need to enhance the behavior. Can you read the following two examples easily?

Don't know about you, but my feeling is that the behavior is not well represented, and thus we need to touch our in-brain JVM to catch it up.

The problem is that the conciseness introduce at the language level is not flexible enough to express advanced workflow, like early termination or filtering.
Now let's look at a Java 8 version of this example:

The same code, less lines... but is it really the interesting feature? Conciseness? Really!?
I don't think so, the most important part is that the behavior can be catch by combining two simple behaviors:

• filtering
• mapping (transforming)
• limiting (taking)

Another good fact is that using the second version, the behavior can even be easily testable by simply testing the the predicate (dereferenced from the Test class) is returning the right value.

So there is no need to test if an array will be filtered correctly, it's asserted by the library.

Also, the behavior is easily extendable since now the filtering is no more explicitly hardcoded in the behavior -- however we're free to instantiate a specific behavior introducing partially applied method.

Now, it's true that we have to agree on an API and everybody must know it. But I think it's always the case when you're creating an API, you're fixing names and concepts.
The sad fact, in Java, is that they are trying to reinvent the wheel by renaming well know behavior like limit (take), substream (drop + take) in the Collection API. That's why semantic is hard when someone is creating his own taxonomy/ontology of concepts.
Moreover, they took the opportunity to leave some noise, like the stream() call (probably for backward compatibility) which result in a collect method (which is also used to reconstruct a List using a Stream).

Conciseness Expressiveness and Scala

In this section, I won't expand myself into much details about Scala, but I'll just try to show some advantages of the expressiveness that Scala offers... for instance, even implicits (in the call-site) are explicits (in the definition-site).

So I'll just mention two (among many) features brought by Scala which is really missing in Java, both being related to expressing a behavior based on an implicitly available context.

Implicit parameter

In Scala, a method/function can have several parameter blocks, like so:

def add(a:Int)(b:Int) = a+b.

Ok, fine, but the very last parameter block can be declared as implicit, this way:

def persist(user:User)(implicit s:DBSession) = ....

What an implicit parameter block of parameters is, is simply a bunch of parameter that the compiler should be able to find within the compilation stack. And if it can find, in a deterministic way, such parameters they'll be automatically provided at the call-site. An example?

What has just happened there? It's rather simple, when the compiler will fall on the persist call, it'll see that the last block is implicit and not provided so it'll search in the actual scope if there is an object matching the required type. In this case, it finds session.

Again, I don't know about you, but I prefer this at what Hibernate (for instance) does... which is declaring by injection by something using an annotation or what not a session somewhere. And if something goes wrong? I hope you have integration tests, because it'll be check @runtime.
In this case, the compiler will blow out if it cannot find a session object, that's it!

So it's explicit at one single place, and will be implicit at all call sites. And I think it's cool/powerful to have such duality.

For-comprehension

A for-comprehension in Scala is very similar to a for loop in Java when dealing with sequences, however it's more than this. But beofre going futher, here is what is possible using lists:


For those of you having already tried to use the Future API of Java should just find the following really pleasant because it'll be terse and straightforward to chain futures, without pain but with some implicit meanings...

What's going on there? We fetched a user in a future, when it has been fetched we fetched each friend, one by one, within a sequence then we yielded both results in a tuple.
The resulting fut variable is yet another Future that will hold this later tuple if all the fetches successfully returned, otherwise everything fails!
Afterwards, it's still allowed to adapt the contained tuple as a new tuple... Note that we're not dealing with the value yet, we're just describing what has to be done when the result will happen... or not.

I'm not enough courageous to write the code using the Future API in Java, it would be too painful for me, too many brain gymnastic for nothing... and I didn't even talked about the number of  bugs It'd be prone of.

Oh yes, one last thing on this for-comprehension, fetch should have an implicit parameter, the storage session/metadata access and the Futures need an ExecutionContext instance to be executable... but it's not the role of this piece of code to create or even pass them!

Flaws of conciseness in Scala

Mainly the flaws are raising when the code tries to be concise at a such level the the expressiveness itself is penalized. 

For instance, sometimes, I like to tweet implementation in 140 characters like this one; this is just fun to do... I won't expect to have such code dropped as is in a project unless there are a lot of reasons (I can't even image a single one).

Why? Because it breaks all good conventions that are becoming the de facto best practices in Scala. For those interested in, there is a very good keynote by Martin Odersky on that.

The most important is probably to track the status of the types chain by decomposing a multicalls line in several ones, the very next one being to not overuse the wildcard underscore for inline function.

So the code in the tweet can be migrated this way from:

To:


What you only need to understand are these 5 concepts:

1. map
2. flatMap
3. groupBy
4. mapValues
5. flatten

Which one couldn't you guess correctly? Make a guess than look at these rough and limited explained behaviors:

1. List<A> => List<B> OR Map<K,V> => Map<L,W>
2. List<List<A>>  => List<A>
3. List<A> => Map<K,List<A>>
4. Map<k,V> => Map<K,W>
5. List<List<A>> => List<A> OR List<K,V>

I kept it cryptic on purpose because it has been explained so many times on the web!!

But also not that the readable code (that could even be more readable by still using non-verbose tools) is, with comments, less than 2 times taller. But it would require hundreds of line of Java, which I don't even have the energy to write... Or you need to convince me that I should...

Last note: just don't override, nor create operators that haven't a widely known semantic. Haskell is a bad example, even if this language is awesome!

mid-2013, my Spark Odyssey... when Akka sparks Yahoo! finance

This entry is the second of a series that relates some work/test I'm doing using Spark. All information can be found in the very first post of this series.

In the previous post, I've explain how easy it is to consume a Twitter stream using Spark, and seamlessly easy how to attach a sentiment score to each tweet passing a given condition (read: filtered).

In this one, I'm going to tackle another fold of my test, which is fetching the current stock information for a given list of companies.
These information will be provided by the Yahoo finance API...

...well, hum, not exactly an API. Actually, Thanks to @kennyhelsens who shown me this "hack", we'll use a Yahoo! URL that produces a CSV file containing real-time (mostly) information about companies' stock changes. Information are really hard to grasp for this URL's parameter however you can find some out there.

Storyline

In this series, we're trying to uses real-time streams rather that batch processing to give a real-time fluctuation of the health of given companies.

But we've just seen that we're going to consume CSVs that give a snapshot of the stocks changes.

A common solution to transform a static API to a stream is to poll it and to create a streams by appending the results one after the other.

Here, we'll try a pretty simple solution for that, we'll delegate to Akka the task to consume the CSV, to parse it, to split data by company, and to modelize each and finally to push everything to a DStream.

It's rather straightforward and we'll see that it's even simpler that we would expect since, Spark is already playing a good game with Akka by integrating out of the box an API to "consume" actor's messages.

Acting

In this scene, there will be several actors each playing its own roles (sorry Peter Seller's fans -- like me btw).

Here they are:

• scheduler: this one is provided by Akka out-of-the-box and is able to send a "given message" to a given actor at a given interval.
• feeder: this actor will simply fetch the CSV and produce the data to be sent to the below actor.
• receiver: this one will receive a message whenever a data is available for a given company.
• parentReceiver: this one will be created by Spark itself using the StreamingContext and is able to push the received Data to the right RDD in its holding DStream (see ActorReceiver).

Now the scenario:

Sequence from the scheduler to the DStream (current RDD)

Does that make sense? Let's have a short overview on the implementation.

Scheduling the poller

First we have to create a scheduler that will poll every 500ms the server... well using Akka and the separation of concerns and Message Passing Style, we'll just create a scheduled action that passes a message to a given actor:

But even before that, let's prepare the Akka field:

There are a plenty way of creating ActorSystem with Akka, but Spark is preparing some stuffs for us to create one that will respect some of its own internal conventions. This helper is available in the `spark` package, naming AkkaUtils; however it's package protected... That's why I've created my own utils to create the ActorSystem under the spark package -- I'm bad sometime, but I'm so often lazy :-D.

Actually, the Spark utility for Akka will create an ActorSystem using a host and a port for us, that'll be really helpful afterwards... because everything in Spark must be serializable and thus we cannot so easily package an ActorRef (needs an ActorSystem, ...). So that, we'll use URL to create refs rather than using actors picked within a closure in a Spark action.

Recall: that's how Spark manage to recover from a failure, it's take to lineage of the RDD or rebuild everything up from the initial data (not possible in Streams...). But specially, the actions (functions) are dispatched to machine where data reside. So the lineage contains the functions, and a function might contain ActorRefs!

Now that we have an actorSystem available, let's create the scheduler (see last section, Drawbacks, for more info about problems with that methodology):

Nothing relevant to say about that, only maybe the definition of FeederActor...

Feeding Spark with Yahoo data

Here is coming the feeding phase, we won't see how the CSV is read and parsed however we'll see what going on with the messages flow.

What's interesting in this actor is two-folds:

1. it can receive a message containing two data: 
1. an actorRef that it will hold in its state: this one will be the receiver
2. a list of stocks to follow (by building the according call to Yahoo)
2. when consuming the data and producing the data by stock, it sends them one by one to the above actor

Actually, this feeder has two responsibilities which are fetching the data and pushing it to Spark.

But how is this receiver constructed and passed?

Receiver -- our spark stuff

The receiver is actually an actor that will be created internally by Spark for us, for that it must respect yet another trait, Receiver. And a Receiver can only be an Actor (using self definition)!

Four things to point here:

1. this actor is well-defined by extending the Receiver trait, which provide the pushBlock method.
2. it holds a reference to actor that it, it-self, create based on a give URL -- an actor URL... this is where AkkaUtils came handy! This Actor will be the feeder actor.
3. in the hook preStart we tell the feeder actor that this ActorRef is the Spark end-point where data must be sent to publish in the stream.
4. when data arrives, we check in the cache if it's already there (see below), if not we call the Receiver#pushBlock with it.

Why do we have to cache? Actually when the stock market has closed, Yahoo will continually return the last change before closing... so, here, we just avoid telling our health checker that the mood is monotonously increasing or decreasing (or even just flat if zero).

The pushBlock method is the clue to publish to the underlying DStream managed by Spark, actually it will wrap our data into an Internal representation before telling the parent actor to use it: context.parent ! Data(y).

But what is that context.parent?

parentReceiver -- not our spark stuff

This receiver is totally managed by Spark, and so I won't dig into more details than just saying it's being created underneath when doing the following:

Fairly simple, yes, it's true! Actually, the StreamingContext we created at the very start of the project contains everything to create DStream based on Akka, all it needs is a reference to a Receiver (defined in the previous section).

What is this actorStream doing is creating an ActorReceiver (with a Supervisor and blahblah) which will itself create and manage a worker for the given Props, but also it will hold the reference to the current RDD in the DStream.

The worker will be able to send him messages of the form Data(d) using pushBlock, and which it'll push to the DStream.

Concise huh? If only the document was telling us that, I wouldn't have to dig that deeper in the code... the good side-effect is that now I understand it pretty well. 

So, I hope you too!

Where are we?

Up to here, we've created a DStream of data for each tweet paired with 'its' company, but also we have now some info about what the stock change of them.

What will happen in the next blog is that we'll create a multiplexed stream for which we'll compute a value for every window of 60'.

Drawbacks of the/my methodology

At this level, I can see two problems.

The scheduler

Using a scheduler is bad because we cannot ensure that each message are processed sequentially by Spark. So it would say that we push data to an RDD that correspond to the time where the whole CSV has been processed but also the messages sent by the actors!

I put this aside for the moment since I'm not building a real-life project (no-one will use this tool to take decision on the marker ^^).
Moreover, mathematically, let say at the limit, the value will be correct: reducing an amount by 2 at t⁽¹⁾ or at t⁽²⁾ is not really problematic, it's like inverting two ints in a list that we're summing over.

The consumer

I didn't shown its current implementation because its very simple and bad, that is I'm simply opening a connection on the URL, consuming everything until the end before applying the process...

That says: it's blocking and non-reactive.
Also, That says that we loose the power of parallelism since the Akka message dispatcher will have to wait a CSV to be completely processed before handling the next one.

However it's just an implementation detail that can be easily worked around without modifying the solution.

mid-2013, my Spark Odyssey...

Short intro

In this post, I'll talk about yet another technology-tester project I've worked on these last days: Spark.

Long story short, spark is a handy distributed tool that helps in manipulating Big Data from several sources -- even hot streams.

TL;DR

In the first part, I'll put the context in place.

1. I'll start with a gentle bunch of words about Spark and Spark-streaming.
2. Then I'll give some bits about the project I've realized so far, and why.

The second part will discuss some implementation details.

1. Setup: spark streaming
2. Twitter stream filtered by keywords to DStream
3. Tweets' sentiment analyzer
4. Yahoo finance CSV data and the actor consumer
5. Actor consumer publishing on a DStream (back-ended by a dedicated actor)
6. Aggregation (union) based on a 60 seconds length window
7. Publishing the results using Spray

What're missing so far (among other business related things for instance):

1. cluster deployment...
2. ... and configuration
3. probably a storing strategy for further consumptions (Riak, ...?)
4. a web application that shows the results or configure the processes

Context

Spark

I won't discuss Spark that much because there are already a plenty of good blogs/articles on that. However, I'll give you just the intuition that you got the whole thing about it:

• Spark is a collection of functions that works on a Sequence (List) of typed data  --  Dataset = Sequence ; operation = Function
• Chaining these functions will define a workflow for your data  --  combination
• Each data will be processed only once by one or another Spark worker  --  Distributed
• If it fails for some external reason, the data can be fetched again from the source and the workflow to be replayed entirely  --  Resilient

Spark defines an ResilientDistributedDataset which is very different that the classical MR -- because it is distributed by essence, a workflow can easily be iterative and doesn't require intermediate caching/storage.

Also, spark as a sub-project spark-streaming that allows manipulating streamed data. The idea is very simple by defining a DStream as being a sequence of RDDs containing the data for a certain slice of time.

Oh, RDD and DStream are typesafe -- the whole thing being Scala code.

The spark-streaming project is really well integrated and can consume a lot of streams like Twitter, File System, Kafka or even an Akka actor.

Project

This blog relates a small project publicly available on github (spark-bd) that helped me catch the big picture of Spark, after having followed the mini course here.

The project is based on another working group I'm participating in. This project, simply called project 2, has been initiated and is still supported by the great Belgian Big Data user group (http://bigdata.be/).

The idea was to co-create in several workshops an application that given a list of companies to analyse;

• catches the Twitter stream and the Yahoo finance data using Storm;
• manipulates both by either applying a sentiment analysis or simply the changes in price;
• aggregates by window of time both information for each company
• gives (constantly) the information about how is going a given list of companies on the market.

Fairly simple and really funny... the project is still ongoing but you can check its progression here.

It was really funny but I wanted to see how my feeling would compare by writing the same application using my actual preferred language (Scala) and the next technology in the Big Data field I was eagerly looking at (Spark).

IMPORTANT: since if I'm not there yet to make a real comparison (a niche for another blog, later on), I'll only give an insight on my actual work.

Internals

Setup

The setup is very minimal for a Spark application. All we certainly need is reduced to two things:

1. an SBT project declaring as dependencies Spark and Spark-streaming.
2. in a main class/object configure the spark streaming context.

The SBT project only requires an build.sbt in the root folder that contains these lines only:

Where the only lines that are under interest are the two last ones.

Then we can create an object that will start the spark context.

Here we just asked Spark to create a streaming context on the local environment, named Project2 and that will slice the DStream into RDDs of 5 seconds duration each.

Afterwards, we asked the context to start its work... which is nothing up to now, since nothing as been added yet to the context.
So let's do it.

Twitter stream

Here we'll consume the tweets stream provided by the Twitter's API. For that, we'll have to do only two things (a pattern emerge...?):

1. register the 4 needed information for an OAuth2 authentication to the Twitter API (so don't forget to add an application to your Twitter development account).
2. add a tweets stream to the spark context

Actually, the streaming context as convenient methods to add a twitter stream.

For that it'll use the Twitter4J library (it's definitively not the best, but at least we can achieve our goal).

This library needs several keys to be set in the System.properties.

There are a plenty of way to do that, in my project I took something rather straightforward: I've picked the Typesafe config library and added those four configuration keys in an application.conf file, then I load all value in the related System property.

As you may (or not) see, the application.conf file will look for environment variables giving values for the required key/secret/token.

That's done we can start listening for tweets:

Dead simple, after having created a model for a company/stock, we used the configured keywords as tags for the twitter stream.

Behind the sea, Twitter4J will access the stream end-point with these tags, and Twitter to pre-filter all tweets based on those keywords (in hashtag, username, text, ...).

At this stage we've only added the stream, but we didn't yet said anything about how to consume them... So we can simply call print() on the stream to ask spark printing the 10 first incoming event in each RDD.

However, that's not how we want to deal with those tweets right? Let's adapt our stream to add relevant information for our use case.

Sentiment analyzer

Now that we have a stream of tweets it's time to read them and to give them a sentiment score based on a sentiments file. The code that reads it and that assign a score for a tweet is not that important here, but the code is here.

However, what we would like to do is to adapt our stream with this score attached to each tweet.

As said earlier, Spark is a bunch of methods on Sequences of data, so we can now assert that it's true. 

Look at what was done in the map, we picked each tweet, fetch the list of sentiment entries that match the text, and also we kept the whole original status for further computation. This results in a DStream[(List[Sentiment], Status)].

Since we only want those tweets that have at least one Sentiment that matches, we then filtered on the length of this matching list.

We combined both actions to get a resulting filtered and sentiment-analyzed tweets.

Okay great, but what we really want is to have such information grouped by Company/Stock right? So that we can compute the participation of the social network to the health of each company.

To do so, we have to group semantically (based on text's content and required stocks' keywords).

Note that it's not that straightforward because a tweet could relate several companies at once!

That's all! We're now done with the twitter part, we are now getting all Data that gathered the information of a Company/Stock, the tweet and its associated sentiment. Which opens doors for further aggregations/grouping/reducing.

Interlude

Let's now pause a bit, I encourage you to try your own stuffs and playing with the tweets.

For that, I also encourage you to fork/clone the original repo, where you can already start only the twitter feed and print everything in the console by running the appropriate command:

$> sbt

sbt> run-main be.bigdata.p2.P2 twitter print GOOG AAPL

This will run the start spark and filter the tweets based on the keywords associated to both GOOG and AAPL, whose tweets will be analyzed as said in the last section before being printed to the console.

WARN: don't forget to export the required environment variables

Next

In the next blog posts, I'll continue with the explanation on how use Akka to construct Spark streams and how to combine several streams into a single one.

This multiplexed stream will then be used to compute an aggregated data by a window of time.

Having these data in hand, we'll be able to create a Spray server that expose them into a JSON format, helping us creating a client web application that simply consumes the JSON messages.

Function1[-T1, +R] -- What the heck?

In this post, I'll try to cover an important notion of the Scala's Type System -- and a Java's Terra Incognita.

(generic) Type covariance or contravariance.

Ho yeah, I heard you... yet another post... right?
... right! but it seems that it needs a more gentle one, where Category Theory is left apart. Because, Category Theory must come when you want yet more fun, not to get basic concepts!

That's why we'll see what are these constraints in a more pragmatic way, using (important) examples.
Afterwards, I'll let you read the blogs using Category Theory ... if you like.

I thought that taking the problem from the Java side might be easier to explain, thus the following parts

• Problem: Assignments failures using Java Generics
• Solution: Covariance Type in Scala
• Problem: Unuseful generics for method parameter in Java
• Solution: Contravariance Type in Scala

Problem: Java tells me that Kids aren't Humans

Since Java 5, as Java developers, we were really interested in the new breaking feature that was added: Generic Types.

And it was rather interesting enough thanks to the java.util.Collection API, but also the syntactic sugar for (A a: as).

However, we also felt really quickly on the problem that the hierarchy of Generics doesn't span to upper level type (as explain in the Java tutorial).

Let's clarify the situation with an example.

First we define a model with an classical hierarchy:

Okay, with this setup, one might want to create a list of kids and assign it to a list of human. Why? 

1. because they are...
2. because they grow!

Recall: in OO a model (should) represents the domain.

Fine! Here is several tries:

See? You just can't in Java! Because a ArrayList is not considered to be an instance of ArrayList. This will impact all your further algorithms, unless you use the tricky tip to anonymise your type and to constrain it with a the Human type as bound.

At least we spotted the real problem... Let's see the solution that Scala offers.

Solution: Covariance

In Scala, a type can be defined to be covariant to one (or several) of its generic type (roughly speaking).
We can simply check the Scala's List definition: 

sealed abstract classList[+A]

Here, we tells the compiler that List is covariant with its generic type A.

In other words? List will vary accordingly with the variance of A!

And life gets simple (respecting the OO concepts):

Following the type of generic type is pretty cool and useful, but there are cases where it won't help... let's jump to the next section if you don't believe me...

Problem: Java tells me that Kids aren't Human and Integer isn't a Number...

To illustrate this case, we'll create two helper type that define no-arg procedure and an action (a function) with one argument.

Simple as simple.

We also created to action definitions, one that maps a single integer onto a single human, and another one that maps a number onto an adult.

This will help us defining an higher order operation that, for instance, can map a list of integers into a new list of humans. For that, we'll create the ListMapper class (see below) that will map an input type I to an output type O.

Using this ListMapper, we'll try to convert:

• a list of integer into a list of humans 
• a list of numbers into a list of adults

We can easily imagine that the logic should remain in the actions introduced in above. Right?

Let's see how it goes in Java thus:

huh?

It seems that the mapper from Integer to Human is only able to deal with an action that takes exactly these types as input and output resp.

But, it's weird, because something that works on Number should be able to handle any Integer, isn't it?

And, if the returned object is an Adult, it should be ok to assign it to an Human? True?

For sure, but for the same reason as before, Java is not able to deal with such multi-levels hierarchies. crap...

Before going ahead, one might have noticed that the inheritance hierarchies are opposites:

• Number is extended by Integer
• Adult extends Human

But the mapper should be able to accept both pairs of input/output type! damn... the action type doesn't seems to vary the same way for input as for output!

Indeed, an action must be contravariant on its Input and covariant with its Output.

Here we are.

Solution: Contravariance

In the example above, we introduce action that acted as function taking a single argument. And those actions had to behave differently on the type of the input and the output.

Let's check this out in Scala, looking at what looks like such function:

traitFunction1[-T1, +R]

Bloody hell, a function type in Scala will vary in the opposite way as for the covariant types. That's why a minus sign is used on the T1 type.

Simple said: a function F will be extended by any function that respect either or both of these cases:

• its argument is a super-type of the F's argument.
• its type (output type) is a sub-type of the F's type.

Let's see everything in action now:

That was for illustration purpose because the types weren't optimize for maximal genericity, however we can see that we were able to use a function that takes an AnyVal argument and results into an Adult object where a function from an Int to a Human were used!

Conclusion

A lot for nothing? Maybe?

But it's a question that is very common for Scala newcomers. And I think it's one of the most important ones, thus it must be answered in the more precise and accessible way.

Furthermore, it also demonstrates that Scala is even more Object Oriented that Java can be.

--

Please let me know, if it can be more clearer, this blog is meant to be organic, so any comments, concerns, helps are more than welcome

MongoDB and Play2 at ease: using Play-Salat plugin and Embed Mongo plugin

A while ago I've blogged about MongoDB with Play2 using Salat, here.

This post was describing how to integrate Salat easily with Play2 and gave some advice on actions to care of.

Preface

Play 2 gained in popularity and an amazing plugin as emerge for this purpose: play2-salat.

This plugin offers a lot of configuration to hit running instances including replicasets! It integrates very well by applying the advice I talked about in my post, but not only. It defines binders to enable us using Casbah stuffs in our routing and action definition (ObjectId, and so on).

This post is not dedicated to explain how to use it, I'd recommend you to browse the project page (play2-salat), plus the wiki that points to relevant URLs.

Goal

This post is dedicated to developers teams that follows (or not...) the convention of Continuous Delivery, especially the Single Command Environment pattern. That is, the environment must be set up in one single command... in Play2 => play run OR sbt run

Context

Create an application that uses MongoDB as (one of its) persistence backend service, use play2-sala to have access to the `ORM` for our object and easy collections connections.

When runnning in production, of course, a MongoDB instance runs somewhere that can be configured (or a replicatset).

But in Dev?

Embed Instance

When another developer is cloning the related repo, knowing that it's a play application, he's best will would be to enter the directory and launch the application. > BANG <

No running instance...

So I created a Play2 plugin that uses this amazing work which retrieves a mongodb installer, installs it and enable us to launch/stop it... Keep in mind that MongoDB is not JVM based!

Adding this plugin to the application, setup the dev configuration to starts an embed MongoDB and Play2-Salat to target it, will gives the satisfaction to our developer... Moreover if he is a Designer (the only kind of guy that add values to any app ^^) who don't care about MongoDB, at all!

How To

Add the plugin dependencies (used in PlayProject):

     //MY OWN REPO where is deployed the following plugin

     val skalaCloudbeesSnapshots = "Ska La SNAPSHOTS" at "https://repository-andy-petrella.forge.cloudbees.com/snapshot/"

    //THE NEW PLUGIN => EMBED

    lazy val embedMongoPlayPlugin    = "be.nextlab" %% "play-plugins-embed-mongodb" % "0.0.1-SNAPSHOT" changing()

    //THE WORTH ONE

    lazy val salatPlayPlugin         = "se.radley" %% "play-plugins-salat" % "1.0.8"

    //DECLARE the deps

    val appDependencies = Seq(

        embedMongoPlayPlugin,

        salatPlayPlugin

    )

A bit of configuration (application-dev.conf)

embed.mongodb.start=true

embed.mongodb.version=V2_1_1

embed.mongodb.port=27017

mongodb.default.db=meinGot

mongodb.default.host=localhost

mongodb.default.port=27017

And the most only thing that requires a bit of explanation (in conf/play.plugins)

600:se.radley.plugin.salat.SalatPlugin

550:be.nextlab.play.mongodb.EmbedMongoDBPlugin

See? Yes, the Play2-Salat plugin MUST be started AFTER the embed plugin... of course (what an explanation huh).

Code

The one-single-file-of-33-lines plugin can be forked here.

---

That's All, Folks!

Gatling-Tool in SBT or Play : Sample Projects

Content

This post is a direct follow up of this one where I introduced a bit what I did in order to integrate Gatling in SBT and in Play2.
Where this post was more about bits and bytes necessary to accomplish the task, this one will talk about how to use this mess.

Gatling for SBT

Now, that we've a dedicated SBT plugin in hand we can create a sample project that uses it (I already created one here).
In this new project, we'll need to create a file plugins.sbt which will contains the reference to the gatling-sbt-plugin.
Actually, it's the classical way to add a plugin to a SBT project (and the easier and the "semanticest").
We're now prepared to configure our build by using the pieces provided by the plugin.
At the end, we'll be able to write a first test and launch it.

Project Build

First of all, we must create a directory for your project with this basic structure:

Looking at the structure, we can see a Build.scala that will define our project build, a build.properties that defines the SBT version with a single line : sbt.version=0.11.3
And the latest file in the project folder is plugins.sbt which... declares the plugin.
Let's put the Build.scala aside for now, and have a look at the content of the latter. 
Self descriptive isn't it? Yes, we've just told that SBT has to use our gatling plugin... that's all? Not yet.
Actually this line will provide us everything that have been declared in the plugin, such as the Gatling configuration keys, the command, and the basic SBT settings. 
Now, we're gonna us them within our Build.scala. 

Before going in specific details, note that we had to add my own repo (hosted at Cloudbees) in order to fetch the project... but you could also use the URI fetch provided by SBT... 
What are really important to point out is the allSettings declaration and the import of the GatlingPlugin. 
At first, we reuse the default one by using Project.defaultSettings which is being appended the gatling settings, using Gatling.gatlingSettings... that object comes from our plugin! 
This will add all relevant keys for the "gatling-test" Configuration with default values. 
At this stage we've almost finish our build definition... All that remain (and shouldn't... have to figure out any help is welcomed?) is to add two things:

• Declare the test framework to have access to the gatling classes and the custom Simulation :

  "be.nextlab" %% "gatling-sbt-test-framework" % "0.0.1-SNAPSHOT" % "gatling-test"

• Declare the command :

  commands ++= Seq(gatlingTakatak)

After having used everything in your project (don't forget to add the GatlingTest configuration...) you're so close to write your first test.

Gatling Conf

This section will cover the other folder src wherein reside the classical main and test folders.
But there is an other one, gatling-test that is unknown at this stage.
In fact, this folder will be the root for our Gatling tests, holding the configuration file (I provided a basic one in the sample app) and the future simulations that you'll write.

The first is dropped in conf/galing.conf, where the latter is simply simulations.

Write your first test

At this stage, I'll make an assumption that you either know the Gatling api... Nah, the Gatling DSL is sufficiently self descriptive (otherwise check the wiki).

As you can see, it's very very easy (that's why I like Gatling-Tool).
All we had to do, since we're benchmarking google, is to implemnent the apply method of a simple scenario within a class (with must have an empty constructor... a limitation for now) that must extend our custom Simulation (yeah! I know ... I'll choose another name for it that's why GSimulation is needed).
Note the dummy implementations of the interceptors whose aren't needed here.
This Simulation must be located somewhere under the simulations folder.

Now, we can launch it and see where are stored the results.

That's enough boy...
Let it shot the server and generate the result.
 ...
...
Done? ok.

Now, that all bullets have been shot, you can go in this folder gatling-project/target/gatling-test/result.
See? True! Every run will create a specific folder under this one, named run-${scenario-name}-time.
Make some jump in it and locate the index.html file, open it in your browser and see the magic happened. 
NB: check this page for information about the reports. 
So far, so good. But we're writing web apps and we want to stress them... we all know that the google server won't crash (that easy). 
Let's move to Play2 (or you can write your own app w/o Play2 using Spray or another Java framework maybe, then mimic the following).

Gatling for Play2

For this part, I also wrote an application that makes use of the "plugin" introduced in the previous post.
It's quite complicated, not yet polished and might be a good topic for a future post, because I tried to make an application based on Event Sourcing with some workflows and usages of pure functional structure like Lens, Writer, State, Monad and so forth.

But let's stick with the actual topic, if you fork the project and browse a bit its build configuration you'll see that it is near from the one we just talked about. But still that some particularities are presents...

Definition

First of all, since Play2 has its own convention for sources folders (app, test under the root) we'll redefine those folders for gatling as well.
In order to do that, we must adapt the following configuration keys:

Actually, yeah, we've just skipped the "src" folder. Anyway. 
So far, so good... 
I heard you... 
But, do you known that we're already done? 
Allez, let's write a test.

Write

If you started from scratch (without forking the sample app) you can now stress test your index page that will render the default welcome page defined in Play2.
See the below Gist for that, or its follower for a more complicated one using the app.


Look what I've done Dad! 
In the Simulations, all we had to do is to simply declare an available port on which the framework can start the server, and its needed FakeApplication to have access to the routing. 
Then when we had to build a request path, we've just had to do the same as in our templates, that is, use the routes object under the controller package. Amazing. 
Furthermore, we can also reuse our type checked Form to build our feeder. Waouw. 
But, at this stage, there still some boilerplate (parameters). Looser.

Launch

And now, it's time to stress test our server and see how it'd behave on production. (Ok, that's not 100% true, since Gatling is running on the same machine...)

For that, just do the following:

Wait. Again.

Now, go to you result directory (the path hasn't changed). Open in your browser and let the shine comes in.

Wrap Up

At this stage, with small and few injuries we can use Gatling-Tool without having to install or trick stuffs in order to have information about our web app.
This by using either a full SBT application or a Play2 based one.
I know that there still a lot to do, but at least the basic features are there, and I'll enjoy anyone forking it, scrapping it if necessary and let me know.

Last words

I hope you liked these posts, otherwise I'd like to apologize because I did all projects and the blogs on the train, half asleep every morning and evening, back and forth from my actual mission's workplace.

Now I'm gonna sleep in this train...
Oh no, I can't...
I've to write my book now.

Gatling made easy for SBT or/and Play2

Forewords

These last few weeks, I took some time to understand a level deeper how SBT works and what it can provide. Since this post is not related to this learning trip (which was along existing blogs and wikis), I'll jump directly to the idea this new understanding gave me.
A while ago I started (and paused) a work on Gatling-Tool to have it integrated with Play 2.0 (see these posts here and here), this work has been refactored to better integrate with SBT.

What we'll find in this post

In this post, I'm gonna give you all tools in order to stress test you Web based application, either built upon Play 2.0, either only using SBT. By directly starting writing your tests rather than having to configure stuffs or to start others... So this post is composed of two parts:

• How I built these tools (can be skipped -- definitively)
• How to use these tools (it's probably the useful part)

Gatling shot 3 aside projects... for goods

This part is related to what was necessary to have gatling-tool easily integrated with SBT, and after with Play2.

Gatling SBT Test Framework

This project (on github) has a sole and simple goal, to implement the Test Interface (see) used by SBT to integrate new Test Framework (like was done for ScalaTest, Specs, JUnit, and...). This project contains only 2 classes and one convenient trait.

GatlingBootstrap

This class is one of the helper that avoid boilerplates in the future, because it starts the GatlingConfiguration stuff needed by Gatling-Tool to execute. This class requires two things "the path to the gatling configuration file" and "the path to the folder where will be stored the stress results".

Simulation

This trait is another tool in hand that extends the com.excilys.ebi.gatling.core.Predef.Simulation provided by Gatling in order to add it two interceptors whose are pre and post. Those interceptors will be really useful in the future, in order to start/stop a server for instance (e.g. FakeServer in Play2).

GatlingTestFramework

This class is the real processor of the project. Actually, it's also the implementation of the interfaces declared by the SBT test framework which are Framework, Runner2 and some Fingerprints. Basically, what is done there are those tasks:

• Create the gatling bootstrap based on sys properties
• Declare fingerprint to discover Stress Test based on the parent class being be.nextlab.gatling.sbt.plugin.Simulation
• Create the stress tests instances (only classes are handled for now) by reflection
• Call the pre interceptor
• Execute the stress test using the gatling api
• Call the post interceptor
• Generate the reports using the gatlin api

Gatling SBT Plugin

This project (here on github) aims to provide the basics to use the test framework easily within an SBT project. 
Its intent is to have testers able to write stress tests directly, just after having imported the module in SBT. 


It basically provides an SBT Configuration (gatling-test) that extends the SBT's Test one and one Command (takatak). The other goodies are settings which are common for all such stress tested projects using gatling. 
Those settings are essentially the conf file path, the result directories, the libraries (gatling ones principally) and so on. The last important thing that it does is to add the gatling test framework to the already existing list of test frameworks supported out of the box by SBT.

Gatling Play2 Plugin

This misnamed project (no more a Play2 plugin, neither a SBT one... but it's ok for now) brings only one little thing. It comes with a base implementation the custom Simulation declared in the Gatling Test Framework.

This base implementation is Play2 specific while creating a fake server and starting it in the pre interceptor, stoping it in the post one.

This will ease further tests in the Play2 environment.

Save Point

So far so good, if you read this you can now navigate the small projects that were necessary to do what we'll discuss in the next blog...

This one is already too long...

Type-safed Composable Interceptors in Play2.0

This post is just a quick follow-up of this post, which introduced my latest utility for Play 2.0.
I recommend you to have a quick overview on it.

Type-safe composition of interceptors : Premises

Briefly, we'll just see how the first "future work" as been addressed. That is, avoid boilerplate for interceptors composition.

A quick recall, when we had to compose such Interceptors we had to take care our-selves of the validation results and the combined result (tuple or whatever).
The real problem that is under the sea here, is that tuples are not so easily generalizable (no append method, roughly).

So I decided to use the Shapeless library (thx to @milessabin).

Shapeless has an amazing core structure that enables type-and-value chaining (somehow). The HList type is a kind of list but each element is one value and one type. For instance, it has the head value and the Head type on top of the Stack. Here is the kind of stuffs that we can do with Shapeless:

val | = false
val thisIsNotAPipe = "this" :: 15 :: false :: "a" :: | :: HNil


> thisIsNotAPipe: shapeless.::[java.lang.String,shapeless.::[Int,shapeless.::[Boolean,shapeless.::[java.lang.String,shapeless.::[Boolean,shapeless.HNil]]]]] = this :: 15 :: false :: a :: false :: HNil

>Type-safe list of stuffs<

In Action...

While trying to use HList generically I had problem with the implicits that are needed to prepend two lists, but StackOverflow has brought me the answer here.
I'm not gonna tackle here how I did, by I'll demonstrate what is now possible with the new composition functionalities added to Interceptors.

First of all, I had to declare an Interceptor for HList and the related implicit conversion. After what, I added three methods:

• hlist this method on Intercept is able to convert a classical one to a HList one
• ~::~ this one is available to compose any interceptors that is not defined using HList with an HList one. It will create a new Interceptor with the new composed HList as result
• ~:::~ this one enable to compose two Interceptor defined with HList, the results will be the concatanation of the two HList.

Note: concatanation of HLists preserves the type sequence, actually we can see that as if it concat two lists of values and two list of types.

Let's see how we can deal with them:

Easy no? Combine interceptors and use compile-time type-checking to validate the required kinds of items.

Without boilerplate

An Attempt of Play2.0 Action Interceptor

In this post, I'm gonna introduce a piece of code that I laid with the help of two stuffs.
The Security object provided by Play 2.0 out of the box and somehow the Secured one provided by the TypeSafe's plugin util.

The idea here is to leverage the actual functionality wich only enables to provide one Option[String] as result.
What I wanted at first is to satisfy my use case, which is : a lot of actions are secured and I need a username and its id. Where id is an Int.

Here we are, the actual Security and Secured don't permit me to have several extracted values, or to have an Int.

That's why I created this project : https://github.com/andypetrella/steal-play2.

Steal help

The idea is to enable any actions to be preceded by some interceptors that are stealing values either from the request, the cache, the database (or...) and set them as the parameters of a closure that outputs an Action.

But we must also care of the cases when something went wrong during the stealing.

Interceptor

This trait is the core of the solution, it defines:

• the stealing operation: a function that takes (currently) the request and outputs a validated output (Validation from ScalaZ)
• the err callback:  a function that takes the request and the failure (when computing the stolen value) and outputs a valid Result to send back to the client.
• apply: a closure that takes the result (not a Validation) and output an Action.

At this stage and looking at the code below (I've omitted the apply impl because not important here -- just wrapping and unwrapping), it's fairly simple to know how to define such interceptor.

I provided the simplest implementation of this trait that is a case class extending it by defining the two callbacks as fields. Like so:

So now we're armed to do stuff like that:

Where we defined two interceptors that takes a string from the cache and an Int form the cache also (just an illustration). After what, we combined them in an another Intercept using a for-comprehension.

So far, so good, now how to use such combined interceptor within an Action. Check this out:

Monoid

With the help of Monoid (from ScalaZ) and if the case permits it, I defined an implementation of Interceptor that append the successive results one after the another. Reducing and simplifying the composition. Like so:

Note: we used the classical case class. Thanks to an implicit def that wraps the Intercept into the Monoid implementation, using the TypeClass bound declaration. see

What will come next

The next step I've already started is to try using Shapeless to avoid the composition boilerplate. Things are ongoing. Stay tune for that.

One step further, I'll add another parameter to the steal callback, which will be the optional result of the previous computations of other interceptors. That in order to combine them at the function and result levels.

And probably, add all boileplate my self that creates the tuples out-of-th-box in the compose function.

That's all folks!

How Monad Transformer saved my time

Context

These days (this week-end) I wanted to put some work on a Neo4J Rest driver that I'm writing for Play 2.0 in Scala.
The only thing I wanted, actually, is to have the current embryo more functional. I mean I was throwing exceptions (frightening... huh!).
Since this driver is meant to be fully asynchronous (man... it's http, it MUST be) and  non-blocking (Play's philosophy), I was hardly using the Promise thing via the use of the WS api of Play.
This is the kind of thing I've got (briefly):

  def getNode(id:Int) : Promise[Neo4JElement]

Where Neo4JElement stands for the wrapper of all Neo4J Rest response (Json). Hence, it can be either a Failure response (Json with stacktrace and message), it can be a Node, or .... throw an Exception (br...) when the service crashed (f.i.).

Hmm, not so intuitive and goes against the functional paradigm that orders: "you can't ever introduce side-effects, boy!". An exception that blows in my face, is one side effect (head-aches, ...).

Diego Validation to the rescue

Validation is a very simple thing, it holds either a value, either an error...
Ok, why not just Either then. Actually, you're right but Validation that I took in the ScalaZ library contains a lot of thing very helpful for the purpose of validation. But if you worry it, just replace Validation by Either in your mind from here.
Now, here is the getNode signature:

  def getNode(id:Int) : Promise[Validation[Aoutch, Node]]

Isn't it more intuitive? For sure, you get back our relevant type in the signature : Node. Great!

So far, so good now what the heck is Aoutch : something that hurts... and what could hurt a runtime execution... exceptions yeah! Thus, Aoutch is just a shortcut for Either[NonEmptyList[String], Failure]. We can see that we represent with one single type an unexpected exception and a failure (missing node f.i.).

Monad

If you don't know what a Monad is, from here thing about a structure that can evolve in a for-comprehension (in scala it must implement flatMap and map).

Promise and Validation are Monads. And their used one over the another. But what really interests us is the leaf of the chain Node. That introduced some boilerplate code when you try to sequence actions like that:

See... yes we have to skip the first level (validation) to be able to work with the meaningful objects.
But still that we can extract some pattern... no?

Monad Transformer

The pattern that we can extract is kind of two-level composition. I did this composition my-self trying to figure out what we'll be possible.
It was successful but, I've have to introduce a new type and a new method, that was like a flatMap.
So I asked on StackOverflow ^^ (and it was my first question, yeepeee). You can find it here. So I want explain how I did, because the question explains it. But the real question was, is there a well-known functional construction for this problem?.
Thanks to @purefn, I knew that it was the case!

It was time to use Monad Transformer.

Monad Transformer

Briefly (and very roughly), a Monad Transformer is a construction that is able to transform a M[N[_]] into a P[_], where M, N, P are Monads.
I won't explain here how it does, because it would be long, but here is a good link (you've to understand Haskell a bit, sorry).
With the help of such transformer, you'll get you back the opportunity to use for-comprehension... with the wrapped-twice type as bound value.
Here is the the transformer for our Promise[Validation[E, A]]:

And how we can now link nodes:



Awesome! No! We get 3 async and non-blocking calls, totally typed checked and resistant to exceptions and failures... In 5 lines.

Future work

At the SO question, @purefn tolds me that scalaz 7 (snapshot) is defining (fully) this kind of Validation Transformer.
Why didn't I used it, yet:

• I'm trying to use not Snapshot (not a good reason, but still)
• In order to use ValidationT, I'll have first to create an instance of the Type Class Monad. Because the flatMap signature needs it in the context.

Conclusion

I love functional (even if I'm still learning -- back -- the basis ^^)

Gatling and Play2.0: continued

This blog entry is a follow-up of this entry where I introduced a spike I did on Play 2.0 stress tested using Gatling-tool.

At the time writing the above entry, I had to quickly hack gatling to use Akka 2.0 as Play 2.0 uses it, and I didn't wanted to have clashes.

But, thanks to Stéphane Landelle, Gatling is now Akka 2.0 enabled (since 1.1.2).

So that, it was time to give the plugin's embryo a refresh. For that I used another project that aims testing the neo4j rest plugin for Play2.0 I'm also writing. In case of, I've also introduced what I did in this project in this post.

Content

Using the Gatling I first tested how I've been able to stress test:

• simple (get) urls
• mutating (post) urls using server form underground to compile excepted data.
• duration tests (stress testing on a given period basis)

What I liked so much, even in this embryos+, is the ease to create stress tests when coupled with Play2.0 functionalities and Gatling's DSL.

Foreword

The Gatling plugin I'm currently building is located on github here and is based on sbt to build it.

But it's based on the version 1.1.4-SNAPSHOT version of Gatling's libraries (due to some fixes the Stéphane did "for me", while he was at devoxx fr, isn't he gentle !!!). 

At the time writing, you'll have to build gatling and gatling-highcharts locally using maven (quick fast!).

How to

Set up

Having created a Play 2.0 app, you now have to powerful of sbt in hands (especially if you've installed the TypeSafe stack 2.0). So, to stress test your app, you'll have first to build to plugin I told above.

Plugin

First of all, clone/fork this library project on github plugin, after what, you'll just have to run sbt publish-local in the related folder. That's it, you now have the plugin in your local repo.

Project

In your Play 2.0 app, you now are able to define the plugin as `test` dependency, using the following

That's it...

Use

Specs

Personally, for testing I use org.specs2.Specification, for using it with the plugin (at the time writing) you'll have to create the following:

This listing is creating a fake server to enable urls to be tested, and some functions to deal with it.

To create the server, the plugin defines a Util object (be.nextlab.play.gatling.Util), which also defines rough helpers to be used in stress tests.

A full spec should look the following:

Simple Url

You saw, in the provided gist above, that Util also defines a way to simply defines a gatling Simulation (basically a Function0 that returns a scenario builder: Gatling DSL result).

Having that in hand, here is the fragment to stress test the root url:

As you can see, it's pretty simple, but nothing can really be checked in the body of the specs (I'm working on having relevant information to check).

But, at least you can run this test to hit 10 times the root url ramping the numbers of users by 2.

Running it (sbt test) you'll have a new gatling folder in your target folder that contains a  results directory where are located all stress results in an html report (with great charts)

And all you had to do is to define the request headers and the url...

Mutating the server

If you have controllers that mutates the server, you should have define POST urls, which are using the Form feature provided by Play 2.0.

Having did so, you'll be able to stress it very easily using Map, JsObject or in the best case your Model.

Let's say we have a controller controllers.Stuffs that uses case classes models.Stuff. The controller defines a stuffForm and a createStuff action.

Your stress test can now be defined like the following:

In the gist, you can see 5 points to note, they are key-clues to create reusable stress tests.

Nothing is really hardcoded, neither the path to the http end point nor the parameters data.

That's http stress testing using type checked requests. CoOl isn't it? Hey, man, we got back our lovin' type cheking (one of the best scala feature).

Heavy check, duration based

This part is more a Gatling feature highlighting.

This last example is an heavy test that uses looping over a configuration for a given period. This gives you how many users could use your application.

Such test might be shaped the following:

Note

For now, it's NOT an official plugin neither a gatling nor a play 2.0 one, but discussions are on the way for that... stay tuned on twitter or here.

Still Playing... but new players are in

Why?

Because I love to play, with Play 2.0 and scala (still learning).
But also because I'm currently investigating technologies that I might choose for a new product line currently building in my company, NextLab (in Belgium).

With ?

This time, I've more played with client side libraries or frameworks (no I won't post yet another entry on JQuery...), but I also tried how it is easy to create totally async code (so parallelized) using server side ones.
The technologies that will be quickly introduced in this post can be found hereafter, but everything has been packaged in a github repo, and a running instance on Heroku.

At first the current spike was dedicated to the slowly growing Play 2.0's Neo4J REST plugin, we are creating at NextLab. But, in order to demonstrate what could be done with the coupling of these two technologies, I've extended the spike's scope to something more funny.

So let's introduce the technologies:

Client Side

Twitter Bootstrap

An amazing toolbelt helping building responsive website without having to bother boilerplates in HTML, CSS.

Even if it is neat, complete and well thought, Bootstrap comes with another handy factor... it has been built using LESS. And by chance (I know, chance is not part of the equation) LESS is supported by Play 2.0 by default.

Just a note, LESS will let you reuse color codes, mixins, etc etc that Bootstrap has already defined.

Spine.js

As we'll below, we'll have to discuss with WS (json), upon which a REST interface has been added for meaningful resources.

That's where Spine.js comes in the game. This lightweight MVC library brought me the small tools that I needed for fetching, saving resources without having to write not even a single request by hand...

d3.js

Probably my favorite (that might be my mathematician part who's talking), this powerful Data-Drive Document toolkit has taken the right thing by the right end.

Its functional approach of decoupling data from the document, and link them using layouts helps you to concentrate on each part of the data usage independently:

• the incoming/rought data mapping to a representative data
• the mapping from represented data to document (most of the time, one data for one element)

Communication

The communication layer is of course HTTP, with a little help from the emerging HTML5 features. One in particular, Server-Sent Events (here is a great intro).

This, stable, HTML5 feature comes with the handy functionality to let the server sent events to connected client, without hacks; that said Comet or Polling.

Open a connection, push data, and that ONLY when the server needs to.

Server Side

Play 2.0 (Scala)

Of course... But I used some "advanced" feature like,

Async

Async is a Play 2.0 feature that let the server deal with the tasks it has to schedule.

That is, when you think that the server might have to wait for actions being executed before being able to respond a request, Play 2.0 let you, really simply, create Async request (non blocking).

Very handy when you have to call third parties services for instance...

Iteratee

The only way I would consider from now to consume data. Iteratee is a fairly difficult thing to understand (read this wiki) but it gives you the same smart decomposition that in d3.js, that is, decoupling the management of a sources, its useful representation and its computed result.

Akka

Powerful, actor-based, parallelizer, asynchronous task, scheduler library.

Actually I needed, a request to launch async tasks (you know like event generation and dispatching)... so what else!

Neo4J

A database for storing graph... let's use a graph database.

Within NextLab, we started a open sourced Play2.0 plugin for calling the REST api provided by a Neo4J Server (helpful on Heroku). It's still emerging, and continue to evolve a lot because features are implemented on the fly (need), and a re-pass is forecasted to add a meaningful DSL (like FaKod did).

Why do we make this choice to implement a distinct plugin instead of wrapping the java library?

1. yet another library, which brings me too much (I need REST only)
2. I want requests being async and under control

Libraries Repo

See below, we'll use Heroku. So in order to deploy this application wich uses our plugin, I needed to publish somewhere.

This is what can offer Cloudbees. Among other great things like git repo, CI and so on, Cloudbees provides you for free four maven repositories that you can make public if you wish.

So I used sbt to publish on my "snapshot" repo on cloudbees, letting heroky has access to it for downloading the plugin dependency.

Platform

Free, reliable, easy to use with scala and Play 2.0... which else than Heroku?

So what

I think that this post is already too long... However, I can let you play with the resulting app here.

Check out the code (and fork it) there.

Depending on the comment, I will expand this post to other ones to respond potential demands (if there is any ^^).

Good play.

Oh yes, one last note, the application is to Create Stuff which contains dummies. Stuff are created using a simple form that must be fulfilled. Stuffs can be linked one to another by clicking the graph.

Please create Stufs and links, it will be a good test for Neo4J, the plugin and Play 2.0.

Hope you've reach this.

SMAK. hehe