tag:blogger.com,1999:blog-55315535251632611822024-03-14T05:07:39.189+01:00Ska LaOne step beyond! Hacking with Scala...Noootsabhttp://www.blogger.com/profile/07479335886992076807noreply@blogger.comBlogger25125tag:blogger.com,1999:blog-5531553525163261182.post-61916755958168483302013-09-08T16:37:00.000+02:002013-09-08T16:37:01.523+02:00An essay about expressiveness, behavior and conciseness in codeThis week I came across an interesting <a href="https://twitter.com/nicolas_frankel/status/374425127423082496" target="_blank">tweet</a>, which stated <i>that a code is read more often than it is written</i>.<br />
<br />
It can only be true, as any tautology can be. It's like a book, right? Written once, read many... until errata come in the game, or new editions.<br />
<br />
That was my point when I <a href="https://twitter.com/noootsab/status/374486823189221376" target="_blank">replied</a>, by trolling (I admit) that Java is becoming a pain in this area... because of its verbosity.<br />
Hopefully, <a href="https://twitter.com/nicolas_frankel" target="_blank">Nicolas Frankel</a> is clever enough to continue on the discussion tone rather than entering an infinite war where none can win...<br />
So we took the decision to express our point in a <a href="http://blog.frankel.ch/on-the-merits-of-verbosity-and-the-flaws-of-expressiveness" target="_blank">cross-referenced blog</a> where you'll find your own way of coding.<br />
<br />
<h3>
Reading a code</h3>
<div>
Before starting, I would like first to wonder why would we have to read code?</div>
<div>
<br /></div>
<div>
From my point of view, there are several relevant cases:</div>
<div>
<ul>
<li>learning a new library, code express more than doc that's why it must be opened</li>
<li>improving it by adding new features, in that case, we need to have the big picture before starting any hacks</li>
<li>debugging</li>
</ul>
</div>
So what I'll claim based on these three (more? poke me!) cases, is that<br />
<ul>
<li>the two firsts can be eased by a clean code describing a behavior using the right level of conciseness but keeping the expressiveness at its best.</li>
<li>the later can be a real pain when the conciseness of the code has been badly chosen</li>
</ul>
<br />
<h3>
Some words about conciseness and Java</h3>
<div>
Here I'll brush quickly some additions that Java received over the last couple of versions, a lot involved conciseness.</div>
<div>
<br /></div>
<div>
Let's take List and the for-loop.</div>
<div>
<script src="https://gist.github.com/andypetrella/6430645.js?file=coll_simple.java"></script></div>
<div>
<br /></div>
<div>
See? Everything is about making the code more concise without removing any logic nor features.</div>
<div>
<br /></div>
<div>
However, everything is not that shiny, mostly when you need to add more logic, or let say when one need to enhance the behavior. Can you read the following two examples easily?<br />
<script src="https://gist.github.com/andypetrella/6430645.js?file=coll_adv.java"></script><br />
Don't know about you, but my feeling is that the behavior is not well represented, and thus we need to touch our in-brain JVM to catch it up.<br />
<br />
The problem is that the conciseness introduce at the language level is not flexible enough to express advanced workflow, like early termination or filtering.<br />
Now let's look at a Java 8 version of this example:<br />
<script src="https://gist.github.com/andypetrella/6430645.js?file=coll_adv8.java"></script></div>
<div>
<br />
The same code, less lines... but is it really the interesting feature? Conciseness? Really!?<br />
I don't think so, the most important part is that the behavior can be catch by combining two simple behaviors:<br />
<br />
<ul>
<li>filtering</li>
<li>mapping (transforming)</li>
<li>limiting (taking)</li>
</ul>
<div>
Another good fact is that using the second version, the behavior can even be easily testable by simply testing the the predicate (dereferenced from the Test class) is returning the right value.</div>
<div>
So there is no need to test if an array will be filtered correctly, it's asserted by the library.</div>
<div>
Also, the behavior is easily extendable since now the filtering is no more explicitly hardcoded in the behavior -- however we're free to instantiate a specific behavior introducing partially applied method.</div>
<br />
Now, it's true that we have to agree on an API and everybody must know it. But I think it's always the case when you're creating an API, you're fixing names and concepts.<br />
The sad fact, in Java, is that they are trying to reinvent the wheel by renaming well know behavior like limit (take), substream (drop + take) in the Collection API. That's why semantic is hard when someone is creating his own taxonomy/ontology of concepts.<br />
Moreover, they took the opportunity to leave some noise, like the stream() call (probably for backward compatibility) which result in a collect method (which is also used to reconstruct a List using a Stream).<br />
<br />
<h3>
<strike>Conciseness</strike> Expressiveness and Scala</h3>
<div>
In this section, I won't expand myself into much details about Scala, but I'll just try to show some advantages of the expressiveness that Scala offers... for instance, even implicits (in the call-site) are explicits (in the definition-site).<br />
<br />
So I'll just mention two (among many) features brought by Scala which is really missing in Java, both being related to expressing a behavior based on an implicitly available context.<br />
<br />
<h4>
Implicit parameter</h4>
<div>
In Scala, a method/function can have several parameter blocks, like so:</div>
<div>
<code>def add(a:Int)(b:Int) = a+b</code>.</div>
<div>
<br /></div>
<div>
Ok, fine, but the very last parameter block can be declared as implicit, this way:</div>
<div>
<code>def persist(user:User)(implicit s:DBSession) = ...</code>.</div>
<div>
<br /></div>
<div>
What an implicit parameter block of parameters is, is simply a bunch of parameter that the compiler should be able to find within the compilation stack. And if it can find, in a deterministic way, such parameters they'll be automatically provided at the call-site. An example?</div>
<div>
<script src="https://gist.github.com/andypetrella/6430645.js?file=implicit.scala"></script></div>
What has just happened there? It's rather simple, when the compiler will fall on the <i>persist</i> call, it'll see that the last block is implicit and not provided so it'll search in the actual scope if there is an object matching the required type. In this case, it finds <i>session</i>.<br />
<br />
Again, I don't know about you, but I prefer this at what Hibernate (for instance) does... which is declaring by injection by something using an annotation or what not a session somewhere. And if something goes wrong? I hope you have integration tests, because it'll be check @runtime.<br />
In this case, the compiler will blow out if it cannot find a session object, that's it!<br />
<br />
So it's explicit at one single place, and will be implicit at all call sites. And I think it's cool/powerful to have such duality.<br />
<br />
<h4>
For-comprehension</h4>
A for-comprehension in Scala is very similar to a for loop in Java when dealing with sequences, however it's more than this. But beofre going futher, here is what is possible using lists:<br />
<script src="https://gist.github.com/andypetrella/6430645.js?file=for.scala"></script><br />
<br />
For those of you having already tried to use the Future API of Java should just find the following really pleasant because it'll be terse and straightforward to chain futures, without pain but with some implicit meanings...<br />
<script src="https://gist.github.com/andypetrella/6430645.js?file=futures.scala"></script><br />
What's going on there? We fetched a user in a future, when it has been fetched we fetched each friend, one by one, within a sequence then we yielded <i>both</i> results in a tuple.<br />
The resulting <i>fut variable</i> is yet another Future that will hold this later tuple if all the fetches successfully returned, otherwise everything fails!<br />
Afterwards, it's still allowed to adapt the contained tuple as a new tuple... Note that we're not dealing with the value yet, we're just describing what has to be done when the result will happen... or not.<br />
<br />
I'm not enough courageous to write the code using the Future API in Java, it would be too painful for me, too many brain gymnastic for nothing... and I didn't even talked about the number of bugs It'd be prone of.<br />
<br />
Oh yes, one last thing on this for-comprehension, <i>fetch</i> should have an implicit parameter, the storage session/metadata access and the <i>Futures</i> need an <a href="http://docs.scala-lang.org/overviews/core/futures.html" target="_blank">ExecutionContext</a> instance to be executable... but it's not the role of this piece of code to create or even pass them!<br />
<br /></div>
<h3>
Flaws of conciseness in Scala</h3>
<div>
Mainly the flaws are raising when the code tries to be concise at a such level the the expressiveness itself is penalized. </div>
<div>
For instance, sometimes, I like to tweet implementation in 140 characters like <a href="https://twitter.com/noootsab/status/373106446260383744" target="_blank">this one</a>; this is just fun to do... I won't expect to have such code dropped as is in a project unless there are a lot of reasons (I can't even image a single one).</div>
<div>
<br /></div>
Why? Because it breaks all good conventions that are becoming the de facto best practices in Scala. For those interested in, there is a <a href="http://parleys.com/play/51c1994ae4b0d38b54f4621b/chapter0/about" target="_blank">very good keynote</a> by Martin Odersky on that.<br />
<br />
The most important is probably to track the status of the types chain by decomposing a multicalls line in several ones, the very next one being to not overuse the wildcard <i>underscore</i> for inline function.<br />
<br />
So the code in the tweet can be migrated this way from:<br />
<script src="https://gist.github.com/andypetrella/6430645.js?file=MRTweet.scala"></script><br />
To:<br />
<script src="https://gist.github.com/andypetrella/6430645.js?file=readableMR.scala"></script><br />
<br />
What you only need to understand are these 5 concepts:<br />
<ol>
<li>map</li>
<li>flatMap</li>
<li>groupBy</li>
<li>mapValues</li>
<li>flatten</li>
</ol>
<ul>
</ul>
<div>
Which one couldn't you guess correctly? Make a guess than look at these rough and limited explained behaviors:</div>
<ol>
<li><code>List<A> => List<B></code> <b>OR</b> <code>Map<K,V> =><k> Map</k><L,W></code></li>
<li><code>List<List<A>><list> => List</list><A></code></li>
<li><code>List<A> => Map<k list=""><K,List<A>></k></code></li>
<li><code>Map<k v=""><k,V> => Map<K,W></k></code></li>
<li><code>List<list><List<A>> => List<A></list></code> <b>OR</b> <code>List<map><Map<K,V>> => Map</map></code><K,V></li>
<div>
</div>
</ol>
</div>
I kept it cryptic on purpose because it has been explained so many times on the web!!
<br />
<div>
<br /></div>
<div>
But also not that the readable code (that could even be more readable by still using non-verbose tools) is, with comments, less than 2 times taller. But it would require hundreds of line of Java, which I don't even have the energy to write... <b><span style="font-size: xx-small;">Or you need to convince me that I should...</span></b></div>
<div>
<b><br /></b></div>
<div>
<b>Last note: just don't override, nor create operators that haven't a widely known semantic. Haskell is a bad example, even if this language is awesome!</b><br />
<b></b><br />
<a name='more'></a><b><br /></b></div>
<h3>
Wrap up</h3>
<div>
I agree that conciseness is most of time relying on the fact that a common language must be shared by the coders, readers and writers. But isn't it the case for all APIs we're using?</div>
<div>
<br /></div>
<div>
Also, I tried to state that conciseness can really quickly be a pain when it shoots features out of the languages rather than introduce new concepts with more flexibility.</div>
<div>
<br /></div>
<div>
Conciseness is great for lazy man with a ROM-like brain, so in Scala, take care to use it carefully and when it's trivial or, maybe, not part of the behavior (like in debug prints).</div>
<div>
<br /></div>
<div>
Expressiveness that enables a code to only rely on a behavior is what verbose languages are missing the most... even if versions after versions they (Java) sneaks into the language with new embryos of solution.</div>
<div>
<br /></div>
<div>
And I didn't even talked about POJO or immutability.</div>
<div>
<br /></div>
<div>
Hope you enjoyed the read and you have a plenty of concerns/remarks on it.</div>
<div>
<br /></div>
<div>
Thanks again to Nicolas to bootstrap this <i>metaphilosophistically </i>rumination on what an readable code should present. Don't forget, if you didn't yet to read his own rumination <a href="http://blog.frankel.ch/on-the-merits-of-verbosity-and-the-flaws-of-expressiveness" target="_blank"><b>here</b></a>.</div>
Noootsabhttp://www.blogger.com/profile/07479335886992076807noreply@blogger.com0tag:blogger.com,1999:blog-5531553525163261182.post-63253082902376044502013-08-19T00:53:00.000+02:002013-08-19T00:53:33.490+02:00mid-2013, my Spark Odyssey... Rolling out Streams in the Spray RiverIn this post, I'm going to relate the last bits of the project using Spark streaming that started discussing <a href="http://ska-la.blogspot.be/2013/08/mid-2013-my-spark-odyssey.html" target="_blank">here</a> and continued <a href="http://ska-la.blogspot.be/2013/08/mid-2013-my-spark-odyssey-when-akka.html" target="_blank">here</a>.<br />
<br />
Up to now, the project hosted on <a href="https://github.com/andypetrella/spark-bd" target="_blank">github</a>, has been updated with a JavaScript web application that can show the streams that were created in the first two posts (see above). This web application is not the purpose of this post, it's purpose is about the way I was able to combine streams and present the results in a <a href="http://spray.io/" target="_blank">Spray</a> server.<br />
<br />
<h2>
Goal</h2>
So what we will see here is how Spark Streaming enables two streams to be combine when they are, somehow, containing data about the same topics.<br />
<br />
Then we'll see how it is possible to package to amount of data that is coming over the web using an aggregation method like a <i>windowed summarization</i>.<br />
<br />
Since a data is only interesting when it can be consumed by yet another analytic tool, it's pretty helpful to set up a server that enables some access to them -- to be consume by an OLAP application for instance; rather than reflushing everything in HDFS.<br />
For that, we'll use the <i>Spray server</i> that Spark helps us to create, and we'll add it routes that renders the aggregated value (RDD) as Json document to be sent over the wire.<br />
<h2>
Rollin' <i>to</i> the river</h2>
<div>
In the first blog, it was configured a <i>DStream</i> that carried on data for each tweet matching a list of stocks ; these data were associated with a sentiment score and packaged in a dedicated structure named <i>TwitterData</i>.</div>
<div>
<br /></div>
<div>
In the next blog, we created yet another <i>DStream</i> that was responsible to poll the Yahoo! finance API to fetch stock changes in real time, for that another structure was created: <i>YahooData </i>that contains information about the actual changes of a given stock.</div>
<div>
<br /></div>
<div>
With that in hand, we can easily figure out that both <i>DStream</i>s are continuously giving information about the same topic: a stock. That's why we can setup a parent trait that define only one thing, that is a <i>stock</i>.</div>
<div>
<br /></div>
<div>
Here are the structures and the related streams:</div>
<div>
<script src="https://gist.github.com/andypetrella/6207142.js?file=structures.scala"></script></div>
<div>
The <i>asInstanceOf[Data]</i> are required because DStream is not covariant (probably because it is mutable).</div>
<div>
<br /></div>
<div>
Thus, what we could do now it to merge both of them into a single one! I can't wait more, let's do it:</div>
<div>
<script src="https://gist.github.com/andypetrella/6207142.js?file=combine_trivial.scala"></script></div>
<div>
<br /></div>
<div>
What? You feel a bit frustrated because it's so simple? Completely understood, I was astonished as well how easy we can combine streams of packets where the time was part of the problem without having to do anything...</div>
<div>
<br /></div>
<div>
Cool, so now we have access to a full river of data coming in real time for a bunch of given stocks, but in real life (and with even more streams) it can be problematic to keep track of everything, or redistribute the whole thing. Let's see what we could do.</div>
<h2>
Compaction (aggregation)</h2>
<div>
A common workaround of the problem mention above (the capacity one) is to compute aggregated value for a window of time that slides on the stream.</div>
<div>
<br /></div>
<div>
To give a simple example, if we have a stream of integers that arrive at a very high rate, what can be done is to compute several stats on them on a regular basis. That is to say, we can choose to compute the mean and standard deviation of the integers coming the last 30 seconds, and do that every 5s.</div>
<div>
This would reduce the need to store only doubles every 5s, rather than, maybe thousands of doubles. Having overlapping sliding window being helpful when applying regression for instance (more component of the variance is kept).</div>
<div>
<br /></div>
<div>
So here, what we are going to do is to compute an aggregation value which an overall score for each stocks present in the stream. </div>
<div>
This score will simply be the sum over the sentiment (as -1 or 1) for Twitter part, itself added to the sum of the Yahoo changes (converted to -1 or 1) -- I'm not a financial guy so I've chosen something easy and straightforward to compute... it's not related to the technologies. Let's see how we could achieve that using the Spark API for <i>DStream</i>.</div>
<div>
<script src="https://gist.github.com/andypetrella/6207142.js?file=compaction.scala"></script></div>
<div>
<br /></div>
<div>
Nah! Okay, I'll start by the trick <code><span class="n" style="background-color: white; color: #333333; font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 11.818181991577148px; line-height: 14.545454025268555px; white-space: pre;">map</span><span class="o" style="background-color: white; font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 11.818181991577148px; font-weight: bold; line-height: 14.545454025268555px; white-space: pre;">(</span><span class="n" style="background-color: white; color: #333333; font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 11.818181991577148px; line-height: 14.545454025268555px; white-space: pre;">x</span><span style="background-color: white; font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 11.818181991577148px; line-height: 14.545454025268555px; white-space: pre;"> </span><span class="k" style="background-color: white; font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 11.818181991577148px; font-weight: bold; line-height: 14.545454025268555px; white-space: pre;">=></span><span style="background-color: white; font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 11.818181991577148px; line-height: 14.545454025268555px; white-space: pre;"> </span><span class="o" style="background-color: white; font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 11.818181991577148px; font-weight: bold; line-height: 14.545454025268555px; white-space: pre;">(</span><span class="n" style="background-color: white; color: #333333; font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 11.818181991577148px; line-height: 14.545454025268555px; white-space: pre;">x</span><span class="o" style="background-color: white; font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 11.818181991577148px; font-weight: bold; line-height: 14.545454025268555px; white-space: pre;">.</span><span class="n" style="background-color: white; color: #333333; font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 11.818181991577148px; line-height: 14.545454025268555px; white-space: pre;">stock</span><span class="o" style="background-color: white; font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 11.818181991577148px; font-weight: bold; line-height: 14.545454025268555px; white-space: pre;">,</span><span style="background-color: white; font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 11.818181991577148px; line-height: 14.545454025268555px; white-space: pre;"> </span><span class="nc" style="background-color: white; color: #445588; font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 11.818181991577148px; font-weight: bold; line-height: 14.545454025268555px; white-space: pre;">List</span><span class="o" style="background-color: white; font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 11.818181991577148px; font-weight: bold; line-height: 14.545454025268555px; white-space: pre;">(</span><span class="n" style="background-color: white; color: #333333; font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 11.818181991577148px; line-height: 14.545454025268555px; white-space: pre;">x</span><span class="o" style="background-color: white; font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 11.818181991577148px; font-weight: bold; line-height: 14.545454025268555px; white-space: pre;">)))</span></code>. Actually, this is rather convenient for what will follow, that is we created a <i>pair</i> composed of a <i><b>key</b></i> being the stock and a <i><b>value</b></i> being the singleton of the initial data. This singleton thing will become really helpful when a "reduce phase" will happen.</div>
<div>
<br /></div>
<div>
Exactly, an aggregation over a sliding window being a classical use case, the Spark API has already integrated it for (thankfully should I say). So that, we only have to call the <span style="background-color: white; color: #333333; font-family: Consolas, 'Liberation Mono', Courier, monospace; font-size: 11.818181991577148px; line-height: 14.545454025268555px; white-space: pre;">reduceByKeyAndWindow</span> method with the reduce function (<i>List.appendAll</i>) and the window's duration, 60 seconds. Note that the sliding time wasn't given, I've chosen to keep the original slicing time from the DStream creation, 5 seconds.</div>
<div>
<br /></div>
<div>
I have to warn something, nevertheless, this function is only available here because we had adapted to something that looks like a KVP, that's why we can reduce <i>byKey</i> as well.</div>
<div>
Since the key is the stock, we now have the list of all Data for a 60 seconds window grouped by stock.</div>
<div>
<br /></div>
<div>
Now, recalling what was said earlier, we only want to keep track of a score, not every Data. For that, we mapped the values only to something smaller, the overall score and the number of tweets and yahoo information that participated to this score.</div>
<div>
<br /></div>
<div>
Tada! We're done if the Spark thing... yeah actually, for this post at least (next will come the cluster deployment... but that's for the future!).</div>
<div>
<br /></div>
<div>
True, we're now collecting real time information about a list of stock associated with a score and the sources participation. </div>
<div>
<br /></div>
<div>
However, they're remain in memory... that's not really helpful right? Actually, once would request these data to perform their own analysis on them, so we need to open them to the outside world.</div>
<h2>
Delivering</h2>
<div>
Mmmmh, this part is a bit tricky because looking at the DStream API you don't have that much choice for that. We can persist (to memory, ...), we can save to an object file or a text file, but redistribute these persisted data would be costy for me, and I'm lazy.</div>
<div>
<br /></div>
<div>
But I like to read code, so I read the Spark code again, I felt again on the AkkaUtils class used in my previous post to create the actor system. And, in fact, this helper is able to create a Spray server as well -- you know this super power server thing that is able to deliver resources at an insane rate thanks to Akka!? If no, open this <a href="http://spray.io/introduction/what-is-spray/" target="_blank">link</a> in a new tab and read it afterwards (or now), you'll love it! </div>
<div>
<br /></div>
<div>
Given that, I thought to my self that it'd open some door to laziness, which I liked a lot.</div>
<div>
<br /></div>
<div>
The idea is to start such Spray server that would deliver an actor holding a representation of the aggregated information grouped by stock, with the value being the list of these aggregated data.</div>
<div>
WARN: This is certainly not the most efficient way to do this because the state of the actor grows monotonically, but it sufficient for now.</div>
<div>
<br /></div>
<div>
To do so, I've reused the trick from the previous post, to create a class in the <i>spark</i> package that starts the Spray server using <i>AkkaUtils</i>. Afterwards, the route are declared to render the data accumulated in the following actor:</div>
<div>
<script src="https://gist.github.com/andypetrella/6207142.js?file=deliver_actor.scala"></script></div>
<div>
<br /></div>
<div>
This actor will be simply populated by the <i>DStream</i> itself using the <i>transform</i> function:</div>
<div>
<script src="https://gist.github.com/andypetrella/6207142.js?file=publish.scala"></script></div>
<div>
<br /></div>
<div>
I wanted to show it to you because of the <b>FIXME</b>, which is still not fixed. Actually, the function that is given to <i>transform </i>will be part of the lineage of the RDDs, so it has to be serializable. And if I simply reuse an actor as a closure, Akka will tell me (actually how Spray uses the Akka serialization) that it's not possible to make the actor serializable since it doesn't have any actor system in the context... (when Spray calls the serialization).</div>
<div>
To counter that, I took the hard way, which is simply "recreating" actor system and actor within each <i>RDD</i>'s transform call! <b>And it's very bad</b>. Sorry about that...</div>
<div>
<br /></div>
<div>
If you want more information about this service, just check out this little class <a href="https://github.com/andypetrella/spark-bd/blob/master/src/main/scala/spray.scala" target="_blank">here</a>, we're you'll find the JSON serialization as well for the "REST" operations.</div>
<h2>
HTMLization</h2>
<div>
Since reading JSON on the fly is not my passion neither part of my resume (which should be somewhere there! or maybe there? Arf, I dunno, anyway...), I've created a very lightweight and not really user friendly interface that shows the results in real-time.</div>
<div>
<br /></div>
<div>
It's all located in the <a href="https://github.com/andypetrella/spark-bd/tree/master/src/main/webapp/web" target="_blank">web</a> folder and is intensively using Bootstrap from Twitter, AngularJs from Google and D3.js from Mike.</div>
<div>
<br /></div>
<div>
To start it, you'll need to start the whole thing as described in the README, afterwards you'll be show this kind of stuffs:</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://raw.github.com/andypetrella/spark-bd/master/preview-ui.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="300" src="https://raw.github.com/andypetrella/spark-bd/master/preview-ui.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">My fairly UI</td></tr>
</tbody></table>
<h2>
What should come next...</h2>
<div>
... When I'll have some time.</div>
<div>
The next logical step, for me at least, from here is now to deploy everything on a cluster. Because, for now, back in the first post, the context was created locally, so everything to going fine... maybe due to this fact.</div>
<div>
<br /></div>
<div>
So that'll be the next thing in this series, deploy on Amazon or on a local Vagrantized cluster of VMs ^^.</div>
<div>
<br /></div>
<div>
If any is likely to help, give me pointer or make any comments, <b>Please do!</b></div>
<div>
<br /></div>
<div>
Hope you like the read, if only someone reached this point without getting bored. If you do, a small tweet to poke me would be awesome (@noootsab).</div>
Noootsabhttp://www.blogger.com/profile/07479335886992076807noreply@blogger.com0tag:blogger.com,1999:blog-5531553525163261182.post-5428580644130656522013-08-16T00:44:00.002+02:002013-08-16T00:44:44.718+02:00mid-2013, my Spark Odyssey... when Akka sparks Yahoo! financeThis entry is the second of a series that relates some work/test I'm doing using Spark. All information can be found in the very <a href="http://ska-la.blogspot.be/2013/08/mid-2013-my-spark-odyssey.html" target="_blank">first post of this series</a>.<br />
<br />
In the previous post, I've explain how easy it is to consume a Twitter stream using Spark, and seamlessly easy how to attach a sentiment score to each tweet passing a given condition (read: <i>filtered</i>).<br />
<br />
In this one, I'm going to tackle another fold of my test, which is fetching the current stock information for a given list of companies.<br />
These information will be provided by the Yahoo finance API...<br />
<br />
...well, hum, not exactly an API. Actually, Thanks to <a href="https://twitter.com/kennyhelsens" target="_blank">@kennyhelsens</a> who shown me this "hack", we'll use a Yahoo! URL that produces a CSV file containing real-time (mostly) information about companies' stock changes. Information are really hard to grasp for this URL's parameter however you can find some out <a href="http://cliffngan.net/a/13" target="_blank">there</a>.<br />
<br />
<h2>
Storyline</h2>
<div>
In this series, we're trying to uses real-time streams rather that batch processing to give a real-time fluctuation of the health of given companies.</div>
<div>
But we've just seen that we're going to consume CSVs that give a snapshot of the stocks changes.</div>
<div>
A common solution to transform a static API to a stream is to poll it and to create a streams by appending the results one after the other.</div>
<div>
<br /></div>
<div>
Here, we'll try a pretty simple solution for that, we'll delegate to <b>Akka</b> the task to consume the CSV, to parse it, to split data by company, and to modelize each and finally to push everything to a <code>DStream</code>.</div>
<div>
<br /></div>
<div>
It's rather straightforward and we'll see that it's even simpler that we would expect since, Spark is already playing a good game with Akka by integrating out of the box an API to "consume" actor's messages.</div>
<h2>
Acting</h2>
<div>
In this scene, there will be several actors each playing its own roles (sorry <a href="http://en.wikipedia.org/wiki/Peter_Sellers" target="_blank">Peter Seller</a>'s fans<i> -- like me btw</i>).</div>
<div>
Here they are:</div>
<div>
<ul>
<li><b>scheduler</b>: this one is provided by Akka out-of-the-box and is able to send a "given message" to a given actor at a given interval.</li>
<li><b>feeder</b>: this actor will simply fetch the CSV and produce the data to be sent to the below actor.</li>
<li><b>receiver</b>: this one will receive a message whenever a data is available for a given company.</li>
<li><b>parentReceiver</b>: this one will be created by Spark itself using the <code>StreamingContext</code> and is able to push the received <code>Data</code> to the right RDD in its holding DStream (see <code><a href="https://github.com/mesos/spark/blob/v0.7.3/streaming/src/main/scala/spark/streaming/receivers/ActorReceiver.scala#L91" target="_blank">ActorReceiver</a></code>).</li>
</ul>
</div>
<div>
<br /></div>
<div>
Now the scenario:</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://2.bp.blogspot.com/-LDaorJUhJ78/Ug1BxvnH7uI/AAAAAAAAB6I/5g-vSG8HyoA/s1600/SparkAkkaConsumeOustideWorld.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="640" src="http://2.bp.blogspot.com/-LDaorJUhJ78/Ug1BxvnH7uI/AAAAAAAAB6I/5g-vSG8HyoA/s640/SparkAkkaConsumeOustideWorld.png" width="547" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Sequence from the scheduler to the DStream (current RDD)</td></tr>
</tbody></table>
<div>
Does that make sense? Let's have a short overview on the implementation.</div>
<h4>
<br /></h4>
<h4>
Scheduling the poller</h4>
<div>
First we have to create a scheduler that will poll every 500ms the server... well using Akka and the separation of concerns and Message Passing Style, we'll just create a scheduled action that passes a message to a given actor:</div>
<div>
<br /></div>
<div>
But even before that, let's prepare the Akka field:</div>
<div>
<script src="https://gist.github.com/andypetrella/6207142.js?file=SparkAkka.scala"></script></div>
<div>
<br /></div>
<div>
There are a plenty way of creating <code>ActorSystem</code> with Akka, but Spark is preparing some stuffs for us to create one that will respect some of its own internal conventions. This helper is available in the `spark` package, naming <code>AkkaUtils</code>; however it's package protected... That's why I've created my own <i>utils</i> to create the <code>ActorSystem</code> under the spark package -- I'm bad sometime, but I'm so often lazy :-D.</div>
<div>
<br /></div>
<div>
Actually, the Spark utility for Akka will create an <code>ActorSystem</code> using a host and a port for us, that'll be really helpful afterwards... because everything in Spark must be serializable and thus we cannot so easily package an <code>ActorRef</code> (needs an <code>ActorSystem</code>, ...). So that, we'll use URL to create refs rather than using actors picked within a closure in a Spark action.</div>
<blockquote class="tr_bq">
<b>Recall:</b> that's how Spark manage to recover from a failure, it's take to lineage of the <code>RDD</code> or rebuild everything up from the initial data (not possible in Streams...). But specially, the actions (functions) are dispatched to machine where data reside. So the lineage contains the functions, and a function might contain <code>ActorRef</code>s!</blockquote>
<div>
Now that we have an <code>actorSystem</code> available, let's create the scheduler (see last section, <b>Drawbacks</b>, for more info about problems with that methodology):</div>
<div>
<div>
<script src="https://gist.github.com/andypetrella/6207142.js?file=scheduler.scala"></script></div>
</div>
<div>
Nothing relevant to say about that, only maybe the definition of <code>FeederActor</code>...</div>
<h4>
<br /></h4>
<h4>
Feeding Spark with Yahoo data</h4>
<div>
Here is coming the feeding phase, we won't see how the CSV is read and parsed however we'll see what going on with the messages flow.</div>
<div>
<div>
<script src="https://gist.github.com/andypetrella/6207142.js?file=feeder.scala"></script></div>
</div>
<div>
What's interesting in this actor is two-folds:</div>
<div>
<ol>
<li>it can receive a message containing two data: </li>
<ol>
<li>an <b>actorRef</b> that it will hold in its state: this one will be the <b>receiver</b></li>
<li>a list of <b>stocks</b> to follow (by building the according call to Yahoo)</li>
</ol>
<li>when consuming the data and producing the data by stock, it <b>sends</b> them one by one to the above actor</li>
</ol>
<div>
Actually, this feeder has two responsibilities which are fetching the data and pushing it to Spark.</div>
</div>
<div>
But how is this receiver constructed and passed?</div>
<h4>
<br /></h4>
<h4>
Receiver -- our spark stuff</h4>
<div>
The receiver is actually an actor that will be created internally by Spark for us, for that it must respect yet another trait, <b>Receiver</b>. And a <code>Receiver</code> can only be an <code>Actor</code> (using self definition)!</div>
<div>
<div>
<script src="https://gist.github.com/andypetrella/6207142.js?file=receiver.scala"></script></div>
</div>
<div>
Four things to point here:</div>
<div>
<ol>
<li>this actor is well-defined by extending the Receiver trait, which provide the <code>pushBlock</code> method.</li>
<li>it holds a reference to actor that it, it-self, create based on a give URL -- an actor URL... this is where <code>AkkaUtils</code> came handy! This Actor will be the <b>feeder</b> actor.</li>
<li>in the hook <code>preStart</code> we tell the <b>feeder</b> actor that this <code>ActorRef</code> is the Spark end-point where data must be sent to publish in the stream.</li>
<li>when data arrives, we check in the cache if it's already there (see below), if not we call the <code>Receiver#pushBlock</code> with it.</li>
</ol>
<div>
Why do we have to cache? Actually when the stock market has closed, Yahoo will continually return the last change before closing... so, here, we just avoid telling our health checker that the mood is monotonously increasing or decreasing (or even just flat if zero).</div>
</div>
<div>
<br /></div>
<div>
The pushBlock method is the clue to publish to the underlying <code>DStream</code> managed by Spark, actually it will wrap our data into an Internal representation before <b>telling</b> the parent actor to use it: <code>context.parent ! Data(y)</code>.</div>
<div>
<br /></div>
<div>
But what is that <code>context.parent</code>?</div>
<h4>
<br /></h4>
<h4>
parentReceiver -- not our spark stuff</h4>
<div>
This receiver is totally managed by Spark, and so I won't dig into more details than just saying it's being created underneath when doing the following:</div>
<div>
<div>
<script src="https://gist.github.com/andypetrella/6207142.js?file=receiver.scala"></script></div>
</div>
<div>
Fairly simple, yes, it's true! Actually, the <code>StreamingContext</code> we created at the very start of the project contains everything to create <code>DStream</code> based on Akka, all it needs is a reference to a <b>Receiver</b> (defined in the previous section).</div>
<div>
<br /></div>
<div>
What is this <b>actorStream</b> doing is creating an <code>ActorReceiver</code> (with a <code>Supervisor</code> and blahblah) which will itself create and manage a worker for the given <code>Props</code>, but also it will hold the reference to the current <code>RDD</code> in the <code>DStream</code>.</div>
<div>
<br /></div>
<div>
The worker will be able to send him messages of the form <code>Data(d)</code> using <code>pushBlock</code>, and which it'll push to the <code>DStream</code>.</div>
<div>
<br /></div>
<div>
Concise huh? If only the document was telling us that, I wouldn't have to dig that deeper in the code... the good side-effect is that now I understand it pretty well. </div>
<div>
So, I hope you too!</div>
<h2>
Where are we?</h2>
<div>
Up to here, we've created a <code>DStream</code> of data for each tweet paired with 'its' company, but also we have now some info about what the stock change of them.</div>
<div>
What will happen in the next blog is that we'll create a multiplexed stream for which we'll compute a value for every window of 60'.</div>
<h2>
Drawbacks of the/my methodology</h2>
<div>
At this level, I can see two problems.</div>
<h4>
<br /></h4>
<h4>
The scheduler</h4>
Using a scheduler is bad because we cannot ensure that each message are processed sequentially by Spark. So it would say that we push data to an RDD that correspond to the time where the whole CSV has been processed but also the messages sent by the actors!<br />
<br />
I put this aside for the moment since I'm not building a real-life project (no-one will use this tool to take decision on the marker ^^).<br />
Moreover, mathematically, let say at the limit, the value will be correct: reducing an amount by 2 at t⁽¹⁾ or at t⁽²⁾ is not really problematic, it's like inverting two ints in a list that we're summing over.<br />
<h4>
<br /></h4>
<h4>
The consumer</h4>
I didn't shown its current implementation because its very simple and bad, that is I'm simply opening a connection on the URL, consuming everything until the end before applying the process...<br />
<br />
That says: it's blocking and non-reactive.<br />
Also, That says that we loose the power of parallelism since the Akka message dispatcher will have to wait a CSV to be completely processed before handling the next one.<br />
<br />
However it's just an implementation detail that can be easily worked around without modifying the solution.Noootsabhttp://www.blogger.com/profile/07479335886992076807noreply@blogger.com0Slins, Juprelle, Belgium50.7277297 5.563796599999932450.6875277 5.4831155999999321 50.7679317 5.6444775999999326tag:blogger.com,1999:blog-5531553525163261182.post-89869925104545809392013-08-12T01:29:00.001+02:002013-08-12T01:29:10.650+02:00mid-2013, my Spark Odyssey... <h2>
Short intro</h2>
<div>
In this post, I'll talk about yet another technology-tester project I've worked on these last days: <a href="http://spark-project.org/" target="_blank">Spark</a>.</div>
<div>
Long story short, spark is a handy distributed tool that helps in manipulating Big Data from several sources -- even hot streams.</div>
<h2>
TL;DR</h2>
<div>
In the first part, I'll put the context in place.<br />
<br />
<ol>
<li>I'll start with a gentle bunch of words about Spark and Spark-streaming.</li>
<li>Then I'll give some bits about the project I've realized so far, and why.</li>
</ol>
</div>
<div>
The second part will discuss some implementation details.<br />
<br />
<ol>
<li>Setup: spark streaming</li>
<li>Twitter stream filtered by keywords to DStream</li>
<li>Tweets' sentiment analyzer</li>
<li>Yahoo finance CSV data and the actor consumer</li>
<li>Actor consumer publishing on a DStream (back-ended by a dedicated actor)</li>
<li>Aggregation (union) based on a 60 seconds length window</li>
<li>Publishing the results using Spray</li>
</ol>
<div>
What're missing so far (among other business related things for instance):</div>
<div>
<ol>
<li>cluster deployment...</li>
<li>... and configuration</li>
<li>probably a storing strategy for further consumptions (Riak, ...?)</li>
<li>a web application that shows the results or configure the processes</li>
</ol>
</div>
<br />
<h2>
Context</h2>
</div>
<h3>
Spark</h3>
<div>
I won't discuss Spark that much because there are already a plenty of <a href="http://strata.oreilly.com/2012/08/seven-reasons-why-i-like-spark.html" target="_blank">good</a> <a href="http://www.wired.com/wiredenterprise/2013/06/yahoo-amazon-amplab-spark/" target="_blank">blogs</a>/<a href="http://blog.samibadawi.com/2012/03/hive-pig-scalding-scoobi-scrunch-and.html" target="_blank">articles</a> on that. However, I'll give you just the intuition that you got the whole thing about it:</div>
<div>
<ul>
<li>Spark is a collection of functions that works on a Sequence (List) of typed data -- <i><b>Dataset</b> = Sequence ; operation = Function</i></li>
<li>Chaining these functions will define a workflow for your data -- <i>combination</i></li>
<li>Each data will be processed only once by one or another Spark worker -- <b><i>Distributed</i></b></li>
<li>If it fails for some external reason, the data can be fetched again from the source and the workflow to be replayed entirely -- <b><i>Resilient</i></b></li>
</ul>
<div>
Spark defines an <b>R<span style="font-size: xx-small;">esilient</span>D<span style="font-size: xx-small;">istributed</span>D<span style="font-size: xx-small;">ataset</span></b> which is very different that the classical MR -- because it is distributed by essence, a workflow can easily be iterative and doesn't require intermediate caching/storage.</div>
</div>
<div>
<br /></div>
<div>
Also, spark as a sub-project <i>spark-streaming</i> that allows manipulating streamed data. The idea is very simple by defining a <b>DStream</b> as being a sequence of RDDs containing the data for a certain slice of time.</div>
<div>
<br /></div>
<div>
Oh, RDD and DStream are <b>typesafe</b> -- the whole thing being <b>Scala</b> code.</div>
<div>
<br /></div>
<div>
The spark-streaming project is really well integrated and can consume a lot of streams like <b>Twitter</b>, File System, Kafka or even an <b>Akka actor</b>.<br />
<br /></div>
<h3>
Project</h3>
<div>
This blog relates a small project publicly available on github (<a href="https://github.com/andypetrella/spark-bd" target="_blank">spark-bd</a>) that helped me catch the big picture of Spark, after having followed the mini course <a href="http://ampcamp.berkeley.edu/big-data-mini-course/" target="_blank">here</a>.</div>
<div>
<br /></div>
<div>
The project is based on another working group I'm participating in. This project, simply called project 2, has been initiated and is still supported by the great Belgian Big Data user group (<a href="http://bigdata.be/">http://bigdata.be/</a>).</div>
<div>
The idea was to co-create in several workshops an application that given a list of companies to analyse;</div>
<div>
<ul>
<li>catches the Twitter stream and the Yahoo finance data using <a href="https://github.com/nathanmarz/storm/wiki/Concepts" target="_blank">Storm</a>;</li>
<li>manipulates both by either applying a sentiment analysis or simply the changes in price;</li>
<li>aggregates by window of time both information for each company</li>
<li>gives (constantly) the information about how is going a given list of companies on the market.</li>
</ul>
<div>
Fairly simple and really funny... the project is still ongoing but you can check its progression <a href="https://github.com/bigdatabe/p2" target="_blank">here</a>.</div>
</div>
<div>
<br /></div>
<div>
It was really funny but I wanted to see how my feeling would compare by writing the same application using my actual preferred language (Scala) and the next technology in the Big Data field I was eagerly looking at (Spark).</div>
<div>
<br /></div>
<div>
<b>IMPORTANT: </b>since if I'm not there yet to make a real comparison (a niche for another blog, later on), I'll only give an insight on my actual work.</div>
<div>
<h2>
<br /></h2>
<h2>
Internals</h2>
</div>
<h3>
Setup</h3>
<div>
The setup is very minimal for a Spark application. All we certainly need is reduced to two things:</div>
<div>
<ol>
<li>an SBT project declaring as dependencies Spark and Spark-streaming.</li>
<li>in a main class/object configure the spark streaming context.</li>
</ol>
<div>
The SBT project only requires an build.sbt in the root folder that contains these lines only:</div>
</div>
<div>
<script src="https://gist.github.com/andypetrella/6207142.js?file=build.sbt"></script></div>
<div>
<br />
Where the only lines that are under interest are the two last ones.<br />
<br />
Then we can create an object that will start the spark context.</div>
<div>
<script src="https://gist.github.com/andypetrella/6207142.js?file=start.scala"></script></div>
<div>
Here we just asked Spark to create a streaming context on the <i>local</i> environment, named <i>Project2</i> and that will slice the <i>DStream</i> into <i>RDD</i>s of 5 seconds duration each.<br />
<br />
Afterwards, we asked the context to start its work... which is nothing up to now, since nothing as been added yet to the context.<br />
So let's do it.<br />
<br />
<h3>
Twitter stream</h3>
</div>
<div>
Here we'll consume the tweets stream provided by the Twitter's API. For that, we'll have to do only two things (a pattern emerge...?):</div>
<div>
<ol>
<li>register the 4 needed information for an OAuth2 authentication to the Twitter API (so don't forget to add an application to your Twitter development account).</li>
<li>add a tweets stream to the spark context</li>
</ol>
<div>
Actually, the streaming context as convenient methods to add a twitter stream.</div>
<div>
For that it'll use the Twitter4J library (it's definitively not the best, but at least we can achieve our goal).</div>
<div>
<br /></div>
<div>
This library needs several keys to be set in the <code>System.properties</code>.</div>
<div>
There are a plenty of way to do that, in my project I took something rather straightforward: I've picked the Typesafe config library and added those four configuration keys in an <i>application.conf</i> file, then I load all value in the related System property.</div>
</div>
<div>
<br /></div>
<div>
<div>
<script src="https://gist.github.com/andypetrella/6207142.js?file=twitter-config.scala"></script><br />
<br />
As you may (or not) see, the application.conf file will look for <b>environment variables</b> giving values for the required key/secret/token.<br />
<br />
That's done we can start listening for tweets:<br />
<br />
<script src="https://gist.github.com/andypetrella/6207142.js?file=listen-tweets.scala"></script><br />
<div>
<br /></div>
<div>
Dead simple, after having created a model for a company/stock, we used the configured keywords as tags for the twitter stream.</div>
<div>
Behind the sea, Twitter4J will access the stream end-point with these tags, and Twitter to pre-filter all tweets based on those keywords (in hashtag, username, text, ...).</div>
<div>
<br /></div>
<div>
At this stage we've only added the stream, but we didn't yet said anything about how to consume them... So we can simply call <code>print()</code> on the stream to ask spark printing the 10 first incoming event in each RDD.</div>
<br /></div>
<div>
</div>
</div>
<div>
However, that's not how we want to deal with those tweets right? Let's adapt our stream to add relevant information for our use case.</div>
<div>
<br /></div>
<h3>
Sentiment analyzer</h3>
<div>
Now that we have a stream of tweets it's time to read them and to give them a sentiment score based on a <a href="https://github.com/andypetrella/spark-bd/blob/master/src/main/resources/sentiments.txt" target="_blank">sentiments file</a>. The code that reads it and that assign a score for a tweet is not that important here, but the code is <a href="https://github.com/andypetrella/spark-bd/blob/master/src/main/scala/sentiments.scala" target="_blank">here</a>.</div>
<div>
<br /></div>
<div>
However, what we would like to do is to adapt our stream with this score attached to each tweet.</div>
<div>
<br /></div>
<div>
<script src="https://gist.github.com/andypetrella/6207142.js?file=statusWithNotEmptySentiments.scala"></script></div>
<div>
<br /></div>
<div>
As said earlier, Spark is a bunch of methods on Sequences of data, so we can now assert that it's true. </div>
<div>
<br /></div>
<div>
Look at what was done in the <code>map</code>, we picked each tweet, fetch the list of sentiment entries that match the text, and also we kept the whole original status for further computation. This results in a <code>DStream[(List[Sentiment], Status)]</code>.</div>
<div>
<br /></div>
<div>
Since we only want those tweets that have at least one Sentiment that matches, we then filtered on the length of this matching list.</div>
<div>
<br /></div>
<div>
We combined both actions to get a resulting filtered and sentiment-analyzed tweets.</div>
<div>
<br /></div>
<div>
Okay great, but what we <b>really</b> want is to have such information grouped by Company/Stock right? So that we can compute the participation of the social network to the health of each company.</div>
<div>
<div>
<br /></div>
</div>
<div>
To do so, we have to group semantically (based on text's content and required stocks' keywords).</div>
<div>
<div>
<script src="https://gist.github.com/andypetrella/6207142.js?file=sentimentsAndStatusByStock.scala"></script></div>
</div>
<div>
Note that it's not that straightforward because a tweet could relate several companies at once!</div>
<div>
<br /></div>
<div>
That's all! We're now done with the twitter part, we are now getting all Data that gathered the information of a Company/Stock, the tweet and its associated sentiment. Which opens doors for further aggregations/grouping/reducing.</div>
<div>
<br /></div>
<h2>
Interlude</h2>
<div>
Let's now pause a bit, I encourage you to try your own stuffs and playing with the tweets.</div>
<div>
For that, I also encourage you to fork/clone the original <a href="https://github.com/andypetrella/spark-bd" target="_blank">repo</a>, where you can already start only the twitter feed and print everything in the console by running the appropriate command:</div>
<div>
$> <span style="background-color: white; color: #777777; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 15px; line-height: 25px;">sbt</span></div>
<div>
sbt> <span style="background-color: white; color: #777777; font-family: Helvetica, arial, freesans, clean, sans-serif; font-size: 15px; line-height: 25px;">run-main be.bigdata.p2.P2 twitter print GOOG AAPL</span></div>
<div>
<br /></div>
<div>
This will run the start spark and filter the tweets based on the keywords associated to both GOOG and AAPL, whose tweets will be analyzed as said in the last section before being printed to the console.</div>
<div>
<b>WARN: don't forget to export the required environment variables</b></div>
<div>
<br /></div>
<h2>
Next</h2>
<div>
In the next blog posts, I'll continue with the explanation on how use Akka to construct Spark streams and how to combine several streams into a single one.</div>
<div>
<br /></div>
<div>
This multiplexed stream will then be used to compute an aggregated data by a window of time.</div>
<div>
<br /></div>
<div>
Having these data in hand, we'll be able to create a Spray server that expose them into a JSON format, helping us creating a client web application that simply consumes the JSON messages.</div>
Noootsabhttp://www.blogger.com/profile/07479335886992076807noreply@blogger.com0tag:blogger.com,1999:blog-5531553525163261182.post-83434934825570233772013-07-14T23:58:00.001+02:002013-07-14T23:58:48.965+02:00Function1[-T1, +R] -- What the heck?In this post, I'll try to cover an important notion of the Scala's Type System -- and a Java's Terra Incognita.<br />
<h2>
<span style="font-size: large; font-weight: normal;">(<i>generic</i>) Type covariance or contravariance.</span></h2>
Ho yeah, I heard you... <b>yet another post</b>... right?<br />
... right! but it seems that it needs a more gentle one, where Category Theory is left apart. <span style="font-size: xx-small;">Because, Category Theory must come when you want yet more fun, not to get basic concepts!</span><br />
<span style="font-size: xx-small;"><br /></span>
That's why we'll see what are these constraints in a more pragmatic way, using (important) examples.<br />
Afterwards, I'll let you read the blogs using Category Theory ... if you like.<br />
<br />
I thought that taking the problem from the Java side might be easier to explain, thus the following parts<br />
<ul>
<li><b>Problem</b>: Assignments failures using <b>Java</b> Generics</li>
<ul>
<li><b>Solution</b>: Covariance Type in <b>Scala</b></li>
</ul>
<li><b>Problem</b>: Unuseful generics for method parameter in <b>Java</b></li>
<ul>
<li><b>Solution</b>: Contravariance Type in <b>Scala</b></li>
</ul>
</ul>
<div>
<h2>
Problem: Java tells me that Kids aren't Humans</h2>
</div>
<div>
Since Java 5, as Java developers, we were really interested in the new breaking feature that was added: <b>Generic Types</b>.</div>
<div>
And it was rather interesting enough thanks to the <i>java.util.Collection</i> API, but also the syntactic sugar <b>for (A a: as)</b>.</div>
<div>
However, we also felt really quickly on the problem that the hierarchy of Generics <i>doesn't span</i> to upper level type (as explain in the <a href="http://docs.oracle.com/javase/tutorial/java/generics/inheritance.html">Java tutorial</a>).</div>
<div>
<br /></div>
<div>
Let's clarify the situation with an example.</div>
<div>
First we define a model with an classical hierarchy:</div>
<div>
<script src="https://gist.github.com/andypetrella/5995607.js?file=model.java"></script></div>
<div>
<br /></div>
<div>
Okay, with this setup, one might want to create a list of kids and assign it to a list of human. Why? </div>
<div>
<ol>
<li>because they are...</li>
<li>because they grow!</li>
</ol>
<div>
Recall: in OO a model (should) represents the domain.</div>
</div>
<div>
<br /></div>
<div>
Fine! Here is several tries:</div>
<div>
<div>
<script src="https://gist.github.com/andypetrella/5995607.js?file=co.java"></script></div>
</div>
<div>
<br /></div>
<div>
See? You just can't in Java! Because a <b>ArrayList<kid></kid></b> is not considered to be an instance of <b>ArrayList<human></human></b>. This will impact all your further algorithms, unless you use the tricky tip to anonymise your type and to constrain it with a the <i>Human</i> type as bound.</div>
<div>
<br /></div>
<div>
At least we spotted the real problem... Let's see the solution that <b>Scala</b> offers.</div>
<h3>
Solution: Covariance</h3>
<div>
In Scala, a type can be defined to be covariant to one (or several) of its generic type (roughly speaking).<br />
We can simply check the Scala's <b>List</b> definition: </div>
<div>
<h4 class="signature" id="signature" style="background-color: #d7d7d7; background-image: url(http://www.scala-lang.org/api/current/lib/signaturebg2.gif); background-repeat: repeat no-repeat; border: 0px; clear: both; font-family: monospace; font-size: 11.5pt; font-weight: inherit; line-height: 18px; margin: 0px; min-height: 18px; padding: 8px; text-shadow: white 0px 1px 0px; vertical-align: baseline;">
<span class="modifier_kind" style="border: 0px; display: inline; float: left; font-family: inherit; font-size: 15.454545021057129px; font-style: inherit; margin: 0px; padding: 0px; position: static; text-shadow: white 0px 1px 0px; vertical-align: baseline; width: auto;"><span class="modifier" style="border: 0px; font-family: inherit; font-size: 15.454545021057129px; font-style: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">sealed abstract </span><span class="kind" style="border: 0px; font-family: inherit; font-size: 15.454545021057129px; font-style: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">class</span></span><span class="symbol" style="border: 0px; display: inline; font-family: inherit; font-size: 15.454545021057129px; font-style: inherit; margin: 0px; padding: 0px 0px 0px 0.7em; text-shadow: white 0px 1px 0px; vertical-align: baseline;"><span class="name" style="border: 0px; display: inline-block; font-family: inherit; font-size: 15.454545021057129px; font-style: inherit; font-weight: bold; margin: 0px; padding: 0px; vertical-align: baseline;">List</span><span class="tparams" style="border: 0px; font-family: inherit; font-size: 15.454545021057129px; font-style: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">[<span name="A" style="border: 0px; font-family: inherit; font-size: 15.454545021057129px; font-style: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">+A</span>]</span></span></h4>
</div>
<div>
<span style="font-size: 15px;">Here, we tells the compiler that </span><b style="font-size: 15px;">List</b><span style="font-size: 15px;"> is </span><b style="font-size: 15px;">covariant</b><span style="font-size: 15px;"> with its generic type </span><b style="font-size: 15px;">A</b><span style="font-size: 15px;">.</span><br />
<span style="font-size: 15px;"></span></div>
<div>
<span style="font-size: 15px;"><br /></span>
<span style="font-size: 15px;">In other words? <b>List</b> will vary accordingly with the variance of <b>A</b>!</span><br />
<br />
And life gets simple (respecting the OO concepts):<br />
<div>
<script src="https://gist.github.com/andypetrella/5995607.js?file=Co.scala"></script></div>
</div>
<div>
<br /></div>
<div>
Following the type of generic type is pretty cool and useful, but there are cases where it won't help... let's jump to the next section if you don't believe me...</div>
<div>
<h2>
Problem: Java tells me that Kids aren't Human and Integer isn't a Number...</h2>
<div>
To illustrate this case, we'll create two helper type that define no-arg procedure and an action (a function) with one argument.</div>
<div>
<div>
<script src="https://gist.github.com/andypetrella/5995607.js?file=actions.java"></script></div>
</div>
<div>
Simple as simple.</div>
<div>
<br /></div>
<div>
We also created to action definitions, one that maps a single integer onto a single human, and another one that maps a number onto an adult.</div>
<div>
<br /></div>
<div>
This will help us defining an higher order operation that, for instance, can map a list of integers into a new list of humans. For that, we'll create the <b>ListMapper</b> class (see below) that will map an input type <b>I</b> to an output type <b>O</b>.</div>
<div>
<br /></div>
<div>
Using this ListMapper, we'll try to convert:</div>
<div>
<ul>
<li>a list of integer into a list of humans </li>
<li>a list of numbers into a list of adults</li>
</ul>
<div>
We can easily imagine that the logic should remain in the actions introduced in above. Right?</div>
</div>
<div>
Let's see how it goes in Java thus:</div>
<div>
<div>
<script src="https://gist.github.com/andypetrella/5995607.js?file=contra.java"></script></div>
</div>
<div>
huh?</div>
<div>
It seems that the mapper from <b>Integer</b> to <b>Human</b> is only able to deal with an action that takes exactly these types as input and output resp.</div>
<div>
But, it's weird, because something that works on <b>Number</b> should be able to handle any <b>Integer</b>, isn't it?</div>
<div>
And, if the returned object is an <b>Adult</b>, it should be ok to assign it to an <b>Human</b>? True?</div>
<div>
<br /></div>
<div>
For sure, but for the same reason as before, Java is not able to deal with such multi-levels hierarchies. crap...</div>
<div>
<br /></div>
<div>
Before going ahead, one might have noticed that the inheritance hierarchies are opposites:</div>
<div>
<ul>
<li>Number is extended by Integer</li>
<li>Adult extends Human</li>
</ul>
<div>
But the mapper should be able to accept both pairs of input/output type! damn... the action type doesn't seems to vary the same way for input as for output!</div>
</div>
<div>
<br /></div>
<div>
Indeed, an action must be <b>contravariant</b> on its Input and covariant with its Output.</div>
<div>
<br /></div>
<div>
Here we are.</div>
<h3>
Solution: Contravariance</h3>
In the example above, we introduce action that acted as function taking a single argument. And those actions had to behave differently on the type of the input and the output.<br />
<br />
Let's check this out in Scala, looking at what looks like such function:<br />
<div>
<h4 class="signature" id="signature" style="background-color: #d7d7d7; background-image: url(http://www.scala-lang.org/api/current/lib/signaturebg2.gif); background-repeat: repeat no-repeat; border: 0px; clear: both; font-family: monospace; font-size: 11.5pt; font-weight: inherit; line-height: 18px; margin: 0px; min-height: 18px; padding: 8px; text-shadow: white 0px 1px 0px; vertical-align: baseline;">
<span class="modifier_kind" style="border: 0px; display: inline; float: left; font-family: inherit; font-size: 15.454545021057129px; font-style: inherit; margin: 0px; padding: 0px; position: static; text-shadow: white 0px 1px 0px; vertical-align: baseline; width: auto;"><span class="kind" style="border: 0px; font-family: inherit; font-size: 15.454545021057129px; font-style: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">trait</span></span><span class="symbol" style="border: 0px; display: inline; font-family: inherit; font-size: 15.454545021057129px; font-style: inherit; margin: 0px; padding: 0px 0px 0px 0.7em; text-shadow: white 0px 1px 0px; vertical-align: baseline;"><span class="name" style="border: 0px; display: inline-block; font-family: inherit; font-size: 15.454545021057129px; font-style: inherit; font-weight: bold; margin: 0px; padding: 0px; vertical-align: baseline;">Function1</span><span class="tparams" style="border: 0px; font-family: inherit; font-size: 15.454545021057129px; font-style: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">[<span name="T1" style="border: 0px; font-family: inherit; font-size: 15.454545021057129px; font-style: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">-T1</span>, <span name="R" style="border: 0px; font-family: inherit; font-size: 15.454545021057129px; font-style: inherit; margin: 0px; padding: 0px; vertical-align: baseline;">+R</span>]</span></span></h4>
</div>
<div>
Bloody hell, a function type in Scala will vary in the <b>opposite</b> way as for the covariant types. That's why a minus sign is used on the T1 type.<br />
<br />
Simple said: a function <b>F</b> will be extended by any function that respect either or both of these cases:<br />
<ul>
<li>its argument is a <b>super-type</b> of the <b>F</b>'s argument.</li>
<li>its type (output type) is a <b>sub-type</b> of the <b>F</b>'s type.</li>
</ul>
<div>
Let's see everything in action now:</div>
<div>
<div>
<script src="https://gist.github.com/andypetrella/5995607.js?file=Contra.scala"></script></div>
</div>
<div>
That was for illustration purpose because the types weren't optimize for maximal genericity, however we can see that we were able to use a function that takes an <b>AnyVal</b> argument and results into an <b>Adult</b> object where a function from an <b>Int</b> to a <b>Human</b> were used!</div>
<h2>
Conclusion</h2>
<div>
A lot for nothing? Maybe?</div>
<div>
But it's a question that is very common for Scala newcomers. And I think it's one of the most important ones, thus it must be answered in the more precise and accessible way.</div>
<div>
<br /></div>
<div>
Furthermore, it also demonstrates that <b>Scala is even more Object Oriented that Java can be</b>.</div>
<div>
<br /></div>
<div>
--</div>
<div>
Please let me know, if it can be more clearer, this blog is meant to be organic, so any comments, concerns, helps are more than welcome</div>
</div>
</div>
Noootsabhttp://www.blogger.com/profile/07479335886992076807noreply@blogger.com0tag:blogger.com,1999:blog-5531553525163261182.post-65955410212531893982012-08-13T23:05:00.000+02:002012-08-19T22:36:32.941+02:00MongoDB and Play2 at ease: using Play-Salat plugin and Embed Mongo pluginA while ago I've blogged about MongoDB with Play2 using Salat, <a href="http://ska-la.blogspot.be/2012/03/play-20-and-salat-mongodb-dao-provider.html" target="_blank">here</a>.<br />
<br />
This post was describing how to integrate Salat easily with Play2 and gave some advice on actions to care of.<br />
<br />
<h4>
Preface</h4>
Play 2 gained in popularity and an amazing plugin as emerge for this purpose: <a href="https://github.com/leon/play-salat" target="_blank">play2-salat</a>.<br />
<br />
This plugin offers a lot of configuration to hit running instances including replicasets! It integrates very well by applying the advice I talked about in my post, but not only. It defines binders to enable us using Casbah stuffs in our routing and action definition (ObjectId, and so on).<br />
<br />
This post is not dedicated to explain how to use it, I'd recommend you to browse the project page (<a href="https://github.com/leon/play-salat" target="_blank">play2-salat</a>), plus the wiki that points to relevant URLs.<br />
<br />
<h4>
Goal</h4>
<div>
This post is dedicated to developers teams that follows (or not...) the convention of Continuous Delivery, especially the <a href="http://refcardz.dzone.com/refcardz/continuous-delivery-patterns" target="_blank">Single Command Environment</a> pattern. That is, the environment must be set up in one single command... in Play2 => <i><span style="font-family: 'Courier New', Courier, monospace;">play run</span></i> OR <i><span style="font-family: 'Courier New', Courier, monospace;">sbt run</span></i><br />
<i><span style="font-family: 'Courier New', Courier, monospace;"><br /></span></i></div>
<h4>
Context</h4>
<div>
Create an application that uses MongoDB as (one of its) persistence backend service, use play2-sala to have access to the `<i>ORM</i>` for our object and easy collections connections.</div>
<div>
When runnning in production, of course, a MongoDB instance runs somewhere that can be configured (or a replicatset).</div>
<div>
<br /></div>
<div>
But in Dev?</div>
<div>
<br /></div>
<h4>
Embed Instance</h4>
<div>
When another developer is cloning the related repo, knowing that it's a play application, he's best will would be to enter the directory and launch the application. > BANG <</div>
<div>
No running instance...</div>
<div>
<br /></div>
<div>
So I created a Play2 plugin that uses this <a href="https://github.com/michaelmosmann/embedmongo.flapdoodle.de" target="_blank">amazing work</a> which retrieves a mongodb installer, installs it and enable us to launch/stop it... Keep in mind that MongoDB is not JVM based!</div>
<div>
<br /></div>
<div>
Adding this plugin to the application, setup the dev configuration to starts an embed MongoDB and Play2-Salat to target it, will gives the satisfaction to our developer... Moreover if he is a Designer (the only kind of guy that add values to any app ^^) who don't care about MongoDB, at all!</div>
<div>
<br /></div>
<h4>
How To</h4>
<div>
Add the plugin dependencies (used in <b>PlayProject</b>):</div>
<div>
<br /></div>
<div>
<span style="color: #38761d;"><code></code></span></div>
<div>
<span style="color: #38761d;"> //MY OWN REPO where is deployed the following plugin</span></div>
<div>
<span style="color: #38761d;"> val skalaCloudbeesSnapshots = "Ska La SNAPSHOTS" at "https://repository-andy-petrella.forge.cloudbees.com/snapshot/"</span></div>
<div>
<span style="color: #38761d;"><br /></span></div>
<div>
<span style="color: #38761d;"> //THE NEW PLUGIN => EMBED</span></div>
<div>
<div>
<span style="color: #38761d;"> lazy val embedMongoPlayPlugin = "be.nextlab" %% "play-plugins-embed-mongodb" % "0.0.1-SNAPSHOT" changing()</span></div>
<div>
<span style="color: #38761d;"><br /></span></div>
<div>
<span style="color: #38761d;"> //THE WORTH ONE</span></div>
<div>
<span style="color: #38761d;"> lazy val salatPlayPlugin = "se.radley" %% "play-plugins-salat" % "1.0.8"</span></div>
<div>
<span style="color: #38761d;"><br /></span></div>
<div>
<span style="color: #38761d;"> //DECLARE the deps</span></div>
<div>
<span style="color: #38761d;"> val appDependencies = Seq(</span></div>
<div>
<span style="color: #38761d;"> embedMongoPlayPlugin,</span></div>
<div>
<span style="color: #38761d;"> salatPlayPlugin</span></div>
<div>
<span style="color: #38761d;"> )</span></div>
</div>
<div>
<span style="color: #38761d;"></span></div>
<div>
<br /></div>
<div>
A bit of configuration (<b>application-dev.conf</b>)</div>
<div>
<span style="color: #38761d;"><code></code></span></div>
<div>
<div>
<div>
<span style="color: #38761d;">embed.mongodb.start=true</span></div>
<div>
</div>
<div>
<span style="color: #38761d;">embed.mongodb.version=V2_1_1</span></div>
<div>
</div>
<span style="color: #38761d;">embed.mongodb.port=27017</span></div>
<div>
<span style="color: #38761d;"><br /></span></div>
<div>
<span style="color: #38761d;">mongodb.default.db=meinGot</span></div>
<div>
<span style="color: #38761d;">mongodb.default.host=localhost</span></div>
<div>
<span style="color: #38761d;">mongodb.default.port=27017</span></div>
</div>
<div>
<span style="color: #38761d;"></span></div>
<div>
<br /></div>
<div>
And the most only thing that requires a bit of explanation (in <b>conf/play.plugins</b>)</div>
<div>
<span style="color: #38761d;"><code></code></span></div>
<div>
<div>
<span style="color: #38761d;">600:se.radley.plugin.salat.SalatPlugin</span></div>
<div>
<span style="color: #38761d;">550:be.nextlab.play.mongodb.EmbedMongoDBPlugin</span></div>
</div>
<div>
<span style="color: #38761d;"></span></div>
<div>
See? Yes, the Play2-Salat plugin <b>MUST</b> be started <b>AFTER</b> the embed plugin... of course (what an explanation huh).</div>
<div>
<br /></div>
<h4>
Code</h4>
<div>
The one-single-file-of-33-lines plugin can be forked <a href="https://github.com/andypetrella/play-plugins-embed-mongodb" target="_blank">here</a>.</div>
<div>
<br /></div>
<div>
<hr />
</div>
<div>
<br /></div>
<div>
<span style="font-size: x-large;"><b>That's All, Folks!</b></span></div>
<div>
<br /></div>
Noootsabhttp://www.blogger.com/profile/07479335886992076807noreply@blogger.com0tag:blogger.com,1999:blog-5531553525163261182.post-78990334989176716322012-07-20T22:39:00.003+02:002012-07-20T22:43:00.149+02:00Gatling-Tool in SBT or Play : Sample Projects<h2>
Content</h2>
This post is a direct follow up of this <a href="http://ska-la.blogspot.com/2012/07/gatling-made-easy-for-sbt-orand-play2.html" target="_blank">one</a> where I introduced a bit what I did in order to integrate Gatling in SBT and in Play2.<br />
Where this post was more about bits and bytes necessary to accomplish the task, this one will talk about how to use this mess.
<br />
<h2>
Gatling for SBT</h2>
Now, that we've a dedicated SBT plugin in hand we can create a sample project that uses it (I already created one <a href="https://github.com/andypetrella/gatling-sbt-sample" target="_blank">here</a>).<br />
In this new project, we'll need to create a file <i>plugins.sbt</i> which will contains the reference to the <i>gatling-sbt-plugin</i>.<br />
Actually, it's the classical way to add a plugin to a SBT project (and the easier and the "semanticest").<br />
We're now prepared to configure our build by using the pieces provided by the plugin.<br />
<span style="background-color: white;">At the end, we'll be able to write a first test and launch it.</span><br />
<h3>
Project Build</h3>
First of all, we must create a directory for your project with this basic structure:<br />
<code><script src="https://gist.github.com/3152811.js?file=project-structure">
</script></code><br />
<span style="background-color: white;">Looking at the structure, we can see a </span><i style="background-color: white;">Build.scala</i><span style="background-color: white;"> that will define our project build, a </span><i style="background-color: white;">build.properties</i><span style="background-color: white;"> that defines the SBT version with a single line : </span><span style="background-color: white;"><i>sbt.version=0.11.3</i></span><br />
<span style="background-color: white;">And the latest file in the </span><i>project</i><span style="background-color: white;"> folder is </span><i>plugins.sbt</i><span style="background-color: white;"> which... declares the plugin.</span><br />
<span style="background-color: white;">Let's put the Build.scala aside for now, and have a look at the content of the latter. </span><br />
<script src="https://gist.github.com/3152811.js?file=plugins.sbt">
</script><span style="background-color: white;">Self descriptive isn't it? Yes, we've just told that SBT has to use our gatling plugin... that's all? Not yet.</span><br />
<span style="background-color: white;">Actually this line will provide us everything that have been declared in the plugin, such as the Gatling configuration keys, the command, and the basic SBT settings. </span><br />
<span style="background-color: white;">Now, we're gonna us them within our Build.scala. </span><br />
<span style="background-color: white;"><script src="https://gist.github.com/3152811.js?file=Build.scala">
</script></span><br />
<span style="background-color: white;">Before going in specific details, note that we had to add my own repo (hosted at Cloudbees) in order to fetch the project... but you could also use the URI fetch provided by SBT... </span><br />
<span style="background-color: white;">What are really important to point out is the </span><i style="background-color: white;">allSettings</i><span style="background-color: white;"> declaration and the import of the GatlingPlugin. </span><br />
<span style="background-color: white;">At first, we reuse the default one by using </span><i style="background-color: white;">Project.defaultSettings</i><span style="background-color: white;"> which is being appended the gatling settings, using </span><i style="background-color: white;"><b>Gatling.gatlingSettings</b></i><span style="background-color: white;">... that object comes from our plugin! </span><br />
<span style="background-color: white;">This will add all relevant keys for the "<i>gatling-test</i>" Configuration with default values. </span><br />
<span style="background-color: white;">At this stage we've almost finish our build definition...
All that remain (and shouldn't... have to figure out </span><small style="background-color: white;"><span style="font-size: xx-small;">any help is welcomed?</span></small><span style="background-color: white;">) is to add two things:</span><br />
<ul>
<li>Declare the test framework to have access to the gatling classes and the custom Simulation : <pre>"be.nextlab" %% "gatling-sbt-test-framework" % "0.0.1-SNAPSHOT" % "gatling-test"</pre>
</li>
<li>Declare the command : <pre>commands ++= Seq(gatlingTakatak)</pre>
</li>
</ul>
After having used everything in your project (don't forget to add the <i>GatlingTest</i> configuration...) you're so close to <u>write your first test</u>.
<br />
<h3>
Gatling Conf</h3>
This section will cover the other folder <i>src</i> wherein reside the classical <i>main</i> and <i>test</i> folders.<br />
But there is an other one, <i><b>gatling-test</b></i> that is unknown at this stage.<br />
In fact, this folder will be the root for our Gatling tests, holding the <i>configuration file</i> (I provided a basic one in the sample app) and the <i>future simulations</i> that you'll write.<br />
<br />
The first is dropped in <i>conf/galing.conf</i>, where the latter is simply <i>simulations</i>.
<br />
<h3>
Write your first test</h3>
At this stage, I'll make an assumption that you either know the Gatling api... Nah, the Gatling DSL is sufficiently self descriptive (otherwise check the <a href="https://github.com/excilys/gatling/wiki/HTTP" target="_blank">wiki</a>).<br />
<script src="https://gist.github.com/3152811.js?file=SampleSimulation.scala">
</script><br />
As you can see, it's very very easy (<i>that's why I like Gatling-Tool</i>).<br />
All we had to do, since we're benchmarking <a href="https://www.google.com/" target="_blank">google</a>, is to implemnent the <i><b>apply</b></i> method of a simple scenario within a class (with must have an empty constructor... <u>a limitation for now</u>) that must extend our custom Simulation <span style="font-size: xx-small;">(yeah! I know ... I'll choose another name for it that's why <i>GSimulation</i> is needed)</span>.<br />
Note the dummy implementations of the interceptors whose aren't needed here.<br />
This Simulation must be located somewhere under the <i>simulations</i> folder.<br />
<br />
Now, we can launch it and see where are stored the results.<br />
<br />
<code>
</code>
That's enough boy...<br />
Let it shot the server and generate the result.<br />
...<br />
...<br />
Done? ok.<br />
<br />
Now, that all bullets have been shot, you can go in this folder <i><b>gatling-project/target/gatling-test/result</b></i>.<br />
See? True! Every run will create a specific folder under this one, named <i>run-${scenario-name}-time</i>.<br />
<span style="background-color: white;">Make some jump in it and locate the </span><i style="background-color: white;">index.html</i><span style="background-color: white;"> file, open it in your browser and see the magic happened. </span><br />
<span style="background-color: white;">NB: check <a href="https://github.com/excilys/gatling/wiki/Reports" target="_blank">this page</a> for information about the reports. </span><br />
<span style="background-color: white;">So far, so good. But we're writing web apps and we want to stress them... we all know that the google server won't crash (that easy). </span><br />
<span style="background-color: white;">Let's move to <b>Play2</b> (or you can write your own app w/o Play2 using Spray or another Java framework maybe, then mimic the following).</span><br />
<h2>
Gatling for Play2</h2>
For this part, I also wrote an <a href="https://github.com/andypetrella/gatling-play2-sample" target="_blank">application</a> that makes use of the "plugin" introduced in the previous post.
<br />
It's quite complicated, not yet polished and might be a good topic for a future post, because I tried to make an application based on <b>Event Sourcing</b> with some workflows and usages of pure functional structure like <b>Lens</b>, <b>Writer</b>, <b>State</b>, <b>Monad</b> and so forth.<br />
<br />
But let's stick with the actual topic, if you fork the project and browse a bit its build configuration you'll see that it is near from the one we just talked about. <span style="background-color: white;">But still that some particularities are presents...</span><br />
<h3>
Definition</h3>
First of all, since Play2 has its own convention for sources folders (<i>app</i>, <i>test</i> under the root) we'll redefine those folders for gatling as well.<br />
In order to do that, we must <u>adapt</u> the following configuration keys:<br />
<span style="background-color: white;"><script src="https://gist.github.com/3152811.js?file=adaptKeys.sbt">
</script></span><br />
<span style="background-color: white;">Actually, yeah, we've just skipped the "src" folder. Anyway. </span><br />
<span style="background-color: white;">So far, so good... </span><br />
<span style="background-color: white;">I heard you... </span><br />
<span style="background-color: white;">But, do you known that we're already done? </span><br />
<span style="background-color: white;"><i>Allez</i>, let's write a test.</span><br />
<h3>
Write</h3>
If you started from scratch (without forking the sample app) you can now stress test your <i><b>index</b></i> page that will render the default <i><b>welcome</b></i> page defined in Play2.<br />
See the below Gist for that, or its follower for a more complicated one using the app.<br />
<span style="background-color: white;"><script src="https://gist.github.com/3152811.js?file=simple.scala">
</script></span><br />
<script src="https://gist.github.com/3152811.js?file=CreateAndLogin.scala">
</script><br />
<span style="background-color: white;">Look what I've done Dad! </span><br />
<span style="background-color: white;">In the Simulations, all we had to do is to simply declare an available <i>port</i> on which the framework can start the server, and its needed <i>FakeApplication</i> to have access to the routing. </span><br />
<span style="background-color: white;">Then when we had to build a request path, we've just had to do the same as in our templates, that is, use the </span><i style="background-color: white;">routes</i><span style="background-color: white;"> object under the controller package. <span style="font-size: x-small;">Amazing</span>. </span><br />
<span style="background-color: white;">Furthermore, we can also reuse our type checked </span><i style="background-color: white;">Form</i><span style="background-color: white;"> to build our feeder. <span style="font-size: x-small;">Waouw</span>. </span><br />
<span style="background-color: white;">But, at this stage, there still some boilerplate (parameters). <span style="font-size: x-small;">Looser</span>.</span><br />
<h3>
Launch</h3>
And now, it's time to stress test our server and see how it'd behave on production. (Ok, that's not 100% true, since Gatling is running on the same machine...)<br />
<br />
For that, just do the following:<br />
<script src="https://gist.github.com/3152811.js?file=play.console">
</script><br />
<code></code>Wait.
Again.<br />
<br />
Now, go to you result directory (the path hasn't changed). <span style="background-color: white;">Open in your browser and let the shine comes in.</span><br />
<h2>
Wrap Up</h2>
At this stage, with small and few injuries we can use <a href="http://gatling-tool.org/" target="_blank">Gatling-Tool</a> without having to install or trick stuffs in order to have information about our web app.<br />
<span style="background-color: white;">This by using either a full SBT application or a Play2 based one.</span><br />
<span style="background-color: white;">I know that there still a lot to do, but at least the basic features are there, and I'll enjoy anyone forking it, scrapping it if necessary and let me know.</span><br />
<h2>
Last words</h2>
I hope you liked these posts, otherwise I'd like to apologize because I did all projects and the blogs on the train, half asleep every morning and evening, back and forth from my actual mission's workplace.<br />
<br />
Now I'm gonna sleep in this train...<br />
Oh no, I can't...<br />
I've to write my book now.Noootsabhttp://www.blogger.com/profile/07479335886992076807noreply@blogger.com0tag:blogger.com,1999:blog-5531553525163261182.post-40137367377083796872012-07-20T22:36:00.000+02:002012-07-20T22:40:50.792+02:00Gatling made easy for SBT or/and Play2<h2>
Forewords</h2>
These last few weeks, I took some time to understand a level deeper how SBT works and what it can provide.
Since this post is not related to this learning trip (which was along existing blogs and wikis), I'll jump directly to the idea this new understanding gave me.<br />
A while ago I started (and paused) a work on Gatling-Tool to have it integrated with Play 2.0 (see these posts <a href="http://ska-la.blogspot.com/2012/03/gatling-tool-plugin-for-play-20.html" target="_blank">here</a> and <a href="http://ska-la.blogspot.com/2012/04/gatling-and-play20-continued.html" target="_blank">here</a>), this work has been refactored to better integrate with SBT.
<br />
<h2>
What we'll find in this post</h2>
In this post, I'm gonna give you all tools in order to stress test you Web based application, either built upon Play 2.0, either only using SBT. By directly starting writing your tests rather than having to configure stuffs or to start others...
So this post is composed of two parts:
<br />
<ul>
<li>How I built these tools (can be skipped -- definitively)</li>
<li><a href="http://ska-la.blogspot.be/2012/07/gatling-tool-in-sbt-or-play-sample.html" target="_blank">How to use these tools</a> (it's probably the useful part)</li>
</ul>
<h2>
Gatling shot 3 aside projects... for goods</h2>
This part is related to what was necessary to have gatling-tool easily integrated with SBT, and after with Play2.
<br />
<h3>
Gatling SBT Test Framework</h3>
This project (on <a href="https://github.com/andypetrella/gatling-sbt-test-framework" target="_blank">github</a>) has a sole and simple goal, to implement the Test Interface (<a href="https://github.com/harrah/test-interface" target="_blank">see</a>) used by SBT to integrate new Test Framework (like was done for ScalaTest, Specs, JUnit, and...).
This project contains only 2 classes and one convenient trait.
<br />
<h4>
GatlingBootstrap</h4>
This class is one of the helper that avoid boilerplates in the future, because it starts the GatlingConfiguration stuff needed by Gatling-Tool to execute.
This class requires two things "the path to the gatling configuration file" and "the path to the folder where will be stored the stress results".
<br />
<h4>
Simulation</h4>
This trait is another tool in hand that extends the <i>com.excilys.ebi.gatling.core.Predef.Simulation</i> provided by Gatling in order to add it two interceptors whose are <i>pre</i> and <i>post</i>.
Those interceptors will be really useful in the future, in order to start/stop a server for instance (e.g. <i>FakeServer</i> in Play2).
<br />
<h4>
GatlingTestFramework</h4>
This class is the real processor of the project.
Actually, it's also the implementation of the interfaces declared by the SBT test framework which are <i>Framework</i>, <i>Runner2</i> and some <i>Fingerprint</i>s.
Basically, what is done there are those tasks:
<br />
<ul>
<li>Create the gatling bootstrap based on sys properties</li>
<li>Declare fingerprint to discover Stress Test based on the parent class being <i>be.nextlab.gatling.sbt.plugin.Simulation</i></li>
<li>Create the stress tests instances (only classes are handled for now) by reflection</li>
<li>Call the <i>pre</i> interceptor</li>
<li>Execute the stress test using the gatling api</li>
<li>Call the <i>post</i> interceptor</li>
<li>Generate the reports using the gatlin api</li>
</ul>
<h3>
Gatling SBT Plugin</h3>
This project (<a href="https://github.com/andypetrella/gatling-sbt-plugin" target="_blank">here</a> on github) aims to provide the basics to use the test framework easily within an SBT project<span style="background-color: white;">. </span><br />
<span style="background-color: white;">Its intent is to have testers able to write stress tests directly, just after having imported the module in SBT. </span><br />
<span style="background-color: white;"><br /></span><br />
<span style="background-color: white;">It basically provides an SBT Configuration (</span><b style="background-color: white;">gatling-test</b><span style="background-color: white;">) that extends the SBT's Test one and one Command (</span><b style="background-color: white;">takatak</b><span style="background-color: white;">).
The other goodies are <i>settings</i> which are common for all such stress tested projects using gatling. </span><br />
<span style="background-color: white;">Those settings are essentially the conf file path, the result directories, the libraries (gatling ones principally) and so on.
The last important thing that it does is to add the gatling test framework to the already existing list of test frameworks supported out of the box by SBT.</span><br />
<h3>
Gatling Play2 Plugin</h3>
This misnamed project <small><span style="font-size: xx-small;">(no more a Play2 plugin, neither a SBT one... but it's ok for now)</span></small> brings only one little thing.
It comes with a base implementation the custom <i>Simulation</i> declared in the Gatling Test Framework.<br />
<br />
This base implementation is Play2 specific while creating a fake server and starting it in the <i>pre</i> interceptor, stoping it in the <i>post</i> one.<br />
<br />
This will ease further tests in the Play2 environment.
<br />
<h2>
Save Point</h2>
So far so good, if you read this you can now navigate the small projects that were necessary to do what we'll discuss in the next blog...<br />
<br />
This one is already too long...Noootsabhttp://www.blogger.com/profile/07479335886992076807noreply@blogger.com0tag:blogger.com,1999:blog-5531553525163261182.post-35133401633918458162012-06-16T18:38:00.000+02:002012-06-16T18:38:29.273+02:00Type-safed Composable Interceptors in Play2.0This post is just a quick follow-up of this <a href="http://ska-la.blogspot.be/2012/06/attempt-of-play20-action-interceptor.html" target="_blank">post</a>, which introduced my latest utility for Play 2.0.<br />
I recommend you to have a quick overview on it.<br />
<br />
<br />
<h2>
Type-safe composition of interceptors : Premises</h2>
<br />
Briefly, we'll just see how the first "future work" as been addressed. That is, <i>avoid boilerplate for interceptors composition</i>.<br />
<br />
A quick recall, when we had to compose such <i>Interceptors</i> we had to take care our-selves of the validation results and the combined result (tuple or whatever).<br />
The real problem that is under the sea here, is that tuples are not so easily generalizable (no append method, roughly).<br />
<br />
So I decided to use the Shapeless library (thx to @milessabin).<br />
<br />
Shapeless has an amazing core structure that enables type-and-value chaining (somehow). The <i>HList</i> type is a kind of list but each element is one <i>value</i> and one <i>type</i>. For instance, it has the <i>head value</i> and the <i>Head type</i> on top of the Stack. Here is the kind of stuffs that we can do with Shapeless:<br />
<code></code><br />
<code><i>val | = false</i></code><br />
<code><i>val thisIsNotAPipe = "this" :: 15 :: false :: "a" :: | :: HNil</i></code><br />
<code><br /></code><br />
<code><b>> thisIsNotAPipe: shapeless.::[java.lang.String,shapeless.::[Int,shapeless.::[Boolean,shapeless.::[java.lang.String,shapeless.::[Boolean,shapeless.HNil]]]]] = this :: 15 :: false :: a :: false :: HNil</b></code><br />
<br />
<i>>Type-safe list of stuffs<</i><br />
<i></i><br />
<h2>
<i>In Action...</i></h2>
<br />
While trying to use <i>HList</i> generically I had problem with the implicits that are needed to prepend two lists, but StackOverflow has brought me the answer <a href="http://stackoverflow.com/questions/11036334/shapeless-prepend-implicit-not-found/11037915" target="_blank">here</a>.<br />
I'm not gonna tackle here how I did, by I'll demonstrate what is now possible with the new composition functionalities added to <i>Interceptors</i>.<br />
<br />
First of all, I had to declare an Interceptor for HList and the related implicit conversion. After what, I added three methods:<br />
<br />
<br />
<ul>
<li><b>hlist</b> this method on <i>Intercept</i> is able to convert a classical one to a <i>HList</i> one</li>
<li><b>~::~ </b>this one is available to compose any interceptors that is not defined using HList with an HList one. It will create a new Interceptor with the new composed HList as result</li>
<li><b>~:::~ </b>this one enable to compose two Interceptor defined with HList, the results will be the concatanation of the two HList.</li>
</ul>
<div>
Note: concatanation of HLists preserves the type sequence, actually we can see that as if it concat two lists of values and two list of types.</div>
<div>
<br /></div>
<div>
Let's see how we can deal with them:</div>
<div>
<script src="https://gist.github.com/2941878.js?file=security.scala">
</script></div>
<div>
<br /></div>
<div>
Easy no? Combine interceptors and use compile-time type-checking to validate the required kinds of items.</div>
<div>
<br /></div>
<div>
<b>Without boilerplate</b></div>
<div>
<br /></div>
<br />
<br />Noootsabhttp://www.blogger.com/profile/07479335886992076807noreply@blogger.com0tag:blogger.com,1999:blog-5531553525163261182.post-66163269321319569072012-06-15T23:10:00.001+02:002012-06-15T23:10:41.020+02:00An Attempt of Play2.0 Action InterceptorIn this post, I'm gonna introduce a piece of code that I laid with the help of two stuffs.<br />
The <a href="https://github.com/playframework/Play20/blob/master/framework/src/play/src/main/scala/play/api/mvc/Security.scala" target="_blank">Security</a> object provided by Play 2.0 out of the box and somehow the <a href="https://github.com/typesafehub/play-plugins/blob/master/util/src/main/scala/com/typesafe/plugin/package.scala" target="_blank">Secured</a> one provided by the TypeSafe's plugin <i>util</i>.<br />
<br />
The idea here is to leverage the actual functionality wich only enables to provide one <i>Option[String]</i> as result.<br />
What I wanted at first is to satisfy my use case, which is : <i>a lot of actions are secured and I need a username </i><i style="font-weight: bold;">and its id</i>. Where <b>id</b> is an Int.<br />
<br />
Here we are, the actual <i>Security</i> and <i>Secured</i> don't permit me to have several extracted values, or to have an Int.<br />
<br />
That's why I created this project : <a href="https://github.com/andypetrella/steal-play2">https://github.com/andypetrella/steal-play2</a>.<br />
<br />
<br />
<h2>
Steal <i><a href="http://en.wikipedia.org/wiki/Steal_(basketball)" target="_blank"><span style="font-size: xx-small;">help</span></a></i></h2>
<br />
<br />
The idea is to enable any actions to be preceded by some interceptors that are stealing values either from the request, the cache, the database (or...) and set them as the parameters of a closure that outputs an Action.<br />
<br />
But we must also care of the cases when something went wrong during the stealing.<br />
<br />
<br />
<h3>
Interceptor</h3>
<br />
<br />
This trait is the core of the solution, it defines:<br />
<br />
<ul>
<li>the <b>stealing</b> operation: a function that takes (currently) the request and outputs a validated output (Validation from ScalaZ)</li>
<li>the <b>err</b> callback: a function that takes the request and the failure (when computing the stolen value) and outputs a valid Result to send back to the client.</li>
<li><b>apply</b>: a closure that takes the result (not a Validation) and output an <i>Action</i>.</li>
</ul>
<div>
At this stage and looking at the code below (I've omitted the apply impl because not important here -- just wrapping and unwrapping), it's fairly simple to know how to define such interceptor.</div>
<div>
<script src="https://gist.github.com/2938623.js?file=interceptor.scala">
</script></div>
<div>
<br /></div>
<div>
I provided the simplest implementation of this trait that is a case class extending it by defining the two callbacks as fields. Like so:</div>
<div>
<script src="https://gist.github.com/2938623.js?file=intercept.scala">
</script></div>
<div>
<br /></div>
<div>
So now we're armed to do stuff like that:</div>
<div>
<script src="https://gist.github.com/2938623.js?file=both.scala">
</script></div>
<div>
<br /></div>
<div>
Where we defined two interceptors that takes a string from the cache and an Int form the cache also (just an illustration). After what, we combined them in an another <i>Intercept</i> using a for-comprehension.</div>
<div>
<br /></div>
<div>
So far, so good, now how to use such combined interceptor within an <i>Action</i>. Check this out:</div>
<div>
<script src="https://gist.github.com/2938623.js?file=bothAction.scala">
</script></div>
<div>
<br /></div>
<div>
<h3>
Monoid</h3>
</div>
<div>
With the help of Monoid (from ScalaZ) and if the case permits it, I defined an implementation of <i>Interceptor</i> that append the successive results one after the another. Reducing and simplifying the composition. Like so:</div>
<div>
<script src="https://gist.github.com/2938623.js?file=dummyMonoidComposition.scala">
</script></div>
<div>
<br /></div>
<div>
Note: we used the classical case class. Thanks to an <i>implicit def</i> that wraps the <i>Intercept</i> into the Monoid implementation, using the TypeClass bound declaration. <a href="https://github.com/andypetrella/steal-play2/blob/master/app/steal/intercept.scala#L106" target="_blank">see</a></div>
<div>
<br /></div>
<div>
<h2>
What will come next</h2>
</div>
<div>
The next step I've already started is to try using <a href="https://github.com/milessabin/shapeless" target="_blank">Shapeless</a> to avoid the composition boilerplate. Things are ongoing. Stay tune for that.</div>
<div>
<br /></div>
<div>
One step further, I'll add another parameter to the <i>steal</i> callback, which will be the optional result of the previous computations of other interceptors. That in order to combine them at the function and result levels.</div>
<div>
<br /></div>
<div>
And probably, add all boileplate my self that creates the tuples out-of-th-box in the compose function.</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<b><i>That's all folks!</i></b></div>Noootsabhttp://www.blogger.com/profile/07479335886992076807noreply@blogger.com0tag:blogger.com,1999:blog-5531553525163261182.post-73319802041200908142012-06-11T11:45:00.000+02:002012-06-11T11:45:04.135+02:00How Monad Transformer saved my time<br />
<h2>
Context</h2>
<br />
These days (this week-end) I wanted to put some work on a <a href="https://github.com/andypetrella/neo4j-rest-play-plugin" target="_blank">Neo4J Rest driver</a> that I'm writing for Play 2.0 in Scala.<br />
The only thing I wanted, actually, is to have the current embryo more functional. I mean <i>I was throwing exceptions</i> (frightening... huh!).<br />
Since this driver is meant to be fully asynchronous (man... it's http, it MUST be) and non-blocking (Play's philosophy), I was hardly using the <code>Promise</code> thing via the use of the WS api of Play.<br />
This is the kind of thing I've got (briefly):<br />
<code></code><br />
<code> def getNode(id:Int) : Promise[Neo4JElement]</code><br />
<br />
Where <code>Neo4JElement</code> stands for the wrapper of all Neo4J Rest response (Json). Hence, it can be either a <b>Failure</b> response (Json with stacktrace and message), it can be a <b>Node</b>, or .... throw an <b>Exception</b> (br...) when the service crashed (f.i.).<br />
<br />
Hmm, not so intuitive and goes against the functional paradigm that orders: "you can't ever introduce side-effects, boy!". An exception that blows in my face, is one side effect (head-aches, ...).<br />
<br />
<h2>
<strike>Diego</strike> Validation to the rescue</h2>
<br />
Validation is a very simple thing, it holds either a value, either an error...<br />
Ok, why not just <code>Either</code> then. Actually, you're right but <code>Validation</code> that I took in the ScalaZ library contains a lot of thing very helpful for the purpose of validation. But if you worry it, just replace <code>Validation</code> by <code>Either</code> in your mind from here.<br />
Now, here is the <code>getNode</code> signature:<br />
<code></code><br />
<code> def getNode(id:Int) : Promise[Validation[Aoutch, Node]]</code><br />
<br />
Isn't it more intuitive? For sure, you get back our relevant type in the signature : <code>Node</code>. Great!<br />
<br />
So far, so good now what the heck is <code>Aoutch</code> : something that hurts... and what could hurt a runtime execution... exceptions yeah! Thus, <code>Aoutch</code> is just a shortcut for <code>Either[NonEmptyList[String], Failure]</code>. We can see that we represent with one single type an <i>unexpected exception</i> and a <i>failure</i> (missing node f.i.).<br />
<br />
<h2>
Monad</h2>
<br />
<small>If you don't know what a Monad is, from here thing about a structure that can evolve in a for-comprehension (in scala it must implement <code>flatMap</code> and <code>map</code>).</small><br />
<br />
<code>Promise</code> and <code>Validation</code> are Monads. And their used one over the another. But what really interests us is the leaf of the chain <code>Node</code>. That introduced some boilerplate code when you try to sequence actions like that:<br />
<script src="https://gist.github.com/2909215.js?file=without.scala">
</script><br />
See... yes we have to skip the first level (validation) to be able to work with the meaningful objects.<br />
But still that we can extract some pattern... no?<br />
<br />
<h2>
Monad Transformer</h2>
<br />
The pattern that we can extract is kind of two-level composition. I did this composition my-self trying to figure out what we'll be possible.<br />
It was successful but, I've have to introduce a new type and a new method, that was like a <code>flatMap</code>.<br />
So I asked on StackOverflow ^^ (and it was my first question, yeepeee). You can find it <a href="http://stackoverflow.com/questions/10968057/async-computation-with-validation-in-scala-using-scalaz" target="_blank">here</a>. So I want explain how I did, because the question explains it. But the real question was, <b>is there a well-known functional construction for this problem?</b>.<br />
Thanks to @purefn, I knew that it was the case!<br />
<br />
It was time to use <i>Monad Transformer</i>.<br />
<br />
<h2>
Monad Transformer</h2>
<br />
Briefly (and very roughly), a Monad Transformer is a construction that is able to transform a <code>M[N[_]]</code> into a <code>P[_]</code>, where M, N, P are Monads.<br />
I won't explain here how it does, because it would be long, but here is a good <a href="http://en.wikibooks.org/wiki/Haskell/Monad_transformers" target="_blank">link</a> (you've to understand Haskell a bit, sorry).<br />
With the help of such transformer, you'll get you back the opportunity to use for-comprehension... with the wrapped-twice type as bound value.<br />
Here is the the transformer for our <code>Promise[Validation[E, A]]</code>:<br />
<script src="https://gist.github.com/2909215.js?file=monadTransformer.scala">
</script><br />
And how we can now link nodes:<br />
<br />
<script src="https://gist.github.com/2909215.js?file=with.scala">
</script><br />
<br />
Awesome! No! We get 3 async and non-blocking calls, totally typed checked and resistant to exceptions and failures... In 5 lines.<br />
<br />
<h2>
Future work</h2>
<br />
At the SO question, @purefn tolds me that scalaz 7 (snapshot) is defining (fully) this kind of <i>Validation Transformer</i>.<br />
Why didn't I used it, yet:<br />
<br />
<ul>
<li>I'm trying to use not Snapshot (not a good reason, but still)</li>
<li>In order to use <code>ValidationT</code>, I'll have first to create an instance of the Type Class <code>Monad</code>. Because the <code>flatMap</code> signature needs it in the context.</li>
</ul>
<h2>
Conclusion</h2>
<br />
I love functional (even if I'm still learning -- back -- the basis ^^)<br />Noootsabhttp://www.blogger.com/profile/07479335886992076807noreply@blogger.com4tag:blogger.com,1999:blog-5531553525163261182.post-84883125402263090202012-04-22T17:03:00.000+02:002012-04-22T17:03:10.736+02:00Gatling and Play2.0: continuedThis blog entry is a follow-up of this <a href="http://ska-la.blogspot.com/2012/03/gatling-tool-plugin-for-play-20.html" target="_blank">entry</a> where I introduced a spike I did on Play 2.0 stress tested using <a href="http://gatling-tool.org/" target="_blank">Gatling-tool</a>.<br />
<br />
At the time writing the above entry, I had to quickly hack gatling to use Akka 2.0 as Play 2.0 uses it, and I didn't wanted to have clashes.<br />
<br />
But, thanks to <a href="https://twitter.com/#!/slandelle" target="_blank">Stéphane Landelle</a>,<b> Gatling is now Akka 2.0 enabled </b>(since 1.1.2).<br />
<br />
So that, it was time to give the <a href="https://github.com/andypetrella/gatling-play2-plugin" target="_blank">plugin</a>'s embryo a refresh. For that I used another <a href="https://github.com/andypetrella/neo4j-play2.0-plugin-test" target="_blank">project</a> that aims testing the <a href="https://github.com/andypetrella/neo4j-rest-play-plugin" target="_blank">neo4j rest plugin</a> for Play2.0 I'm also writing. In case of, I've also introduced what I did in this project in this <a href="http://ska-la.blogspot.com/2012/04/still-playing-but-new-players-are-in.html" target="_blank">post</a>.<br />
<br />
<h2>
Content</h2>
Using the Gatling I first tested how I've been able to stress test:<br />
<br />
<ul>
<li>simple (get) urls</li>
<li>mutating (post) urls using server <a href="http://www.playframework.org/documentation/2.0/ScalaForms" target="_blank">form</a> underground to compile excepted data.</li>
<li>duration tests (stress testing on a given period basis)</li>
</ul>
<div>
What I liked so much, even in this embryos+, is the ease to create stress tests when coupled with Play2.0 functionalities and Gatling's DSL.</div>
<div>
<h2>
Foreword</h2>
</div>
<div>
The Gatling plugin I'm currently building is located on github <a href="https://github.com/andypetrella/gatling-play2-plugin" target="_blank">here</a> and is based on sbt to build it.</div>
<div>
<br /></div>
<div>
But it's based on the version 1.1.4-SNAPSHOT version of Gatling's libraries (due to some fixes the Stéphane did "for me", while he was at devoxx fr, isn't he gentle !!!). </div>
<div>
<span style="background-color: #f3f3f3;"><i>At the time writing, you'll have to build <a href="https://github.com/excilys/gatling" target="_blank">gatling</a> and <a href="https://github.com/excilys/gatling-highcharts" target="_blank">gatling-highcharts</a> locally using maven (quick fast!).</i></span></div>
<div>
<br /></div>
<div>
<h2>
How to</h2>
</div>
<div>
<h3>
Set up</h3>
</div>
<div>
Having created a Play 2.0 app, you now have to powerful of sbt in hands (especially if you've installed the TypeSafe stack 2.0). So, to stress test your app, you'll have first to build to plugin I told above.</div>
<div>
<h4>
Plugin</h4>
</div>
<div>
First of all, clone/fork this library project on github <a href="https://github.com/andypetrella/gatling-play2-plugin" target="_blank">plugin</a>, after what, you'll just have to run <code>sbt publish-local</code> in the related folder. That's it, you now have the plugin in your local repo.</div>
<div>
<h4>
Project</h4>
</div>
<div>
In your Play 2.0 app, you now are able to define the plugin as `test` dependency, using the following</div>
<div>
<script src="https://gist.github.com/2464268.js?file=deps.scala">
</script></div>
<div>
That's it...</div>
<div>
<br /></div>
<div>
<h3>
Use</h3>
</div>
<div>
<h4>
Specs</h4>
</div>
<div>
Personally, for testing I use <i>org.specs2.Specification</i>, for using it with the plugin (at the time writing) you'll have to create the following:</div>
<div>
<script src="https://gist.github.com/2464268.js?file=boilerplate.scala">
</script></div>
<div>
This listing is creating a fake server to enable urls to be tested, and some functions to deal with it.</div>
<div>
<br /></div>
<div>
To create the server, the plugin defines a Util object (<i>be.nextlab.play.gatling.Util</i>), which also defines rough helpers to be used in stress tests.</div>
<div>
<br /></div>
<div>
A full spec should look the following:</div>
<div>
<script src="https://gist.github.com/2464268.js?file=gatling-specs.scala">
</script></div>
<div>
<br /></div>
<div>
<h4>
Simple Url</h4>
</div>
<div>
You saw, in the provided gist above, that Util also defines a way to simply defines a gatling Simulation (basically a Function0 that returns a scenario builder: Gatling DSL result).</div>
<div>
<br /></div>
<div>
Having that in hand, here is the fragment to stress test the root url:</div>
<div>
<script src="https://gist.github.com/2464268.js?file=root_stress_test.scala">
</script></div>
<div>
<br /></div>
<div>
As you can see, it's pretty simple, but nothing can really be checked in the body of the specs (I'm working on having relevant information to check).</div>
<div>
But, at least you can run this test to hit 10 times the root url ramping the numbers of users by 2.</div>
<div>
<br /></div>
<div>
Running it (<i>sbt test</i>) you'll have a new <i>gatling</i> folder in your target folder that contains a <i> results</i> directory where are located all stress results in an html report (with great charts)</div>
<div>
<br /></div>
<div>
And all you had to do is to define the request headers and the url...</div>
<div>
<br /></div>
<div>
<h4>
Mutating the server</h4>
</div>
<div>
If you have controllers that mutates the server, you should have define POST urls, which are using the Form feature provided by Play 2.0.</div>
<div>
<br /></div>
<div>
Having did so, you'll be able to stress it very easily using Map, JsObject or in the best case your <b>Model</b>.</div>
<div>
<br /></div>
<div>
Let's say we have a controller <code>controllers.Stuffs</code> that uses case classes <code>models.Stuff</code>. The controller defines a <i>stuffForm</i> and a <i>createStuff</i> action.</div>
<div>
<br /></div>
<div>
Your stress test can now be defined like the following:</div>
<div>
<script src="https://gist.github.com/2464268.js?file=mutating-server.scala">
</script></div>
<div>
<br /></div>
<div>
In the gist, you can see 5 points to note, they are key-clues to create reusable stress tests.</div>
<div>
<br /></div>
<div>
Nothing is really hardcoded, neither the path to the http end point nor the parameters data.</div>
<div>
<br /></div>
<div>
<b>That's http stress testing using <span style="background-color: yellow;">type checked</span><span style="background-color: white;"> </span>requests</b>. CoOl isn't it? Hey, man, we got back our lovin' type cheking (one of the best scala feature).</div>
<div>
<br /></div>
<div>
<h4>
Heavy check, duration based</h4>
</div>
<div>
This part is more a Gatling feature highlighting.</div>
<div>
<br /></div>
<div>
This last example is an heavy test that uses looping over a configuration for a given period. This gives you how many users could use your application.</div>
<div>
<br /></div>
<div>
Such test might be shaped the following:</div>
<div>
<script src="https://gist.github.com/2464268.js?file=heavy_test.scala">
</script></div>
<div>
<br /></div>
<div>
<h2>
Note</h2>
</div>
<div>
For now, it's <b>NOT</b> an official plugin neither a gatling nor a play 2.0 one, but discussions are on the way for that... stay tuned on <a href="https://twitter.com/#!/noootsab" target="_blank">twitter</a> or here.</div>Noootsabhttp://www.blogger.com/profile/07479335886992076807noreply@blogger.com0tag:blogger.com,1999:blog-5531553525163261182.post-30667283971779522222012-04-18T00:11:00.000+02:002012-04-18T00:11:09.773+02:00Still Playing... but new players are in<h2>
Why?</h2>
Because I love to play, with <a href="http://www.playframework.org/" target="_blank">Play 2.0</a> and scala (still learning).<br />
But also because I'm currently investigating technologies that I might choose for a new product line currently building in my company, <a href="http://www.nextlab.be/" target="_blank">NextLab</a> (in Belgium).<br />
<br />
<h2>
With ?</h2>
This time, I've more played with client side libraries or frameworks (no I won't post yet another entry on JQuery...), but I also tried how it is easy to create totally async code (so parallelized) using server side ones.<br />
The technologies that will be quickly introduced in this post can be found hereafter, but everything has been packaged in a <a href="https://github.com/andypetrella/neo4j-play2.0-plugin-test" target="_blank">github</a> repo, and a running <a href="http://neo4j-play20-plugin-test.herokuapp.com/" target="_blank"><span style="background-color: yellow; color: blue;"><b>instance</b></span></a> on <a href="http://www.heroku.com/" target="_blank">Heroku</a>.<br />
<br />
At first the current spike was dedicated to the slowly growing <a href="https://github.com/andypetrella/neo4j-rest-play-plugin" target="_blank">Play 2.0's Neo4J REST plugin</a>, we are creating at NextLab. But, in order to demonstrate what could be done with the coupling of these two technologies, I've extended the spike's scope to something more funny.<br />
<br />
So let's introduce the technologies:<br />
<h4>
Client Side</h4>
<div>
<h5>
<a href="http://twitter.github.com/bootstrap/">Twitter Bootstrap</a></h5>
An amazing toolbelt helping building responsive website without having to bother boilerplates in HTML, CSS. <br />
<br />
Even if it is neat, complete and well thought, Bootstrap comes with another handy factor... it has been built using LESS. And by chance (I know, chance is not part of the equation) LESS is supported by Play 2.0 by default.<br />
<br />
Just a note, LESS will let you reuse color codes, mixins, etc etc that Bootstrap has already defined.<br />
<h5>
<a href="http://spinejs.com/">Spine.js</a></h5>
As we'll below, we'll have to discuss with WS (json), upon which a REST interface has been added for meaningful resources.<br />
<br />
That's where Spine.js comes in the game. This lightweight MVC library brought me the small tools that I needed for fetching, saving resources without having to write not even a single request by hand...<br />
<h5>
<a href="http://mbostock.github.com/d3/">d3.js</a></h5>
Probably my favorite (that might be my mathematician part who's talking), this powerful Data-Drive Document toolkit has taken the right thing by the right end.<br />
<br />
Its functional approach of decoupling data from the document, and link them using layouts helps you to concentrate on each part of the data usage independently:<br />
<ul>
<li>the incoming/rought data mapping to a representative data</li>
<li>the mapping from represented data to document (most of the time, one data for one element)</li>
</ul>
</div>
<h4>
Communication</h4>
The communication layer is of course HTTP, with a little help from the emerging HTML5 features. One in particular, Server-Sent Events (<a href="http://www.html5rocks.com/en/tutorials/eventsource/basics/">here</a> is a great intro).<br />
<br />
This, stable, HTML5 feature comes with the handy functionality to let the server sent events to connected client, without hacks; that said Comet or Polling.<br />
<br />
Open a connection, push data, and that ONLY when the server needs to.<br />
<div>
<h4>
Server Side</h4>
<h5>
<span style="font-size: small;">
Play 2.0 (Scala)</span></h5>
Of course... But I used some "advanced" feature like,<br />
<h6>
<span style="font-size: x-small;">
Async</span></h6>
Async is a Play 2.0 feature that let the server deal with the tasks it has to schedule.<br />
<br />
That is, when you think that the server might have to wait for actions being executed before being able to respond a request, Play 2.0 let you, really simply, create Async request (non blocking).<br />
<br />
Very handy when you have to call third parties services for instance...<br />
<div>
<h6>
<span style="font-size: x-small;">
Iteratee</span></h6>
</div>
<div>
The only way I would consider from now to consume data. Iteratee is a fairly difficult thing to understand (read this <a href="https://github.com/playframework/Play20/wiki/Iteratees">wiki</a>) but it gives you the same smart decomposition that in d3.js, that is, decoupling the management of a sources, its useful representation and its computed result.</div>
<h5>
<a href="http://akka.io/"><span style="font-size: x-small;">Akka</span></a></h5>
Powerful, actor-based, parallelizer, asynchronous task, scheduler library.<br />
<br />
Actually I needed, a request to launch async tasks (you know like event generation and dispatching)... so what else!<br />
<h5>
<a href="http://neo4j.org/">Neo4J</a></h5>
A database for storing graph... let's use a graph database.<br />
<br />
Within NextLab, we started a open sourced <a href="https://github.com/andypetrella/neo4j-rest-play-plugin">Play2.0 plugin</a> for calling the REST api provided by a Neo4J Server (helpful on Heroku). It's still emerging, and continue to evolve a lot because features are implemented on the fly (need), and a re-pass is forecasted to add a meaningful DSL (like <a href="https://github.com/FaKod/neo4j-scala">FaKod</a> did).<br />
<br />
Why do we make this choice to implement a distinct plugin instead of wrapping the java library?<br />
<ol>
<li>yet another library, which brings me too much (I need REST only)</li>
<li>I want requests being async and under control</li>
</ol>
</div>
<div>
<h4>
Libraries Repo</h4>
See below, we'll use Heroku. So in order to deploy this application wich uses our plugin, I needed to publish somewhere.<br />
<br />
This is what can offer <a href="http://www.cloudbees.com/">Cloudbees</a>. Among other great things like git repo, CI and so on, Cloudbees provides you <b>for free</b> four maven repositories that you can make public if you wish.<br />
<br />
So I used sbt to publish on my "snapshot" repo on cloudbees, letting heroky has access to it for downloading the plugin dependency.<br />
<h4>
Platform</h4>
</div>
<div>
Free, reliable, easy to use with scala and Play 2.0... which else than <a href="http://www.heroku.com/">Heroku</a>?</div>
<div>
<h2>
</h2>
<h2>
So what</h2>
</div>
<div>
I think that this post is already too long... However, I can let you play with the resulting app <a href="http://neo4j-play20-plugin-test.herokuapp.com/">here</a>.</div>
<div>
<br /></div>
<div>
Check out the code (and fork it) <a href="https://github.com/andypetrella/neo4j-play2.0-plugin-test">there</a>.</div>
<div>
<br /></div>
<div>
Depending on the comment, I will expand this post to other ones to respond potential demands (if there is any ^^).</div>
<div>
<br /></div>
<div>
Good play.</div>
<div>
<br /></div>
<div>
Oh yes, one last note, the application is to <b>Create Stuff</b> which contains dummies. Stuff are created using a simple form that must be fulfilled. Stuffs can be linked one to another by clicking the graph.</div>
<div>
<br /></div>
<div>
Please create Stufs and links, it will be a good test for Neo4J, the plugin and Play 2.0.</div>
<div>
<br /></div>
<div>
Hope you've reach this.</div>
<div>
<br /></div>
<div>
SMAK. <span style="font-size: xx-small;">hehe</span></div>Noootsabhttp://www.blogger.com/profile/07479335886992076807noreply@blogger.com0tag:blogger.com,1999:blog-5531553525163261182.post-10361396827254675552012-03-22T22:28:00.000+01:002012-03-22T22:28:16.206+01:00TypeSafe Stack 2.0 missing "play debug" like featureA quick one to help players that are using TypeSafe stack instead of the Play! 2.0 distribution package.<br />
Because I discussed some points on the groups and I saw related StackOverflow entries, that this post might avoid in the future ^^.<br />
<br />
<br />
<h3>
TypeSafe Stack</h3>
<br />
With its second version, the <a href="http://typesafe.com/stack" target="_blank">TypeSafe stack</a> stroke a hit integrating <a href="http://akka.io/" target="_blank">Akka 2.0</a>, <a href="http://www.playframework.org/" target="_blank">Play 2.0</a> and... its amazing <a href="http://typesafe.com/products/console" target="_blank">console</a> built on top of <a href="http://www.mrspeaker.net/2012/03/15/typesafe-console/" target="_blank">both</a> technologies.<br />
<br />
With the <a href="http://scala-ide.org/" target="_blank">Scala IDE 2.0</a> (yeah a lot of 2.0), this stack is ready to tackle the SpringSource Tool Suite, but I don't want to make the comparison here neither explain all of these components... would be long and longer.<br />
<br />
But once you've installed the stack and you want to Play! around, they recommend you to use the giter8 template from the <i><a href="https://github.com/typesafehub" target="_blank">typesafehub</a></i> on github (it also contains a lot of plugins, which you might want).<br />
<br />
<br />
<h3>
SBT instead of Play launcher</h3>
<br />
Using the stack, you won't have the play tool in your hands to generate application and so on, because the way to go is to use <i>g8</i> and <i>sbt</i>.<br />
<br />
This is not an issue but there are some points you'll need to have in mind:<br />
<br />
<ul>
<li><b>the secret is not generated at creation</b>: they're not so far, because the secret is only a random string, and an issue on giter8 is on-going. So, you can <b>create a random string by your own</b> until it will be done. I've also proposed that a new command in sbt might be helpful to regenerate the secret.</li>
<li><b>play debug</b> isn't available: when you need to debug your Play! 2.0 app you need sbt to activate the <span style="font-family: Consolas, Menlo, Monaco, 'Lucida Console', 'Liberation Mono', 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', 'Courier New', monospace, serif;"><span style="font-size: 14px; line-height: 18px;"><i>jdwp </i></span></span>when running. For that, there is a MVN_OPTS like <b>SBT_OPTS</b> that comes in your help. Set it with the regular options (<i>-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=9999</i>).</li>
</ul>
<div>
<br /></div>
<div>
HTH</div>Noootsabhttp://www.blogger.com/profile/07479335886992076807noreply@blogger.com0tag:blogger.com,1999:blog-5531553525163261182.post-30263696112003072902012-03-15T00:01:00.000+01:002012-03-15T00:01:08.815+01:00Gatling-Tool Plugin for Play 2.0<div>
<h2>
Goal</h2>
</div>
<div>
This post will be a kind of write-up of what I'm trying to do now. And for what I already achieved some tasks.<br />
<br /></div>
<div>
Since I'll soon start products in my company that will be based on technologies like Scala, Play 2.0, Neo4J, MongoDB, Heroku and misc, and that I'm a bit control freak; I wanted to be sure that what I'll build will mach the requirements in terms of capacity.</div>
<div>
<br /></div>
<div>
This is where Gatling-Tool comes in the equation, this is a very powerful (and scala based) stress testing tool, which the name recall.</div>
<div>
<br /></div>
<div>
We're about to have some words on it, but first let me tell you that a GitHub ready to be fork is available with my first step into a gatling plugin, find it <a href="https://github.com/andypetrella/gatling-play2-plugin" target="_blank">here</a>.</div>
<div>
<br /></div>
<h2>
Gatling Tool</h2>
<div>
<a href="http://gatling-tool.org/" target="_blank">Gatling Tool</a> is a cute, smart and intuitive stress testing tool for web application, which a neat DSL for http request and asserts.</div>
<div>
<br /></div>
<div>
The DSL written in scala, and following the good conventions for it will aim anybody to be able to write stress tools. In a way that even non-programmer peoples with a basic understanding will be able to do basic stress tests with a good learning curve.</div>
<div>
<br /></div>
<div>
It's integration with browsers (Firefox, through a proxy) ease much more the work because you'll be able to register like macros (or like <a href="http://www.badboy.com.au/" target="_blank">badboy</a> does) your scenario to be repeated again and again. This is what is called the <a href="https://github.com/excilys/gatling/wiki/Gatling-Recorder" target="_blank">recorder</a>.</div>
<div>
<br /></div>
<div>
Scenarios could be written with a custom external DSL, but they're also available as regular scala code (internal DSL) and there is where the coupling with Play 2.0 scala should pay.</div>
<div>
<br /></div>
<div>
<h2>
Integration with Play2.0</h2>
</div>
<div>
<h3>
Akka 2.0</h3>
</div>
<div>
Fact: since the RC-5, Play 2.0 comes with the Akka 2.0 support. </div>
<div>
Fact: Gatling being it-self based on Akka (and they're right for that) but on a previous version for the stable version (logical because Akka 2.0 is pretty new).</div>
<div>
<br /></div>
<div>
So an integration must go through an update of the Gatling-Tool to the same version of Akka 2.0 in order to be able to use them correctly using the same project (testing phase, but still).</div>
<div>
<br /></div>
<div>
That's why I decided to fork the Gatling-Tool on GitHub (aaah the great world of open source), in order to switch the support of Akka from their 1.x to 2.0.</div>
<div>
<br /></div>
<div>
Even if it is true that I did it roughly (at first), it remains that it works and is the two needed projects for the following are:</div>
<div>
<ul>
<li>https://github.com/andypetrella/gatling</li>
<li>https://github.com/andypetrella/gatling-highcharts</li>
</ul>
<div>
So fork, clone them and build them locally using maven using the classical <code>mvn clean install</code>.</div>
</div>
<div>
<br /></div>
<div>
You'll have a brand new version of gatling 1.1.0-SNAPSHOT.</div>
<div>
<br /></div>
<div>
<b><u>NOTE</u></b>: I had some difficult choices when doing the migration, some are breaking the runtime behavior (a bit) and I'll have to discuss them further with the Gatling team. I've already been contacted by <a href="https://twitter.com/#!/slandelle" target="_blank">Stéphane Landelle</a> who told me that we was interested by the work since it was planned the 1.2 release.</div>
<div>
So don't be afraid, the official release will match the needs soon. (But ping me if you want more info and help me.)</div>
<div>
<br /></div>
<div>
<h3>
Typesafe Stack 2.0</h3>
</div>
<div>
I recommend you to install this brand new stack that integrates all stuffs that you'll need for scala development, including Play 2.0 project. </div>
<div>
Now, simply follows this <a href="http://typesafe.com/stack/download" target="_blank">link</a> for further steps, and then create a play-scala project.</div>
<div>
<br /></div>
<div>
<h3>
Sbt</h3>
</div>
<div>
Since Play 2.0 is using Sbt for building its project, and the custom gatling library we built is in our .m2 repo, we have first to add our local maven repo to the repositories list this way (updating your Build.scala):</div>
<div>
<script src="https://gist.github.com/2039848.js?file=Build.scala">
</script></div>
<div>
<br /></div>
<div>
Tada, now we have a Play 2.0 project having our custom gatling as dependency.</div>
<div>
<br /></div>
<div>
<h3>
Plugin</h3>
</div>
<div>
Before going in further details with integrating Gatling as a Play 2.0 plugin, I'm gonna talk about an uncovered subject in Play 2.0 (or not easy to track); the Plugin feature.</div>
<div>
<br /></div>
<div>
Play 2.0 comes with a pretty easy simple Plugin integration, this through the specific file, in the <i>conf</i> folder, named <i>play.plugins</i>. This file is meant to contain one single line by <u>defined</u> plugin, shaped this way: <code>{priority}:{plugin's class path}</code>.</div>
<div>
<br /></div>
<div>
But what is a plugin finally, this is a classical class extending <i>play.api.Plugin... </i>simply. This <i>Plugin</i> trait only defines three methods which are:</div>
<div>
<ul>
<li>onStart: this adds a hook when the application starts, helpful for initializing objects.</li>
<li>onStop: cleaning the fields.</li>
<li>enabled: helpful to disable the plugin in some specific cases.</li>
</ul>
<div>
Another point, that I have to highlight is that it <u>seems</u> that such Plugin's constructor must have an argument being the <i>application</i> it-self.</div>
</div>
<div>
<br /></div>
<div>
<h4>
Test only</h4>
</div>
<div>
Hey wait gatling should be available in tests only ?! Right!</div>
<div>
The first way to achieve this is the easies also, while implementing <i>isEnabled</i>, you can use the <i>application</i> (remember it is part of the constructor) which has a method <i>isTest</i> that should toggle the plugin.</div>
<div>
<br /></div>
<div>
The second way is to create a Specification that starts a FakeServer (since we'll stress the entire Play 2.0 flows) and give it a FakeApplication which is defined with the Plugins you wish. 1000 words replaced by one Gist:</div>
<div>
<script src="https://gist.github.com/2039848.js?file=TestStress_brief.scala">
</script></div>
<div>
<br /></div>
<div>
<h4>
Gatling integration</h4>
</div>
<div>
Now you're wondering what the heck is that Gatling Plugin class, don't you?</div>
<div>
<h5>
Gatling Plugin</h5>
</div>
<div>
The <a href="https://github.com/andypetrella/gatling-play2-plugin/blob/master/test/gatling/Gatling.scala" target="_blank">Gatling</a> plugin class, located in my repo under the <i>test/gatling</i> folder, is extending the <i>play.Plugin</i> class (which defines dummy implementation of the three methods), this way I can concentrate on the only method I need, <i>onStart</i>. </div>
<div>
<br /></div>
<div>
Actually I need some initialization in order to use gatling, it needs some folder to be defined, including the one interesting use the more: <i>results</i>.</div>
<div>
<br /></div>
<div>
So, the <i>Gatling#onStart</i> method is creating ephemeres folders under the target directory (that can be cleaned) and also the needed Gatling configuration file. <u>And that's it</u>.</div>
<div>
<br /></div>
<div>
We can now stress test our app.</div>
<div>
<br /></div>
<div>
<h5>
How?</h5>
</div>
<div>
Gatling is able to understand Scala written scenarios, those Scala script have only one constraint: being an instance of <i>com.excilys.ebi.gatling.core.scenario.configuration.Simulation</i>.</div>
<div>
<br /></div>
<div>
This trait is actually a <i>Function0</i> and thus defines only one helpful method which is <i>apply()</i>. The latter method is the container for building the stress test that we want to execute.</div>
<div>
<br /></div>
<div>
What is very common now is that you can write Gatling scenarios using the Scala Type System being checked in your favorite IDE, and they will be compiled by Sbt it self when requested (and hot swapped ^^). Where the classical Gatling workflow is to compile them on the fly, using their internal routines.</div>
<div>
<br /></div>
<div>
<h5>
Run 'em</h5>
</div>
<div>
Having simulations written (example <a href="https://github.com/andypetrella/gatling-play2-plugin/blob/master/test/stress/SampleSimulations.scala" target="_blank">here</a>), you can now ask Gatling to run them by creating a gatling runner instance on them. I won't go into deep details because it is not the purpose here, but here is how you can do.</div>
<div>
<script src="https://gist.github.com/2039848.js?file=RunAndGenerate.scala">
</script></div>
<div>
<br /></div>
<div>
<h5>
See 'em in action</h5>
</div>
<div>
That's the easiest part, entering the play environment using <code>sbt</code> in console, you can launch the tests by typing <code>test</code>.</div>
<div>
<br /></div>
<div>
What'll be done is:</div>
<div>
<ol>
<li>enter the specification</li>
<li>create the fake application</li>
<li>load the additional plugin</li>
<li>create the gatling folder and conf</li>
<li>configure the gatling system</li>
<li>create the fake server on 3333</li>
<li>create the simulations</li>
<li>run them</li>
<li>generate reports on them (located in the target folder => look 'em in your browser and you'll see how they're cute)</li>
</ol>
<div>
<h2>
Problems</h2>
</div>
</div>
<div>
There are problems for now when executing several tests, because streams are closed (while generating further reports), that comes from a choice that I've have to do and which is commented on github <a href="https://github.com/andypetrella/gatling/commit/270c11eb14509ae253b1fdf20650c01f26feb1f5#L14R147" target="_blank">here</a>. This is mainly related to a feature that is no more available in Akka 2.0 (for good reasons, I'd say).</div>
<div>
<br /></div>
<div>
<h2>
To be continued</h2>
</div>
<div>
If you want to help me going further, don't hesitate to contact me on my mailbox, or comment this post or on <a href="https://twitter.com/#!/noootsab" target="_blank">twitter</a>.</div>
<div>
<br /></div>
<div>
What I like to have in the future is :</div>
<div>
<ul>
<li>clear Specification for Gatling (preventing the need to define each time the server and plugin)</li>
<li>website for enabling test one by one, or any, or...</li>
<li>redirecting to results reports</li>
<li>more</li>
<li>and more</li>
</ul>
</div>Noootsabhttp://www.blogger.com/profile/07479335886992076807noreply@blogger.com4tag:blogger.com,1999:blog-5531553525163261182.post-655724322985685852012-03-11T13:12:00.001+01:002012-06-21T22:22:10.220+02:00Play 2.0 and Salat (MongoDB DAO provider)<br />
<h1>
Play 2.0 using MongoDB document storage through Salat</h1>
<br />
In this post, I'll cover some points and library that ease such use case.<br />
<br />
I won't go into deep details about Play and MongoDB, instead I'll jump straight to the MongoDB usage.<br />
<br />
Let's talk a bit about Salat<br />
<h2>
Salat</h2>
This free library available on github <a href="https://github.com/novus/salat" target="_blank">here</a> (and deployed on ivy repo, so that easily usable with sbt and Play) is able to deal with case classes and MongoDB as we'd do with JPA.<br />
<br />
That say, <i>case classes</i> can be used directly for storing document by simply declaring a DAO and with the help of some annotations (not mandatory).<br />
<h3>
Grater</h3>
Salat as the notion of Grater that is responsible to the de/serialization of a case class. This through two simple functions, let have a Grater for the type T:<br />
<ul>
<li><i>asDBObject(t:T) </i>: returns a MongoDB representation of <i>t</i></li>
<li><i>asObject(dbo:DBObject)</i> : returns a T instance based on the provided MongoDB content</li>
</ul>
<div>
<br /></div>
<div>
The latter thing to note is how to create such Grater? What do we have to do? The answer is <i>almost nothing</i>, in the package <i>com.novus.salat</i> is available the very handy method <i>grater[Y <: AnyRef]</i>.</div>
<div>
<br /></div>
<div>
What that grater method does is to parse your case class provided as the method generic, and create the related Grater instance. The important thing to note at this stage is that the latter Grater is <b>cached</b> for further needs.</div>
<div>
<br /></div>
<div>
So now, you're already able to deal with you case class. What is missing is the DAO part, that will ease again your job.</div>
<div>
<h3>
SalatDAO</h3>
</div>
<div>
Salat provides another handy structure named <i>SalatDAO</i>. This trait defines all lifecycle operations that you might need for your domain objects.</div>
<div>
<br /></div>
<div>
Its usage is very simple, you just have to define an object that extends this trait by giving the Domain Specific class and Id type for your structure. The last parameter it needs is a reference to the collection.</div>
<div>
<br /></div>
<div>
Here is an excerpt I pulled from the salat <a href="https://github.com/novus/salat/wiki/SalatWithPlay" target="_blank">wiki</a>:<br />
<span class="k" style="background-color: white; border-bottom-width: 0px; border-color: initial; border-image: initial; border-left-width: 0px; border-right-width: 0px; border-style: initial; border-top-width: 0px; color: #333333; font: inherit; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;"><code>object</code></span><code><span style="background-color: white; color: #333333; line-height: 19px;"> </span><span class="nc" style="background-color: white; border-bottom-width: 0px; border-color: initial; border-image: initial; border-left-width: 0px; border-right-width: 0px; border-style: initial; border-top-width: 0px; color: #445588; font: inherit; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">UserDao</span><span style="background-color: white; color: #333333; line-height: 19px;"> </span><span class="k" style="background-color: white; border-bottom-width: 0px; border-color: initial; border-image: initial; border-left-width: 0px; border-right-width: 0px; border-style: initial; border-top-width: 0px; color: #333333; font: inherit; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">extends</span><span style="background-color: white; color: #333333; line-height: 19px;"> </span><span class="nc" style="background-color: white; border-bottom-width: 0px; border-color: initial; border-image: initial; border-left-width: 0px; border-right-width: 0px; border-style: initial; border-top-width: 0px; color: #445588; font: inherit; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">SalatDAO</span><span class="o" style="background-color: white; border-bottom-width: 0px; border-color: initial; border-image: initial; border-left-width: 0px; border-right-width: 0px; border-style: initial; border-top-width: 0px; color: #333333; font: inherit; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">[</span><span class="kt" style="background-color: white; border-bottom-width: 0px; border-color: initial; border-image: initial; border-left-width: 0px; border-right-width: 0px; border-style: initial; border-top-width: 0px; color: #445588; font: inherit; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">User</span><span style="background-color: white; color: #333333; line-height: 19px;">, </span><span class="kt" style="background-color: white; border-bottom-width: 0px; border-color: initial; border-image: initial; border-left-width: 0px; border-right-width: 0px; border-style: initial; border-top-width: 0px; color: #445588; font: inherit; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">ObjectId</span><span class="o" style="background-color: white; border-bottom-width: 0px; border-color: initial; border-image: initial; border-left-width: 0px; border-right-width: 0px; border-style: initial; border-top-width: 0px; color: #333333; font: inherit; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">](</span><span class="n" style="background-color: white; border-bottom-width: 0px; border-color: initial; border-image: initial; border-left-width: 0px; border-right-width: 0px; border-style: initial; border-top-width: 0px; color: #333333; font: inherit; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">collection</span><span style="background-color: white; color: #333333; line-height: 19px;"> </span><span class="k" style="background-color: white; border-bottom-width: 0px; border-color: initial; border-image: initial; border-left-width: 0px; border-right-width: 0px; border-style: initial; border-top-width: 0px; color: #333333; font: inherit; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">=</span><span style="background-color: white; color: #333333; line-height: 19px;"> </span><span class="nc" style="background-color: white; border-bottom-width: 0px; border-color: initial; border-image: initial; border-left-width: 0px; border-right-width: 0px; border-style: initial; border-top-width: 0px; color: #445588; font: inherit; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">DB</span><span class="o" style="background-color: white; border-bottom-width: 0px; border-color: initial; border-image: initial; border-left-width: 0px; border-right-width: 0px; border-style: initial; border-top-width: 0px; color: #333333; font: inherit; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">.</span><span class="n" style="background-color: white; border-bottom-width: 0px; border-color: initial; border-image: initial; border-left-width: 0px; border-right-width: 0px; border-style: initial; border-top-width: 0px; color: #333333; font: inherit; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">connection</span><span class="o" style="background-color: white; border-bottom-width: 0px; border-color: initial; border-image: initial; border-left-width: 0px; border-right-width: 0px; border-style: initial; border-top-width: 0px; color: #333333; font: inherit; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">(</span><span class="s" style="background-color: white; border-bottom-width: 0px; border-color: initial; border-image: initial; border-left-width: 0px; border-right-width: 0px; border-style: initial; border-top-width: 0px; color: #dd1144; font: inherit; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">"users"</span><span class="o" style="background-color: white; border-bottom-width: 0px; border-color: initial; border-image: initial; border-left-width: 0px; border-right-width: 0px; border-style: initial; border-top-width: 0px; color: #333333; font: inherit; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">))</span></code><br />
<span class="o" style="background-color: white; border-bottom-width: 0px; border-color: initial; border-image: initial; border-left-width: 0px; border-right-width: 0px; border-style: initial; border-top-width: 0px; color: #333333; font: inherit; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;"></span><br />
<h2>
<span class="o" style="background-color: white; border-bottom-width: 0px; border-color: initial; border-image: initial; border-left-width: 0px; border-right-width: 0px; border-style: initial; border-top-width: 0px; color: #333333; font: inherit; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 0px; padding-right: 0px; padding-top: 0px;">Play 2.0</span></h2>
<span style="color: #333333;">Having covered the basic of Salat, it's time to use it in our Play 2.0 application.</span><br />
<span style="color: #333333;">There is a cool wiki page, that I wished have discovered before which explain how to use salat with Play 1.2.x. I recommend you to </span><a href="https://github.com/novus/salat/wiki/SalatWithPlay" target="_blank">read it</a><span style="color: #333333;">, even if I'm gonna cover some of the important steps here too.</span><br />
<span style="color: #333333;"></span><br />
<h3>
<span style="color: #333333;">Dependencies</span></h3>
<span style="color: #333333;">This step is more easy that for the previous version of Play. Because now the only two things required are:</span><br />
<ul>
<li><span style="color: #333333;">add the novus repo</span></li>
<li><span style="color: #333333;">add the deps to salat</span></li>
</ul>
<div>
<span style="color: #333333;">Both in the Build.scala in the project folder of your Play 2.0 app.</span></div>
</div>
<br />
<b>**Edit (on 21th June 2012)**</b><br />
Before you continue, this post that explains some basics about how to deal with Salat and Play, you can now choose to simply move to this module <a href="https://github.com/novus/salat/wiki/SalatWithPlay2">https://github.com/novus/salat/wiki/SalatWithPlay2</a>.<br />
It introduced the tricks that I'll explain below, and add amazing functionalities for Model and DAO creation.<br />
So starting at this point, it's no more mandatory to read this post... unless you're curious ;-)<br />
<b>**end of EDIT**</b><br />
<br />
<b>Context</b>
<br />
Here is the <b>main</b> thing I have to discuss here: Salat makes intensive usage of a structure named Context which holds a reference to what have been processed along the classes and structures.<br />
<br />
Such instance is created by default in the package object <i>com.novus.salat.global</i>, and the quick start of Salat recommend to import it along with <i>salat</i> and <i>annotations</i>. <b><span style="color: red;">Don't!</span></b><br />
<b><span style="color: red;"><br /></span></b>
Doing so will fail when using <b>Play 2.0 in DEV</b> mode (only) because of the cool-hot-reload-on-change feature of Play. The specific case where it will fail is the following:<br />
<div>
<i><span style="font-size: large;">If you want to keep a static reference to the Grater instance of a specific case class.</span></i></div>
<div>
<i><br /></i></div>
<div>
<i><script src="https://gist.github.com/2016161.js?file=Model.scala">
</script></i></div>
<div>
<br /></div>
<div>
Why? Because of the (needed for sure!) graters' cache which keeps a reference to the <i>Class</i> instance.</div>
<div>
But this class instance might change on bytecode refresh, moreover the fact that Play has a specific ClassLoader for that (<b>ReloadableClassLoader</b>).</div>
<div>
<br /></div>
<div>
The result is incomprehensible errors when you change your code, which errors saying the there is a mismatch between classes that you even not change yet... </div>
<div>
<br />
<div>
<h3>
The solution (which is referred in the wiki page I told above)</h3>
<h3>
</h3>
</div>
<div>
Instead of importing <i>com.novus.salat.gobal._</i> which only contains an implicit definition of the Context object to use in Salat core system, create a new one using the correct ClassLoader.</div>
<div>
<br />
<div>
<script src="https://gist.github.com/2016161.js?file=salatcontext.scala">
</script></div>
</div>
</div>
<div>
<br /></div>
<div>
In the above Gist, we can see that we simply created a Context that will refer to the provided ClassLoader by the Play app itself.</div>
<div>
<br /></div>
<div>
That will keep enabled the class reloading, without impacting caches instances.</div>
<div>
<br /></div>
<div>
<h2>
<i>That's all folks!</i></h2>
</div>
<div>
Now you're ready to use both Play 2.0 and Salat without messing around with conversion between DSM and MongoDB and so on.</div>
<div>
Have a gode work.</div>Noootsabhttp://www.blogger.com/profile/07479335886992076807noreply@blogger.com6tag:blogger.com,1999:blog-5531553525163261182.post-23243912303438000622012-02-29T23:03:00.001+01:002012-02-29T23:04:38.813+01:00Neo4J with Scala Play! 2.0 on Heroku (Part 9) :: Final<h2 class="custom">
Note</h2>
<div>
This post is a continuation of <a href="http://ska-la.blogspot.com/2012/02/neo4j-with-scala-play-20-on-heroku-part_25.html" target="_blank" title="part 4">this post</a> (which started <a href="http://ska-la.blogspot.com/2012/02/neo4j-with-scala-play-20-on-heroku-part.html">there</a>), which is the last part of a blog suite that aims the use of Neo4j and Play2.0 together on <a href="http://www.heroku.com/" target="_blank">Heroku</a>.<br />
<br />
What have been accomplished so far:<br />
<br />
<ul>
<li>install Play 2.0</li>
<li>install Neo4J</li>
<li>use Dispatch</li>
<li>create model</li>
<li>create persistence service in Neo4J</li>
<li>create views and controllers</li>
</ul>
<div>
Ok, where almost done. Let's see how to deploy the whole app on Heroku.</div>
<div>
<ol>
</ol>
<div>
<h1>
Heroku, here I come</h1>
</div>
</div>
<h2 class="custom">
But wait, who're you?</h2>
</div>
<div>
Heroku is simply one of the best cloud players of the moment, I won't talk too much about it, because i'd have to talk a lot otherwise.<br />
<br /></div>
<div>
But here are some very interesting features and paradigms followed by Heroku.</div>
<div>
<br /></div>
<h3>
Process Centric</h3>
<div>
Where almost all other cloud providers are binding their services to server instances, CPU flop count, memory usages, and other similar metrics, it is a fact that their aren't easily forecastable and hard to track in development phases (even if I encouraged to do it, though).</div>
<div>
<br /></div>
<div>
Heroku comes with a much more easy concept, that is, <i>Web Dynos</i>. A Web Dynos is simply a process that can handle requests. So, what if the requests are too numerous? Just add Dynos. Note that Dynos are existsing for background process, one Dyno by worker.</div>
<div>
<br /></div>
<div>
Costs are very simple too, you have one free Dyno by month, and the rest is billed at low cost by hour.</div>
<div>
<br /></div>
<div>
Thanks for simplicity.<br />
<br /></div>
<h3>
Remote CLI</h3>
<div>
We've just ask how to handle more requests in an efficient way, and answered by adding dynos.</div>
<div>
<br /></div>
<div>
So far so good, but how? That's where comes the Heroku remote CLI that is able to operate remotely on a deployed application behavior.</div>
<div>
<br /></div>
<div>
Thus, adding a dyno is doing that in console : <code>$> heroku dynos 1</code></div>
<br />
Now, alerts on performance are quickly resolved.<br />
<br />
Thanks for rapidity.<br />
<br />
<h3>
Continuous Deployment</h3>
<div style="font-size: medium; font-weight: normal;">
The paradigm followed by Heroku to deploy their app are based on Continuous Deployment.</div>
<div style="font-size: medium; font-weight: normal;">
<br /></div>
<div style="font-size: medium; font-weight: normal;">
Having that, you app should define how you app must be deployed using their <i>Procfile</i>.</div>
<div style="font-size: medium; font-weight: normal;">
<br /></div>
<div style="font-size: medium; font-weight: normal;">
And it will be deployed automatically when the sources are pushed to the Git repo that is created for each application.</div>
<div style="font-size: medium; font-weight: normal;">
<br /></div>
<div style="font-size: medium; font-weight: normal;">
This ensure you to at any time be able to retrieve the sources related to the running instance (for example).</div>
<div style="font-size: medium; font-weight: normal;">
<br /></div>
<div style="font-size: medium; font-weight: normal;">
Thanks for debugging ease.<br />
<br /></div>
<div style="font-size: medium; font-weight: normal;">
</div>
<h3>
Add-Ons</h3>
What to say? A good sdk to create add-ons, a good architecture and service level. It makes a pleiade of powerful add-ons including Neo4J running instances.<br />
<br />
Thanks to be open.<br />
<br />
<div>
<h2 class="custom">
Can I <i>Play!</i> with Heroku?</h2>
</div>
<div>
Of course, you can!</div>
<div>
<br /></div>
<div>
Actually, Heroku has already integrated Play! starting with its first version, and has also added the scala support some time ago.</div>
<div>
<br /></div>
<div>
And finally the Play 2.0 <a href="https://github.com/playframework/Play20/wiki/ProductionHeroku" target="_blank">wiki</a> is explaining how to do...</div>
<div>
<br /></div>
<div>
<h2 class="custom">
Ok, let's Go then.</h2>
</div>
<div>
<h3>
Getting started</h3>
</div>
<div>
First of all, you must have been registered too Heroku. Hopefully, it's free and fast. So go on, and create your account <a href="https://api.heroku.com/signup" target="_blank">here</a>. </div>
<div>
<br /></div>
<div>
Having your account, you can now <a href="http://devcenter.heroku.com/articles/quickstart#step_2_install_the_heroku_toolbelt" target="_blank">install</a> the Heroku toolkit belt. This will gives you acces to your Heroku CLI that can manage your account, apps, and app configuration.</div>
<div>
<br /></div>
<div>
When you're done with the installation, you just have to login using the console command: <code>$> heroku login</code></div>
<div>
<br />
<div>
<h3>
Play! app side</h3>
</div>
<div>
</div>
<br />
What is needed for your app is to have a Git repo and to contain a Heroku process description file.<br />
Since everything is already explain <a href="https://github.com/playframework/Play20/wiki/ProductionHeroku" target="_blank">there</a>, I won't go into deep details.<br />
<div>
<h3>
Create Heroku app</h3>
</div>
<div>
Since we are using Play! and scala, we need a JVM, that stack at Heroku is called <i><a href="http://devcenter.heroku.com/articles/cedar" target="_blank">Cedar</a></i>.</div>
<div>
<br /></div>
<div>
So, to create your app, open a shell and do the following:</div>
<code>heroku create my-playing-app-with-neo4j --stack cedar</code>
<br />
<div>
</div>
<br />
Now, you have an up and running environment to setup and deploy your application. And the application will be named <i>my-playing-app-with-neo4j</i>.<br />
<br />
<div>
<h3>
Neo4J add-on</h3>
</div>
<div>
Ok, but I've to use a Neo4J database, not embedded (too heavy for a cloud). Do I have to install it somewhere and host it myself. Na!</div>
<div>
<br /></div>
<div>
Neo4J's team is actually working on an integration in the Heroku platform, and a beta test add-on is available at the time writing.</div>
<div>
<br /></div>
<div>
That says that to have a running database that we can use, we just have to open a shell (in our app folder) and drop the following command: <code>heroku addons:add neo4j</code> </div>
<br />
You don't believe it, huh?
<br />
<br />
Since you'll need to retrieve the database url and credentials, either you go to the Heroku site and...<br />
Na, just keep your shell and do: <code>heroku addons:open neo4j</code> </div>
<br />
Ta da!<br />
<br />
<div>
<h3>
App update</h3>
</div>
<div>
In my previous post, for the sake of simplicity I left the Neo4J database hard coded to <i>localhost:7474</i>.</div>
<div>
<br /></div>
<div>
But now, we have to update this to use our deployed Neo4J instance and credentials.</div>
<div>
<br /></div>
<div>
We should have (must) define an application configuration property for such paramater, but It is not what I want to illustrate here so let's keep it simple and hard coded.</div>
<div>
<br /></div>
<div>
But we have to add something to the Neo4J's Dispatch url, the credentials. For that we just have to do the following:</div>
<div>
<script src="https://gist.github.com/1944456.js?file=Neo4JRestService.scala">
</script>
</div>
<h3>
SSH key</h3>
<div>
Just a note, to remember you to add your ssh key to Heroku. This is simply accomplished (after you've have created 'em) using the CLI: <code>$> heroku keys:add</code></div>
<br />
Beginning to love this CLI, no?<br />
<br />
<h3>
Procfile</h3>
<div>
This is the Heroku configuration file that tells the continuous deployer (if I can say) how the application will be deployed and are its needs.</div>
<div>
<br /></div>
<div>
This file is located at the root of the Play's application folder and only contains one line:</div>
<code>web: target/start -Dhttp.port=${PORT} ${JAVA_OPTS}</code>
<br />
<code><br /></code><br />
<div>
This simple line tells that we need a web process for the staging application located under target/start
</div>
<div>
Actually, this folder will contains the staging Play! application after Heroku will run <code>sbt clean compile stage</code> on it.<br />
<br /></div>
<h3>
Aaaaand Deploy! (push)</h3>
Getting closer to the end!<br />
<br />
After having added all the necessary sources to the local git repo for your app, (including last update and the Procfile), we can now commit everything and push it to the git repo that Heroku holds for our application.<br />
<br />
Actually, when created the Heroku app, the CLI has updated the git local configuration to add the related remote repo called <i>heroku</i>.<br />
<br />
So, the only thing that left to do is to push: <code>$> git push heroku master</code>
<br />
To test if it is ok: <code>$> heroku ps</code>. This will display the proceses running on Heroku.<br />
<br />
If the process is shown, let's open the application in our default brower (leave your mouse alone and...): <code>$> heroku open</code>.<br />
<br />
I hope that I didn't made too many mistakes and you are now able to see your application running and using Neo4J.<br />
<br />
At least, here is the one I succeed to deploy: <a href="http://scala-plays-with-neo4j.herokuapp.com/">http://scala-plays-with-neo4j.herokuapp.com/</a>.<br />
I've also shared this app on Heroku's <a href="http://gensen.herokuapp.com/" target="_blank">GenSen</a> that is meant to share project template on Heroku.
<br />
<span style="font-size: large;"><br /></span><br />
<span style="font-size: large;">Now, you should love the CLI</span>, but also Heroku, and Neo4J <span style="font-size: x-small;">and Play! 2.0 and Scala</span> <span style="font-size: xx-small;">and Dispatch and arbor.js and...
</span><br />
<br />
<br />
<hr />
Thanks for reading, if someone do have ^^.
Noootsabhttp://www.blogger.com/profile/07479335886992076807noreply@blogger.com0tag:blogger.com,1999:blog-5531553525163261182.post-30414318320990559952012-02-27T23:43:00.000+01:002012-02-29T23:05:15.751+01:00Neo4J with Scala Play! 2.0 on Heroku (Part 8) :: Scala template+Arbor.js to browse Neo4J via Play 2.0<br />
<div>
<h2>
Note</h2>
This post is a continuation of <a href="http://ska-la.blogspot.com/2012/02/neo4j-with-scala-play-20-on-heroku-part_25.html" target="_blank" title="part 4">this post</a>, which is the seventh part of a blog suite that aims the use of Neo4j and Play2.0 together on Heroku.</div>
<div>
<br /></div>
<h1>
Viewing Neo4J Model Object in Play2.0</h1>
<h2>
Goal</h2>
<div>
In this post, I'll talk about some functionalities that Play2.0 offers to create web application/site.</div>
<div>
<br /></div>
<div>
The main goal will be to have html views that enable us to create User, Group and link them, but not only, we'll use <a href="http://arborjs.org/" target="_blank">arbor.js</a> to view what's being created or linked in Neo4J as a... graph of course.</div>
<div>
<br /></div>
<div>
Basically, it will consist into one html page, containing several forms for creating model instance (or link) through AJAX call on Json controllers.<br />
<br />
So let's begin by explaining how to define a querying and persisting controllers using Play 2.0 <i>Form</i>.<br />
<h2>
Controllers</h2>
<div>
</div>
In that case, we'll take basic needs for our use case, that is, to retrieve the users list stored in Neo4J or create a new group.<br />
<h3>
Get Users</h3>
<div>
Briefly, Play 2.0 as the notion of controllers to handle server request, such controllers are bound to urls using a route configuration.</div>
<div>
<br /></div>
<div>
So what we have to do here is to create a controller, let's say <i>Users</i>, with a handler named <i>j_all</i> for list of users rendered in Json.</div>
<div>
<br /></div>
<div>
Using what we've discussed in previous posts, such controller and definition are rather simple, check this out:</div>
</div>
<br />
<script src="https://gist.github.com/1927260.js?file=Users.scala">
</script>
As we can see, we have simple call the Model persistence utility object to retrieve all defined User in Neo4J. Which we are rendering directly in Json thanks to their Formatter. And finally, we stream the result in the http response.<br />
Mmmh, simple no ? Here we did:<br />
<br />
<ol>
<li>send a Json request to Neo4J requesting all nodes that are linked to the root using the kind <i>users</i> (found using the User's ClassManifest)</li>
<li>retrieve the Json response from Neo4J and un-marshall them in a <i>List[User]</i> (using the User Formatter)</li>
<li>re-render them into the expected Model Json Format (again using the Fomatter)</li>
<li>generate the String representation</li>
<li>append it in the response body</li>
<li>define the content type as being Json</li>
</ol>
<div>
In one single line.</div>
<div>
<br /></div>
<div>
To test it, roughly, just use this url <a href="http://localhost:9000/users.json" target="_blank">http://localhost:9000/users.json</a>. This will return a Json encoded response.</div>
<div>
<h3>
Create Group</h3>
<div>
</div>
</div>
<div>
Now, we want to add the possibility to create a new group remotely. For that, we'll create a controller <i>Groups</i> which defines a <i>create</i> handler.</div>
<div>
<br /></div>
<div>
This handler expects to receive a group name. After what, it creates the group instance and persist it in Neo4J.</div>
<div>
<br /></div>
<div>
To recover such request parameter (in a POST since we are creating something and changing the server state), we use a Play 2.0 construction <i>play.api.data.Form</i> that offers a lot of helpers to parse the body into a map of values (can be embedded).</div>
<div>
<br /></div>
<div>
In the following example, the goup name is extracted form the request's body (url encoded) as a <i>nonEmptyText</i> mapped as <i>name</i>. This is a helper mapping for extracting String that cannot be empty.</div>
<br />
<script src="https://gist.github.com/1927260.js?file=Groups.scala">
</script>
As we can see, the Form can be directly rendered in the Model instance by giving an apply and unapply functions after the mapping definition.<br />
<div>
<h2>
Javascript Routing</h2>
<div>
Using static urls are cool... no ok, let's try to use what some calls Web 2.0, you know Ajax.</div>
<div>
<br /></div>
<div>
The problem comes when you have to deal with Urls within Ajax calls. How to keep track of your urls changes for instances.</div>
<div>
<br /></div>
<div>
Pretty hard, so let's forget about hard coded urls in your javascript and use a routes file that can be downloaded client side. This routes file contains all your controllers' url mapping that you want to be exposed in javascript (if I can say).</div>
<div>
<br /></div>
<div>
How it works is simple:</div>
<div>
<ol>
<li>Use <i><a href="https://github.com/playframework/Play20/blob/master/framework/src/play/src/main/scala/play/api/Routes.scala" target="_blank">Routes</a>.javascriptRouter</i> to define a javascript object and the controllers to be remoted</li>
<li>For each of them, you must use the following object <i>controllers.routes.javascript.<controller>.<function></function></controller></i></li>
<li>This object is created at compile time when defining the controller in the route conf file</li>
<li>defines a handler (in the Application controller f.i.) that return the result of the javascriptRouter as being javascript file</li>
<li>route this new controller to what you want (like <i>/js/routes</i>)</li>
</ol>
<div>
Having done that, you are now ready to use the created object in the javascript part.</div>
</div>
<div>
<br /></div>
<div>
If we take the controller <i>controllers.Users.j_one </i>(returns a User base on its given id), we'll have in our javascript access to a js function <i>playRoutes.controllers.Users.j_one(id)</i> that can takes an id.</div>
<div>
<br /></div>
<div>
By using this js function, you'll have in return a js object that defines at least two useful properties:</div>
<div>
<ul>
<li>url: the formatted url for the controller (having compiled the parameter in the url)</li>
<li>ajax(c): a jquery (by default) ajax function that takes a payload object, but already defines the url and the method.</li>
</ul>
<div>
So far so good, but to use all of these stuffs, let's see in a coffeescript (thanks <a href="https://github.com/playframework/Play20/wiki/AssetsCoffeeScript" target="_blank">Play 2.0</a>) example:</div>
</div>
<br />
<script src="https://gist.github.com/1927260.js?file=get_users.coffee">
</script>
In the previous example, I wrote the ajax call my-self using jQuery... so I could have simply use the ajax property. But nevermind, I love sometime to be control freak.
<br />
<br />
C'est chic! No?<br />
<br />
<h2>
Arbor.js</h2>
<div>
For browsing our model graph, I've used arbor.js as the rendering framework, because it's the best one for graph... that's it. </div>
<div>
Since my intent here isn't to explain it, I'll leave you alone with that part. But I recommend you to browse its site <a href="http://arborjs.org/" target="_blank">here</a>.<br />
<br />
So what I did is simply using Users as nodes, all linked to a central root node. Clicking one them will show you their inter-relationships.<br />
<br />
I've also added a select box that helps you showing all users in a chosen group.<br />
<br />
Taking that the next post will be on how to deploy the whole stuff on Heroku. I don't have at this time any instance in the wild, but if you wish you can clone (and fork) <a href="https://github.com/andypetrella/Play20WithNeo4J" target="_blank">my repo on github</a> for this posts' suite.<br />
<br />
But here is a preview of what has been achieved.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://2.bp.blogspot.com/-UHeP3vJf3WA/T0wGNElHqTI/AAAAAAAAAwE/CnPvABqf-Pg/s1600/neo4j-and-play2.0.tiff" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="355" src="http://2.bp.blogspot.com/-UHeP3vJf3WA/T0wGNElHqTI/AAAAAAAAAwE/CnPvABqf-Pg/s400/neo4j-and-play2.0.tiff" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Fun but not so cute -> I'm not a designer... :'(</td></tr>
</tbody></table>
<br /></div>
<div>
</div>
</div>
<hr />
Next <a href="http://ska-la.blogspot.com/2012/02/neo4j-with-scala-play-20-on-heroku-part_29.html">post</a>, the last, will talk about how to deploy this whole thing onto the Heroku PaaS.Noootsabhttp://www.blogger.com/profile/07479335886992076807noreply@blogger.com0tag:blogger.com,1999:blog-5531553525163261182.post-42497078332419970512012-02-25T00:40:00.000+01:002012-02-27T23:46:31.389+01:00Neo4J with Scala Play! 2.0 on Heroku (Part 7) :: DSM+DAO+Neo4J+Play<br />
<div>
<h2>
Note</h2>
This post is a continuation of <a href="http://ska-la.blogspot.com/2012/02/neo4j-with-scala-play-20-on-heroku-part_21.html" target="_blank" title="part 4">this post</a>, which is the sixth part of a blog suite that aims the use of Neo4j and Play2.0 together on Heroku.</div>
<div>
<br /></div>
<h1>
Using Neo4J in Play 2.0... and simple DAO</h1>
<div>
What I'll intent to show is a way to use a Domain Specific Model, persisted in a Neo4J back end service. For such DSM, we'll have an abstract magic Model class that defines generic DAO operations.</div>
<br />
<div>
For simplicity, we'll try to link each category of classes to the root/entry node. For instance, all the Users will be bound to the entry node by a reference of kind <i>user</i>.</div>
<h3>
Model</h3>
I'll choose the very common use case, that is, Users and Groups. Here is its shape:<br />
<ul>
<li>A User has a first name</li>
<li>A Group has a name</li>
<li>A User can be in several Groups</li>
<li>A Group can contain several Users</li>
<li>A User can know several Users</li>
</ul>
<div>
Let's keep the classes definition aside for a few, and stick to the persistence service.</div>
<h3>
Graph Service</h3>
<div>
The Graph Service is an abstraction of what is needed for a Graph Persistence Layer. It is bound to a generic type that defines the model implementation and defines traversal and persistence operations of graph's nodes.
</div>
<script src="https://gist.github.com/1903936.js?file=GraphService.scala">
</script>
<br />
<h3>
Graph Service for Neo4J</h3>
<div>
Let's update now, the service that has been used in the previous post, for Neo4J persistence, in order to have it able to deal with model instance.<br />
<br />
Let's start with the <i>saveNode</i> operation to see what is needed in the model and elsewhere.</div>
<script src="https://gist.github.com/1903936.js?file=Neo4JRestService_1.scala">
</script>
<br />
<div>
In this Gist above, I've enlighted some points that must be found around the Model construction. (A) and (C) are composing a Json Format (as SJson propose), (B) is more related to model abstraction.<br />
<br />
(C) has a special need when used with Dispatch, we could have a Dispatch Handler that can do both action parsing/unmarshalling and direct use in a continuation.
</div>
<h4>
Model</h4>
Now, we are at the right point to talk the Model, since we've met almost all its requirement. So let's build a Magic Model class that can be extended by all concrete model classes.
<br />
<h5>
Skeleton</h5>
That's the easy part, we just define the id property that is an id (part of the Rest Url in Neo4J).
<br />
<h5>
Formatter</h5>
<div>
Ok, this part is simple too in this abstract Model definition because, a Format implementation must be part of the concrete DSM classes. That is, <i>User</i> that extends <i>Model</i> must define a <i>Format[User]</i> instance, and put it in the implicit context.
</div>
<div>
So, at this stage we have Model and User like this:
</div>
<script src="https://gist.github.com/1903936.js?file=ModelUser_1.scala">
</script>
<br />
<h5>
Class -- Relation's kind : F-Bounded</h5>
As we saw in the <i>saveNode</i> method needs to associate the concrete class to a relation kind. But what I wanted is to have a <i>save</i> method in Model, that implies that we cannot (at first glance) give the <i>saveNode</i> the information needed, that is the concrete class.<br />
<br />
For that, we'll use a F-Bounded type for Model, that way we'll be able to give the <i>saveNode</i> method what is the really class... Mmmh ok, let me show you:
<script src="https://gist.github.com/1903936.js?file=Model_FBounded.scala">
</script>
But that's not sufficient, the <i>saveNode</i> method will need to use such available <i>ClassManifest</i> to find the relation it must create.<br />
<br />
I choose a very common and easy solution, which is having a function in the Model companion that helps in registering classes against relation kind.
<script src="https://gist.github.com/1903936.js?file=ModelObject.scala">
</script>
<br />
<h4>
Model Dispatch Handler</h4>
<div>
Now we'll discuss something I find really useful and easy in Dispatch, create a Handler that can handle a Json response from Neo4J into a Model instance.<br />
For that, we have already defined in previous post a way to handle json response into Play's JsValue.<br />
<br />
Now, what we need is to use the implicit formatter of all model concrete classes to create instances. And it'll be the way to reach the goal, except that a problem comes from the Json response of Neo4J: the data is not present at the Json root, but is the value of the data property.<br />
So it breaks our Format if we use it directly.<br />
<br />
That's why the above definition of the Handler takes an extra parameter which is the conversion between JsValue to JsValue, that is to say, a function that goes directly to the data definition.
<script src="https://gist.github.com/1903936.js?file=ModelHandlers.scala">
</script>
</div>
<h4>
saveNode</h4>
<div>
Finally, let's gather all our work in a simple implementation of a generic saveNode function:
<script src="https://gist.github.com/1903936.js?file=saveNode.scala">
</script>
As we can see, it's very easy to handle Neo4J response as DSO and use them directly in the continuation method of the Handler.
</div>
<h3>
Usage</h3>
having all pieces in places (check out the related Git repo <a href="https://github.com/andypetrella/Play20WithNeo4J" target="_blank">here</a>).
We can now really simply create a User and retrieve it updated with its id, or even get it from the database using its id.
<script src="https://gist.github.com/1903936.js?file=spec.scala">
</script>
<br />
<hr />
In the next <a href="http://ska-la.blogspot.com/2012/02/neo4j-with-scala-play-20-on-heroku-part_27.html">Post</a>, we'll create some Play template for viewing such data, but create them also.Noootsabhttp://www.blogger.com/profile/07479335886992076807noreply@blogger.com0tag:blogger.com,1999:blog-5531553525163261182.post-13695863083551898172012-02-21T00:29:00.000+01:002012-02-25T00:43:55.405+01:00Neo4J with Scala Play! 2.0 on Heroku (Part 6) :: Dispatch+Play 2.0<div>
<h2>
Note</h2>
This post is a continuation of <a href="http://ska-la.blogspot.com/2012/02/neo4j-with-scala-play-20-on-heroku-part_2470.html" target="_blank" title="part 4">this post</a>, which is the fifth part of a blog suite that aims the use of Neo4j and Play2.0 together on Heroku.</div>
<div>
<br /></div>
<h1>
Using Neo4J in Play 2.0</h1>
In this post, we'll create a Dispatch Handler that handles Neo4J Rest Json calls in Play's Json object.<br />
Having this in our hands, we'll be able to create a really simple service for dispatching Neo4J operations and use them in Play's views.<br />
<br />
I've compiled the Play app on github, <a href="https://github.com/andypetrella/Play20WithNeo4J" target="_blank">fork me</a>.<br />
<br />
NB: we'll use Dispatch but in case you wish to, you could use the Play's WS feature that might help you a lot (<a href="https://github.com/playframework/Play20/wiki/ScalaWS" target="_blank">check this out</a>).<br />
<br />
<h2>
Declare Dispatch deps</h2>
First of all, we have to update our Play app with the Dispatch dependency. For that, we have to update the sbt configuration file in order to add the related line.<br />
<script src="https://gist.github.com/1872053.js?file=Build.scala">
</script>
<br />
Now, that we have updated the project, let update the application by reloading the configuration (if you're already in sbt console) and rebuild our IDEA project.<br />
<br />
<script src="https://gist.github.com/1872053.js?file=gistfile1.sh">
</script><br />
<h2>
Play's Json Handler</h2>
Our goal is to use the Neo4J Rest Api that returns responses Json encoded.<br />
So here, I'll show how we could have such response directly unmarshalled in Json object. In further posts, we'll use such handling feature to get Model instances directly (which is far more interesting).<br />
<br />
What is necessary for that is to create a piece of code that is capable to take a subject and convert it to a JsValue. And since we love functional programming, let us have this method taking a continuation that accepts a JsValue.<br />
<script src="https://gist.github.com/1872053.js?file=handler.scala">
</script>
In this listing, we see that we use the text parser to consume the response payload, then we ask the Play's Json parse function do its job.<br />
Finally, we use the continuation applied to the parsed result.<br />
<br />
<h2>
Neo4J Service</h2>
Let's gather some utility urls to retrieve node, relations. In other words, urls for common usages.
This service is left simple for further enhancements (next post).
<script src="https://gist.github.com/1872053.js?file=basic_neo4j.scala">
</script>
We see that most functions are there to create urls based on ids, but there is also the root one that directly fetches the entry node.
<br />
<br />
<h2>
A Controller To Rule Them All</h2>
For the sake of this basic usage of our Handler with Neo4J, here are some examples of such requests. (full controller <a href="https://github.com/andypetrella/Play20WithNeo4J/blob/master/app/controllers/Neo4J.scala" target="_blank">here</a>).<br />
<script src="https://gist.github.com/1872053.js?file=controller.scala">
</script>
As we can see, all we had to do is to create the correct url by using id, or Neo4J path conventions, then using the Handler operator ( <span style="background-color: white; font-family: 'Bitstream Vera Sans Mono', 'Courier New', monospace; font-size: 12px; font-weight: bold; line-height: 16px; text-align: left; white-space: pre;">>! </span>... how it's Play, no ?!), we have the facility to use directly JsValue instance to consume the result.<br />
Okay, it's repetitive and the Json traversal is not shared. Let's us put this aside until the next post.<br />
And before going ahead, I've created a pretty simple and naive view and url mapping. So check the sources on github, play it and tests the <i>/rest</i> et al. urls.<br />
<hr />
<a href="http://ska-la.blogspot.com/2012/02/neo4j-with-scala-play-20-on-heroku-part_25.html">Next post: Enhance the handler and service to manage Domain Object.</a>Noootsabhttp://www.blogger.com/profile/07479335886992076807noreply@blogger.com0tag:blogger.com,1999:blog-5531553525163261182.post-25173628421299187682012-02-19T22:32:00.001+01:002012-02-21T00:31:08.164+01:00Neo4J with Scala Play! 2.0 on Heroku (Part 5) :: Dispatch<h2>
Note</h2>
This post is a continuation of <a href="http://ska-la.blogspot.com/2012/02/note-this-post-is-continuation-of-this.html" target="_blank" title="part 4">this post</a>, which is the fourth part of a blog suite that aims the use of Neo4j and Play2.0 together on Heroku.<br />
<h1>
Intro</h1>
In this suite, we're gonna use Neo4J graph database through its HTTP REST api, which is quickly introduced in <a href="http://noootsab.tumblr.com/post/17172741154/neo4j-with-scala-play-2-0-on-heroku-part-2" target="_blank" title="Neo4J">this post</a>.<br />
So that, we'll have to communicate with the server using HTTP, here comes the <a href="http://dispatch.databinder.net/" target="_blank" title="Dispatch wiki">Dispatch</a> scala library.<br />
Neo4J uses Json as the resources representation, we've already discussed this subject in Play 2.0 in <a href="http://noootsab.tumblr.com/post/17447408368/neo4j-with-scala-play-2-0-on-heroku-part-4" target="_blank" title="Json and Play 2.0">this post</a>.<br />
How to stick them together will be discussed in the next post.<br />
Here we'll concentrate on some introduction to Dispatch's DSL for making HTTP requests and on a powerful abstraction of the body parser, that is, the <a href="http://dispatch.databinder.net/Response+Bodies.html" target="_blank" title="Parse Bodies">Handlers</a>.<br />
<h1>
Dispatch</h1>
<h2>
The DSL</h2>
Dispatch is a very powerful library for communicating through the HTTP protocol, offering a DSL for making such queries, but also for using their responses.<br />
Where we all know, that response can be of different content-type, the DSL presents easy handler for them.<br />
Getting back to queries, an HTTP request has been given a method like GET or POST, when dealing with RESTful services, we'll see PUT and DELETE in the game.<br />
When data must be provided we'll have to pass some arguments/parameters in the request payload (or url).<br />
Let's see how those actors compose in the Dispatch's DSL.<br />
<h3>
URL</h3>
The url of the HTTP request is basically composed of the host and port, followed by path elements. Here is how to create an url such url <a href="http://dispatch.databinder.net/URLs+and+Paths">http://dispatch.databinder.net/URLs+and+Paths.html</a><br />
<script src="https://gist.github.com/1865900.js?file=url.scala">
</script><br />
<h3>
Attributes and Headers</h3>
After having build a Request (see above), you now have access to some modifier on it.<br />
The probably best way to learn all of them will to check the source code <a href="https://github.com/dispatch/dispatch/blob/master/core/src/main/scala/requests.scala" target="_blank" title="Request">here</a>.<br />
But here is some examples:<br />
<script src="https://gist.github.com/1865900.js?file=attr_headers.scala">
</script><br />
<h3>
Method</h3>
The method is the quite more simple... juste append the name to the request, in order to change the method from 'GET' to another:<br />
<script src="https://gist.github.com/1865900.js?file=methods.scala">
</script><br />
<h3>
Payload</h3>
As for headers, you better check the code for know every tools, the library put in your hands. Here is some conventions I understood from the code:<br />
<script src="https://gist.github.com/1865900.js?file=payload.scala">
</script><br />
<h2>
Executors</h2>
Dispatch works with executors that execute the queries and accepts handlers for handling reponse.<br />
Such response waiting is configurable by choosing among several implementations, the documentation is well suited on the <a href="http://dispatch.databinder.net/Choose+an+Executor.html" target="_blank" title="Executor">wiki</a>.<br />
But in our case, what we need is synchronousity, because the RESTful service is also out backend service. So the executor comes with the `dispatch` package, this way:<br />
<script src="https://gist.github.com/1865900.js?file=executor.scala">
</script><br />
<h2>
Handlers (Response's Body)</h2>
Here's come the <strike>s</strike>fun... Handling responses.<br />
Actually, this piece of code comes in the game just before we apply the request to the executor, but ok, let's takle it now.<br />
A handler can also be called the response parser, that said, it is responsible to parse the whole content into a new form.<br />
Some existing handlers are:<br />
<script src="https://gist.github.com/1865900.js?file=handlers.scala">
</script>
Basically, a handler defines an operator, a result type and takes a block that deals with such result type instance.<br />
Some examples of such handlers are made on the wiki, especially for Json or html (<a href="http://dispatch.databinder.net/Lift+JSON.html" target="_blank" title="Lift">here</a>, <a href="http://dispatch.databinder.net/TagSoup.html" target="_blank" title="tagsoup">here</a> and <a href="http://dispatch.databinder.net/JSoup.html" target="_blank" title="jsoup">here</a>).<br />
Others handlers exists by default, for redirecting to outstream, to compose or chain handlers.<br />
<hr />
In the <a href="http://getting%20close%20to%20the%20real%20use%20of%20neo4j%20with%20play%21/">next post</a>, we'll see how to use Neo4J Json Rest Api with Play! framework through Dispatch.Noootsabhttp://www.blogger.com/profile/07479335886992076807noreply@blogger.com0tag:blogger.com,1999:blog-5531553525163261182.post-68074823603905351672012-02-19T22:19:00.000+01:002012-02-19T22:20:31.536+01:00Neo4J with Scala Play! 2.0 on Heroku (Part 4) :: Play 2.0/Json<h2>
Note</h2>
This post is a continuation of <a href="http://ska-la.blogspot.com/2012/02/neo4j-with-scala-play-20-on-heroku-part_19.html" target="_blank" title="part 3">this post</a>, which is the thrid part of a blog suite that aims the use of Neo4j and Play2.0 together on Heroku.<br />
<h1>
Play2.0 - Scala - Json</h1>
I'm about to write a quick wrap up, of some Play20's <a href="https://github.com/playframework/Play20/wiki/Scalajson" target="_blank" title="Main Json Page">wiki</a> <a href="https://github.com/playframework/Play20/wiki/ScalaJsonRequests" target="_blank" title="Json Request">entries</a> and <a href="http://stackoverflow.com/questions/8695335/how-to-render-json-response-in-play-framework-v2-0-latest-build-from-git" target="_blank" title="Question on SO">stackoverflow</a> that were all related to Json in Play2.0.<br />
For that, I'll take some usage examples, but also present the underlying libraries (<a href="https://github.com/codahale/jerkson" target="_blank" title="Jerkson on Github">Jerkson</a>) and the used paradigm, <a href="http://debasishg.blogspot.com/2010/07/sjson-now-offers-type-class-based-json.html" target="_blank" title="SJson by debasishg">SJson</a>.<br />
<h2>
Scope</h2>
Giving that the wiki pages are really clean and self-explaining, I'm not gonna enter deeply in how Json must be used with Play 2.0 albeit I'll give some prerequesites in order to help you understand how I'll use the Neo4J REST API.<br />
<h2>
play.api.libs.json</h2>
This package contains everything you'll need in order to work with Json in Play 2.0.<br />
It defines important structure like <em>JsObject</em>, <em>JsArray</em> and even some like <em>JsUndefined</em>.<br />
They usage is very easy since they are based on classical scala's <em>Map</em> and <em>List </em>of <em>JsValue</em>(parent type).<br />
Here is an example of creating a <em>JsArray</em> and iterating items. To test it in a REPL, I recommend you to enter the <em>Play console</em> by using <strong>play</strong> in your repo and the <strong>console</strong> in sbt (to have all libraries loaded).<br />
<script src="https://gist.github.com/1865780.js?file=play-repl-test.scala">
</script>
<br />
For arrays, it's quite easy (maybe wrapping could be annoying but a little of pimping can resolve that).<br />
The Jerkson library powerful comes with <em>JsObject</em> usages. It has defined a very clean DSL for querying Json. Here is an example for querying a property or catch a descendant property.<br />
<script src="https://gist.github.com/1865780.js?file=play-repl-test.scala">
</script>
<br />
<h2>
play.api.libs.json.{Format, Reads, Writes}</h2>
Until now, you saw that Json is usable with Play. But, I guess that you hope more than that, since this framework is here to ease the work.<br />
And you're right.<br />
Play is coming with a SJson flavor for serialization and deserialization of DSO.<br />
Three traits come in the game.<br />
<h3>
Reads</h3>
Reads defines a simple method <em>reads</em>:<br />
<script src="https://gist.github.com/1865840.js?file=Reads.scala">
</script>
Having a <em>Reads</em> defined for a type T, we can now extract such instance from Json.<br />
With the object <em>Reads</em> that defines an <em>implicit Reads</em> instance for most common types like <em>Option</em>, <em>String</em>, <em>Short</em> ...<br />
<h3>
Writes</h3>
Writes, like Reads, is very simple and defines a simple method <em>writes</em>:<br />
<script src="https://gist.github.com/1865840.js?file=Writes.scala">
</script>
Obviously, its purpose is to convert a value of type <em>T</em> into its Json representation and is the inverse function <em>reads</em>.<br />
A <em>Writes</em> object is defined too with conversion to common types.<br />
<h3>
Format</h3>
This trait is there to put the pieces together<br />
<script src="https://gist.github.com/1865840.js?file=Format.scala">
</script>
With the help of this trait, we can now have a serializer of custom domain object from/to Json.<br />
A good practice is to define such Format into the <em>companion</em> object of your DSO, so that it will come in the scope at once when using it.<br />
Here is an example:<br />
<script src="https://gist.github.com/1865840.js?file=example.scala">
</script>
Let's see how to use this Format easily to go back and forth from DSO instances.<br />
<h2>
play.api.libs.json.Json._</h2>
This object is an handy one, that defines four convenient methods. Two are here to play with String and JsValue. The other are to play with types and JsValue.<br />
<script src="https://gist.github.com/1865840.js?file=Json.scala">
</script>
Back to our simple DSO class, we can now do this:<br />
<script src="https://gist.github.com/1865840.js?file=usage.scala">
</script>
<br />
<h2>
Enhancement</h2>
In order to have more control on effects of your serialization, I would recommend you to consider the <a href="http://code.google.com/p/scalaz/" target="_blank" title="Scalaz home page">Scalaz</a> library's <a href="https://github.com/scalaz/scalaz/blob/master/core/src/main/scala/scalaz/Validation.scala" target="_blank" title="Validation scala code">Validation</a> construct.<br />
Indeed, it will help you having more relevant information and all at once if you <em>reads</em> is wrong.<br />
Here is a talk about <a href="http://debasishg.blogspot.com/2011/02/applicatives-for-composable-json.html" target="_blank" title="Scalaz Validation SJson">this</a> (but not in Play).<br />
<hr />
In the next post, we'll talk about the Dispatch library. See it <a href="http://www.blogger.com/#" target="_blank" title="Dispatch">here</a>Noootsabhttp://www.blogger.com/profile/07479335886992076807noreply@blogger.com0tag:blogger.com,1999:blog-5531553525163261182.post-55398215798147657992012-02-19T22:04:00.000+01:002012-02-29T23:12:28.605+01:00Neo4J with Scala Play! 2.0 on Heroku (Part 3) :: Play 2.0/Idea<h1>
Play!2.0 Scala and Idea CI</h1>
This post is a continuation of <a href="http://ska-la.blogspot.com/2012/02/neo4j-this-post-is-continuation-of-this.html" target="_blank">this post</a>, where I’ve introduced Neo4J and how to install it. But it requires that you already followed <a href="http://ska-la.blogspot.com/2012/02/neo4j-with-scala-play-20-on-heroku-part.html" target="_blank" title="install Play20">this post</a>.<br />
<h2>
Goal</h2>
Event if the ScalaIDE 2.0 has just been <a href="http://scala-ide.org/" target="_blank" title="scala ide">released</a>. I still prefer Intellij for editing Scala, because even it's <a href="http://www.jetbrains.com/idea/free_java_ide.html" target="_blank" title="idea ci">community edition</a> has a powerful and polish Scala support (after having installed the plugin...).<br />
And while, playing with the 2.0 version of Play! you'll enjoy the use of CoffeeScript to heal your JS head aches, you'll also loved the CoffeeBrew plugin.<br />
Here the goal is to help you create a Play! 2.0 RC1 scala project, update the sbt configuration for sbt-idea, and finally generate the IDEA module.<br />
Having done such easy tasks, you'll have the full IDEA powerfull in your hands.<br />
<h2>
Create the project</h2>
First of all, we need to create the project for what we'll try to do (using Neo4J through it's REST API).<br />
For that, you should have the <em>play</em> executable in your <em>PATH</em> setup, and able to run the following in console.<br />
<code>play new Play20WithNeo4J</code> Which will prompt you some questions, that you'll answer the following (no color question, don't worry but a little remark later if you're using Windows ^^):<br />
<code> What is the application name? > </code><br />
<code>Play20WithNeo4J </code><br />
<code>Which template do you want to use for this new application? </code><br />
<code> 1 - Create a simple Scala application </code><br />
<code> 2 - Create a simple Java application </code><br />
<code> 3 - Create an empty project </code><br />
<code> > 1 </code><br />
<code> OK, application Play20WithNeo4J is created. </code><br />
<code> Have fun! </code><br />
Basically, it asks you the name of your app, and it's language (note that we've chosen the scala way).<br />
<h2>
Setting sbt-idea</h2>
Play! in the new 2.0 version is using sbt to configure the project and run tasks (such as deploy start/run and so on). The sbt version in used is the 0.11.2 at the time writing, and is embedded with the install.<br />
So that, you can already launch the sbt console by either run <em>play</em> or <em>play console</em> to have all Play! deps on classpath.<br />
Before running any command, let's modify some sbt conf files to have sbt-idea being able to create an IDEA module for our project.<br />
Play 2.0 comes with a default configuration for sbt, so that, those files are already present under the <em>project</em> folder:<br />
<ul>
<li>Build.scala : contains the app information</li>
<li>build.properties : contains the sbt version </li>
<li>plugins.sbt : contains default resolver and play 2.0 RC1 Snapshot deps</li>
</ul>
Starting from there three actions are required in order to import sbt-idea.<br />
<strong>Create build.sbt</strong><br />
We have to create a new file name <em>built.sbt</em> in which you'll add the TypeSafe (Scala company) as a searchable repository.<br />
<pre class="brush: scala">resolvers += Classpaths.typesafeResolver</pre>
<strong>Update the plugins.sbt</strong><br />
Add the needed reference to the plugin sbt-idea and add it to the plugin list of sbt.<br />
<pre class="brush: scala">resolvers ++= Seq(
DefaultMavenRepository,
Resolver.url("Play", url("http://download.playframework.org/ivy-releases/"))(Resolver.ivyStylePatterns),
"Typesafe Repository" at "http://repo.typesafe.com/typesafe/releases/"
)
addSbtPlugin("play" % "sbt-plugin" % "2.0-RC1-SNAPSHOT")
resolvers += "sbt-idea-repo" at "http://mpeltonen.github.com/maven/"
addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "0.11.1-SNAPSHOT")
libraryDependencies += "play" %% "play" % "2.0-RC1-SNAPSHOT"
</pre>
<strong>Update the Build.scala</strong><br />
Add the sbt-idea plugin repository in the resolvers' list.<br />
<pre class="brush: scala">import sbt._
import Keys._
import PlayProject._
object ApplicationBuild extends Build {
val appName = "playbasket"
val appVersion = "1.0"
val sbtIdeaRepo = "sbt-idea-repo" at "http://mpeltonen.github.com/maven/"
val appDependencies = Seq(
)
val main = PlayProject(appName, appVersion, appDependencies, mainLang = SCALA).settings(
// Add your own project settings here
resolvers ++= Seq(
sbtIdeaRepo
)
)
}</pre>
<h2>
Create IDEA module</h2>
Now everything is quite simple. Get back to the project root and use <em>play</em>. When entered the console, the module will be build by simply running <em>idea</em>.<br />
Having the <em>iml</em> created, all you have to do is to open Idea, create a project and import the module file right after.<br />
Note, if you're <strong>encountering problem with the scala environment</strong> (that happens at first time). You'll have to create another module (short-lived), then add it the Scala facet where you'll be able to configure the Scala compiler.<br />
When done, you can delete the module, and getting back to the original, you'll be able now to refer to the Scala compiler/libraries.<br />
<hr />
<a href="http://ska-la.blogspot.com/2012/02/note-this-post-is-continuation-of-this.html" target="_blank" title="Json and Play">Let's move to Json and Play</a>Noootsabhttp://www.blogger.com/profile/07479335886992076807noreply@blogger.com0tag:blogger.com,1999:blog-5531553525163261182.post-12514161966791717112012-02-19T21:58:00.000+01:002013-06-18T10:19:11.392+02:00Neo4J with Scala Play! 2.0 on Heroku (Part 2) :: Neo4J<h1>
Neo4J</h1>
This post is a continuation of this <a href="http://ska-la.blogspot.com/2012/02/neo4j-with-scala-play-20-on-heroku-part.html" target="_blank">post</a>, where I've introduced for what is dedicated this suite.<br />
In the current post, we'll talk a bit of <a href="http://neo4j.org/" target="_blank" title="Neo4J"><strong>Neo4J</strong> </a>and why I've considered it as a back end service for storing data in one of my later spikes.<br />
<h2>
Some works on Graph DB</h2>
A graph database is a kind of <a href="http://en.wikipedia.org/wiki/NoSQL">NoSQL</a>, that stores neither data as KVP nor as Column even nor as Collection of document, but as... Graph.<br />
Ok, ok, it's quite obvious but what to say more, maybe that it's querying is very interesting because it relies on a notion of traversal, that would require joins on joins in classical RDMBS.<br />
The main purpose of such graph storing is high dergeed inter-connected data, as Social data are.<br />
<h2>
Neo4J</h2>
I quickly fall on this product for handling my test domain model that figures the social network use case, where users are connected to users, and participates in groups.<br />
Neo4J is written in Java, where a lot of NoSQL database are C++ based. Moreover, where most of NoSQL databases requires standalone installation to work, Neo4J is able to create <a href="http://docs.neo4j.org/chunked/snapshot/tutorials-java-embedded.html" target="_blank" title="Embedded">Embedded</a> databases running instances (look like the jetty revolution for servlet containers).<br />
Finally, my choice was arrested when I saw its <a href="http://docs.neo4j.org/chunked/snapshot/rest-api.html" target="_blank" title="RESTful Interface">RESTful interface</a>; still in working status but which is promising, with url patterns auto-discovery using the service root response.<br />
Final Finally (I promise), I saw that Neo4J community was huge, that <a href="https://github.com/neo4j/spatial" target="_blank">Spatial</a> was already taken into account (GIS has highly inter-connected data, and it'll be probably my next spike). But also, it offers good usage of Lucene as back end indexing provider.<br />
<h2>
Install It, Start</h2>
Quite simple, and I wouldn't expand my self to much on the topic rather than pointing the extractable <a href="http://neo4j.org/download" target="_blank" title="Neo4J Bundle">http://neo4j.org/download</a>.<br />
Drop the extracted folder wherever you want, and set your <em>PATH</em> to target the <em>Neo4J/bin</em> folder where is located the <em>neo4j</em> executable.<br />
When it's done two choices are up to you:<br />
<ul>
<li>install as a service: <a href="http://docs.neo4j.org/chunked/snapshot/server-installation.html">http://docs.neo4j.org/chunked/snapshot/server-installation.html</a></li>
<li>run it by hand by simply do the following in console: $> <em>Neo4J start</em></li>
</ul>
<h2>
Sanity check...</h2>
The server will run listening on your <a href="http://localhost:7474/" target="_blank" title="http://localhost:7474">localhost:7474/</a>. Using your browser you'll directly be redirected to the <a href="http://localhost:7474/webadmin" target="_blank">webadmin</a> interface.<br />
Since I can talk further on this web admin, I'll introduce some features I loved.<br />
<h2>
... WebAdmin (is your friend)</h2>
The Neo4J <a href="http://docs.neo4j.org/chunked/snapshot/tools-webadmin.html" target="_blank">web amin interface</a> offers a way to query your graph very easily through a simple string using tags (like Lucene query string) in order to match nodes, relationships, paths, indexes and so on.<br />
So far so good, and? Ok it presents you the result in an editable table...<br />
Mmmmh exiting... Ok and you can see your graph using their <a href="http://arborjs.org/" target="_blank">arbor.js</a> based viewing tool. Ha ha!<br />
The other tool I like is the web based console for trying <a href="http://docs.neo4j.org/chunked/snapshot/cypher-query-lang.html" target="_blank">Cypher </a>or Gremlin queries or even to try the HTTP REST interface.<br />
<hr />
<a href="http://ska-la.blogspot.be/2012/02/neo4j-with-scala-play-20-on-heroku-part_19.html" target="_self" title="Create Project, setup IDEA">Let's create the project and configure IDEA</a>Noootsabhttp://www.blogger.com/profile/07479335886992076807noreply@blogger.com1tag:blogger.com,1999:blog-5531553525163261182.post-29583480969971346432012-02-19T21:54:00.000+01:002013-05-16T21:43:08.532+02:00Neo4J with Scala Play! 2.0 on Heroku (Part 1)<h1>
Neo4J with Scala Play! 2.0 on Heroku</h1>
In this new posts series I’ll try to gather all steps of a spike I did building a prototype using scala and a graph database.
<br />
<h1>
Chosen Technologies</h1>
<b>Play! Framework</b> as the web framework, in its 2.0 version built from sources.
<br />
<b>Neo4J</b> as the back end service for storing graph data.
<br />
<b>Scala</b> for telling the computer what it should do...
<br />
Here is an overview of what will be covered in the current suite.
<br />
<ol>
<li>How to install Play! 2.0 from Git (or download the last version that must be > beta)</li>
<li>I<a href="http://ska-la.blogspot.com/2012/02/neo4j-this-post-is-continuation-of-this.html">nstall Neo4J and run it in a Server Mode. Explain its REST/Json Interface.</a></li>
<li><a href="http://ska-la.blogspot.com/2012/02/neo4j-with-scala-play-20-on-heroku-part_19.html">Create a Play! project. Update it to open it in IDEA Community Edition.</a></li>
<li><a href="http://ska-la.blogspot.com/2012/02/note-this-post-is-continuation-of-this.html">An introduction of the Json facilities of Play! Scala. With the help of the SJson paradigm.</a></li>
<li><a href="http://ska-la.blogspot.com/2012/02/neo4j-with-scala-play-20-on-heroku-part_2470.html">Introduction of the Dispatch Scala library for HTTP communication.</a></li>
<li><a href="http://ska-la.blogspot.be/2012/02/neo4j-with-scala-play-20-on-heroku-part_21.html">How to use effeciently Dispatch’s Handler and Play!’s Json functionality together. Create generic Neo4J nodes using the result.</a> </li>
<li><a href="http://ska-la.blogspot.com/2012/02/neo4j-with-scala-play-20-on-heroku-part_25.html">Enhance the previous work to create a persistent service that can re/store <i>domain model instances</i>.</a></li>
<li><a href="http://ska-la.blogspot.com/2012/02/neo4j-with-scala-play-20-on-heroku-part_27.html">Create some views (don’t bother me for ‘em … I’m not a designer ^^) using Scala templates and Jquery ajax for browsing model and creating instances.</a></li>
<li><a href="http://ska-la.blogspot.com/2012/02/neo4j-with-scala-play-20-on-heroku-part_29.html">Deploy the whole stuffs on Heroku.</a></li>
</ol>
<hr />
<h1>
Play 2.0 Framework</h1>
This post intent is not to explain how powerful is Play framework (2.0) is. For that I'd recommend this <a href="https://github.com/playframework/Play20/wiki/Philosophy" target="_blank">wiki page</a>.
<br />
However we'll explain all needed steps to build it from sources.
<br />
When I first wronte this post, the released version wasn't suficient for what I needed to do with Neo4J. But now you could just download the RC2 and unzip it somewhere, find it <a href="http://download.playframework.org/releases/play-2.0-RC2.zip" target="_blank">here</a>.<br />
<h2>
Prerequesites</h2>
In this section, we’ll assume that you’ve already setup your scala and git environment (oh yeah and the JDK as well, and not the JRE only! we’ll need javac). If not please refer to those sites:<br />
<br />
<ul>
<li>http://www.scala-lang.org/node/201 </li>
<li>http://help.github.com/set-up-git-redirect</li>
</ul>
<br />
<h2>
G[e/i]t sources</h2>
First of all, open your preferred Git tool and retrieve the sources (warn: choose a unix like path, otherwise you might encountered problems with spaces for instance).<br />
<br />
Use <i style="background-color: #eeeeee;">git clone git://github.com/playframework/Play20.git</i>. And wait for having all sources downloaded.
<br />
<h2>
First step in sbt</h2>
Seconds later, open a console and do the following to run the built tool used by Play 2.0, that is sbt.<br />
<i style="background-color: #eeeeee;">cd Play20/framework</i><br />
<i style="background-color: #eeeeee;">build</i><br />
<br />
This will launch the embedded sbt (0.11.2) which needs some libraries automatically fetched.
<br />
<h2>
Build and fetch</h2>
While being in the sbt-console, you can now ask sbt to build the framework and fill in the local Ivy repository with needed libraries (Play2.0 runtime deps).<br />
<br />
Enter <i style="background-color: #eeeeee;">build-repository</i> in the console and hit enter.<br />
<br />
Minutes later, you”ll be able to quit the console by CTRL+D, and to check what happens in you Play20 folder.<br />
<br />
Actually, aside the framework folder, you have now a folder named <i>repository</i> that contains every needed deps (includind play).<br />
<br />
Let’s check by listing all files in play: <i style="background-color: #eeeeee;">ls Play20/repository/local/play</i> and find libraries such <i>play</i>, <i>anorm</i>, <i>template</i> for the scala 2.9.1 version.
<br />
<h2>
Done!</h2>
Great!<br />
<br />
You’ve just finished the Play2.0 installation.<br />
You can, for convenience, update your PATH to point to the Play20 folder (where resides the play executable)
<br />
<hr />
Let’s move to the Neo4J setupNoootsabhttp://www.blogger.com/profile/07479335886992076807noreply@blogger.com2