Showing posts with label rest. Show all posts
Showing posts with label rest. Show all posts

Monday, June 11, 2012

How Monad Transformer saved my time


Context


These days (this week-end) I wanted to put some work on a Neo4J Rest driver that I'm writing for Play 2.0 in Scala.
The only thing I wanted, actually, is to have the current embryo more functional. I mean I was throwing exceptions (frightening... huh!).
Since this driver is meant to be fully asynchronous (man... it's http, it MUST be) and  non-blocking (Play's philosophy), I was hardly using the Promise thing via the use of the WS api of Play.
This is the kind of thing I've got (briefly):

  def getNode(id:Int) : Promise[Neo4JElement]

Where Neo4JElement stands for the wrapper of all Neo4J Rest response (Json). Hence, it can be either a Failure response (Json with stacktrace and message), it can be a Node, or .... throw an Exception (br...) when the service crashed (f.i.).

Hmm, not so intuitive and goes against the functional paradigm that orders: "you can't ever introduce side-effects, boy!". An exception that blows in my face, is one side effect (head-aches, ...).

Diego Validation to the rescue


Validation is a very simple thing, it holds either a value, either an error...
Ok, why not just Either then. Actually, you're right but Validation that I took in the ScalaZ library contains a lot of thing very helpful for the purpose of validation. But if you worry it, just replace Validation by Either in your mind from here.
Now, here is the getNode signature:

  def getNode(id:Int) : Promise[Validation[Aoutch, Node]]

Isn't it more intuitive? For sure, you get back our relevant type in the signature : Node. Great!

So far, so good now what the heck is Aoutch : something that hurts... and what could hurt a runtime execution... exceptions yeah! Thus, Aoutch is just a shortcut for Either[NonEmptyList[String], Failure]. We can see that we represent with one single type an unexpected exception and a failure (missing node f.i.).

Monad


If you don't know what a Monad is, from here thing about a structure that can evolve in a for-comprehension (in scala it must implement flatMap and map).

Promise and Validation are Monads. And their used one over the another. But what really interests us is the leaf of the chain Node. That introduced some boilerplate code when you try to sequence actions like that:

See... yes we have to skip the first level (validation) to be able to work with the meaningful objects.
But still that we can extract some pattern... no?

Monad Transformer


The pattern that we can extract is kind of two-level composition. I did this composition my-self trying to figure out what we'll be possible.
It was successful but, I've have to introduce a new type and a new method, that was like a flatMap.
So I asked on StackOverflow ^^ (and it was my first question, yeepeee). You can find it here. So I want explain how I did, because the question explains it. But the real question was, is there a well-known functional construction for this problem?.
Thanks to @purefn, I knew that it was the case!

It was time to use Monad Transformer.

Monad Transformer


Briefly (and very roughly), a Monad Transformer is a construction that is able to transform a M[N[_]] into a P[_], where M, N, P are Monads.
I won't explain here how it does, because it would be long, but here is a good link (you've to understand Haskell a bit, sorry).
With the help of such transformer, you'll get you back the opportunity to use for-comprehension... with the wrapped-twice type as bound value.
Here is the the transformer for our Promise[Validation[E, A]]:

And how we can now link nodes:



Awesome! No! We get 3 async and non-blocking calls, totally typed checked and resistant to exceptions and failures... In 5 lines.

Future work


At the SO question, @purefn tolds me that scalaz 7 (snapshot) is defining (fully) this kind of Validation Transformer.
Why didn't I used it, yet:

  • I'm trying to use not Snapshot (not a good reason, but still)
  • In order to use ValidationT, I'll have first to create an instance of the Type Class Monad. Because the flatMap signature needs it in the context.

Conclusion


I love functional (even if I'm still learning -- back -- the basis ^^)

Wednesday, April 18, 2012

Still Playing... but new players are in

Why?

Because I love to play, with Play 2.0 and scala (still learning).
But also because I'm currently investigating technologies that I might choose for a new product line currently building in my company, NextLab (in Belgium).

With ?

This time, I've more played with client side libraries or frameworks (no I won't post yet another entry on JQuery...), but I also tried how it is easy to create totally async code (so parallelized) using server side ones.
The technologies that will be quickly introduced in this post can be found hereafter, but everything has been packaged in a github repo, and a running instance on Heroku.

At first the current spike was dedicated to the slowly growing Play 2.0's Neo4J REST plugin, we are creating at NextLab. But, in order to demonstrate what could be done with the coupling of these two technologies, I've extended the spike's scope to something more funny.

So let's introduce the technologies:

Client Side

Twitter Bootstrap
An amazing toolbelt helping building responsive website without having to bother boilerplates in HTML, CSS.

Even if it is neat, complete and well thought, Bootstrap comes with another handy factor... it has been built using LESS. And by chance (I know, chance is not part of the equation) LESS is supported by Play 2.0 by default.

Just a note, LESS will let you reuse color codes, mixins, etc etc that Bootstrap has already defined.
Spine.js
As we'll below, we'll have to discuss with WS (json), upon which a REST interface has been added for meaningful resources.

That's where Spine.js comes in the game. This lightweight MVC library brought me the small tools that I needed for fetching, saving resources without having to write not even a single request by hand...
d3.js
Probably my favorite (that might be my mathematician part who's talking), this powerful Data-Drive Document toolkit has taken the right thing by the right end.

Its functional approach of decoupling data from the document, and link them using layouts helps you to concentrate on each part of the data usage independently:
  • the incoming/rought data mapping to a representative data
  • the mapping from represented data to document (most of the time, one data for one element)

Communication

The communication layer is of course HTTP, with a little help from the emerging HTML5 features. One in particular, Server-Sent Events (here is a great intro).

This, stable, HTML5 feature comes with the handy functionality to let the server sent events to connected client, without hacks; that said Comet or Polling.

Open a connection, push data, and that ONLY when the server needs to.

Server Side

Play 2.0 (Scala)
Of course... But I used some "advanced" feature like,
Async
Async is a Play 2.0 feature that let the server deal with the tasks it has to schedule.

That is, when you think that the server might have to wait for actions being executed before being able to respond a request, Play 2.0 let you, really simply, create Async request (non blocking).

Very handy when you have to call third parties services for instance...
Iteratee
The only way I would consider from now to consume data. Iteratee is a fairly difficult thing to understand (read this wiki) but it gives you the same smart decomposition that in d3.js, that is, decoupling the management of a sources, its useful representation and its computed result.
Akka
Powerful, actor-based, parallelizer, asynchronous task, scheduler library.

Actually I needed, a request to launch async tasks (you know like event generation and dispatching)... so what else!
Neo4J
A database for storing graph... let's use a graph database.

Within NextLab, we started a open sourced Play2.0 plugin for calling the REST api provided by a Neo4J Server (helpful on Heroku). It's still emerging, and continue to evolve a lot because features are implemented on the fly (need), and a re-pass is forecasted to add a meaningful DSL (like FaKod did).

Why do we make this choice to implement a distinct plugin instead of wrapping the java library?
  1. yet another library, which brings me too much (I need REST only)
  2. I want requests being async and under control

Libraries Repo

See below, we'll use Heroku. So in order to deploy this application wich uses our plugin, I needed to publish somewhere.

This is what can offer Cloudbees. Among other great things like git repo, CI and so on, Cloudbees provides you for free four maven repositories that you can make public if you wish.

So I used sbt to publish on my "snapshot" repo on cloudbees, letting heroky has access to it for downloading the plugin dependency.

Platform

Free, reliable, easy to use with scala and Play 2.0... which else than Heroku?

So what

I think that this post is already too long... However, I can let you play with the resulting app here.

Check out the code (and fork it) there.

Depending on the comment, I will expand this post to other ones to respond potential demands (if there is any ^^).

Good play.

Oh yes, one last note, the application is to Create Stuff which contains dummies. Stuff are created using a simple form that must be fulfilled. Stuffs can be linked one to another by clicking the graph.

Please create Stufs and links, it will be a good test for Neo4J, the plugin and Play 2.0.

Hope you've reach this.

SMAK. hehe

Tuesday, February 21, 2012

Neo4J with Scala Play! 2.0 on Heroku (Part 6) :: Dispatch+Play 2.0

Note

This post is a continuation of this post, which is the fifth part of a blog suite that aims the use of Neo4j and Play2.0 together on Heroku.

Using Neo4J in Play 2.0

In this post, we'll create a Dispatch Handler that handles Neo4J Rest Json calls in Play's Json object.
Having this in our hands, we'll be able to create a really simple service for dispatching Neo4J operations and use them in Play's views.

I've compiled the Play app on github, fork me.

NB: we'll use Dispatch but in case you wish to, you could use the Play's WS feature that might help you a lot (check this out).

Declare Dispatch deps

First of all, we have to update our Play app with the Dispatch dependency. For that, we have to update the sbt configuration file in order to add the related line.

Now, that we have updated the project, let update the application by reloading the configuration (if you're already in sbt console) and rebuild our IDEA project.


Play's Json Handler

Our goal is to use the Neo4J Rest Api that returns responses Json encoded.
So here, I'll show how we could have such response directly unmarshalled in Json object. In further posts, we'll use such handling feature to get Model instances directly (which is far more interesting).

What is necessary for that is to create a piece of code that is capable to take a subject and convert it to a JsValue. And since we love functional programming, let us have this method taking a continuation that accepts a JsValue.
In this listing, we see that we use the text parser to consume the response payload, then we ask the Play's Json parse function do its job.
Finally, we use the continuation applied to the parsed result.

Neo4J Service

Let's gather some utility urls to retrieve node, relations. In other words, urls for common usages. This service is left simple for further enhancements (next post). We see that most functions are there to create urls based on ids, but there is also the root one that directly fetches the entry node.

A Controller To Rule Them All

For the sake of this basic usage of our Handler with Neo4J, here are some examples of such requests. (full controller here).
As we can see, all we had to do is to create the correct url by using id, or Neo4J path conventions, then using the Handler operator ( >! ... how it's Play, no ?!), we have the facility to use directly JsValue instance to consume the result.
Okay, it's repetitive and the Json traversal is not shared. Let's us put this aside until the next post.
And before going ahead, I've created a pretty simple and naive view and url mapping. So check the sources on github, play it and tests the /rest et al. urls.

Next post: Enhance the handler and service to manage Domain Object.

Sunday, February 19, 2012

Neo4J with Scala Play! 2.0 on Heroku (Part 2) :: Neo4J

Neo4J

This post is a continuation of this post, where I've introduced for what is dedicated this suite.
In the current post, we'll talk a bit of Neo4J and why I've considered it as a back end service for storing data in one of my later spikes.

Some works on Graph DB

A graph database is a kind of NoSQL, that stores neither data as KVP nor as Column even nor as Collection of document, but as... Graph.
Ok, ok, it's quite obvious but what to say more, maybe that it's querying is very interesting because it relies on a notion of traversal, that would require joins on joins in classical RDMBS.
The main purpose of such graph storing is high dergeed inter-connected data, as Social data are.

Neo4J

I quickly fall on this product for handling my test domain model that figures the social network use case, where users are connected to users, and participates in groups.
Neo4J is written in Java, where a lot of NoSQL database are C++ based. Moreover, where most of NoSQL databases requires standalone installation to work, Neo4J is able to create Embedded databases running instances (look like the jetty revolution for servlet containers).
Finally, my choice was arrested when I saw its RESTful interface; still in working status but which is promising, with url patterns auto-discovery using the service root response.
Final Finally (I promise), I saw that Neo4J community was huge, that Spatial was already taken into account (GIS has highly inter-connected data, and it'll be probably my next spike). But also, it offers good usage of Lucene as back end indexing provider.

Install It, Start

Quite simple, and I wouldn't expand my self to much on the topic rather than pointing the extractable http://neo4j.org/download.
Drop the extracted folder wherever you want, and set your PATH to target the Neo4J/bin folder where is located the neo4j executable.
When it's done two choices are up to you:

Sanity check...

The server will run listening on your localhost:7474/. Using your browser you'll directly be redirected to the webadmin interface.
Since I can talk further on this web admin, I'll introduce some features I loved.

... WebAdmin (is your friend)

The Neo4J web amin interface offers a way to query your graph very easily through a simple string using tags (like Lucene query string) in order to match nodes, relationships, paths, indexes and so on.
So far so good, and? Ok it presents you the result in an editable table...
Mmmmh exiting... Ok and you can see your graph using their arbor.js based viewing tool. Ha ha!
The other tool I like is the web based console for trying Cypher or Gremlin queries or even to try the HTTP REST interface.

Let's create the project and configure IDEA

Neo4J with Scala Play! 2.0 on Heroku (Part 1)

Neo4J with Scala Play! 2.0 on Heroku

In this new posts series I’ll try to gather all steps of a spike I did building a prototype using scala and a graph database.

Chosen Technologies

Play! Framework as the web framework, in its 2.0 version built from sources.
Neo4J as the back end service for storing graph data.
Scala for telling the computer what it should do...
Here is an overview of what will be covered in the current suite.
  1. How to install Play! 2.0 from Git (or download the last version that must be > beta)
  2. Install Neo4J and run it in a Server Mode. Explain its REST/Json Interface.
  3. Create a Play! project. Update it to open it in IDEA Community Edition.
  4. An introduction of the Json facilities of Play! Scala. With the help of the SJson paradigm.
  5. Introduction of the Dispatch Scala library for HTTP communication.
  6. How to use effeciently Dispatch’s Handler and Play!’s Json functionality together. Create generic Neo4J nodes using the result. 
  7. Enhance the previous work to create a persistent service that can re/store domain model instances.
  8. Create some views (don’t bother me for ‘em … I’m not a designer ^^) using Scala templates and Jquery ajax for browsing model and creating instances.
  9. Deploy the whole stuffs on Heroku.

Play 2.0 Framework

This post intent is not to explain how powerful is Play framework (2.0) is. For that I'd recommend this wiki page.
However we'll explain all needed steps to build it from sources.
When I first wronte this post, the released version wasn't suficient for what I needed to do with Neo4J. But now you could just download the RC2 and unzip it somewhere, find it here.

Prerequesites

In this section, we’ll assume that you’ve already setup your scala and git environment (oh yeah and the JDK as well, and not the JRE only! we’ll need javac). If not please refer to those sites:

  • http://www.scala-lang.org/node/201 
  • http://help.github.com/set-up-git-redirect

G[e/i]t sources

First of all, open your preferred Git tool and retrieve the sources (warn: choose a unix like path, otherwise you might encountered problems with spaces for instance).

Use git clone git://github.com/playframework/Play20.git. And wait for having all sources downloaded.

First step in sbt

Seconds later, open a console and do the following to run the built tool used by Play 2.0, that is sbt.
cd Play20/framework
build

This will launch the embedded sbt (0.11.2) which needs some libraries automatically fetched.

Build and fetch

While being in the sbt-console, you can now ask sbt to build the framework and fill in the local Ivy repository with needed libraries (Play2.0 runtime deps).

Enter build-repository in the console and hit enter.

Minutes later, you”ll be able to quit the console by CTRL+D, and to check what happens in you Play20 folder.

Actually, aside the framework folder, you have now a folder named repository that contains every needed deps (includind play).

Let’s check by listing all files in play: ls Play20/repository/local/play and find libraries such play, anorm, template for the scala 2.9.1 version.

Done!

Great!

You’ve just finished the Play2.0 installation.
You can, for convenience, update your PATH to point to the Play20 folder (where resides the play executable)

Let’s move to the Neo4J setup