Timothy Perrett's Blog

An introduction to simpler concurrency abstractions

2012-04-27T10:00:00+01:00

No matter what programming language you use to create ””the next big thing™”, when it comes to running your code, there will – eventually – be a thread) that has to execute and compute the result of your program. You may be wondering why you should even care about this threading lark? Well, modern computing hardware is typically not sporting faster clock speeds, but instead features multiple cores, or physical processors. If you have a single threaded program, then you cannot make use of the abundant power that modern hardware makes available; surplus cores simply sit idle and unused. For desktop computers this is less of an issue, but for server based applications, having wasted resources that you already paid for is quite an issue.

A common scenario that you are likely familiar with – either implicitly or explicitly – is for your program to run on a single thread. In this situation the program executes imperatively. For readers familiar with C or a scripting language like PHP or Ruby, this generally means the code runs top to bottom. Consider this trivial example:

// create a variable
var foo = 0

// loop and increment the var with each iteration
def doSomething = 
  for(i <- 1 to 10){ 
    foo += 1 
  }

// check the value of foo
foo

This kind of code, irrespective of the exact syntax, should be familiar to anyone who’s ever used a mainstream programming language. When this program is run with a single thread, the result will, as one might expect, be an integer value of 10. Seemingly straightforward.

Now, reconsider what would happen if you ran this same program on two concurrently executing threads that shared the same memory space. If you’ve never done any multi-threaded programming, the answer to this question may not be obvious: as each thread runs its own counting loop, both thread A and thread B will be setting the value of the foo variable. This will have some wacky side-effects in that one thread will constantly be pulling the rug out from under the others feet. This is not constructive for either thread.

This rather unfortunate scenario has several “solutions” that are found in the majority of mainstream programming languages: one of these solutions is known as synchronisation). As you might have guessed from the name, the two concurrently executing threads are synchronised so that only one thread updates the foo variable at a time. In various programming languages, locks) are often used as a synchronisation mechanism (along with derivatives like Semaphores and Reentrant Locks). Whilst this article isn’t long enough to go into the details of all these things (and its a deep subject!), the general concept is that when a thread needs a shared resource, it locks it for its exclusive use whilst it does its business. For all the while thread A is locking, thread B (or indeed, thread n) is “blocked”, that is to say thread B is waiting on thread A and cannot do any work during that time. This gets awkward. Quickly.

Hopefully this gives you a high-level appreciation for the issues associated in writing concurrent software. Even in a small application, concurrent operations could quickly become a practical nightmare (and often do), let alone in a large-scale distributed application that has both inter-process and inter-machine concurrency concerns. Writing software that makes effective and correct use of these traditional concurrency tools is very, very tricky.

Time for a re-think.

As it turns out, some clever folks realised back in the mid-sixties that manual threading and locking was a poor level of abstraction for writing concurrent software, and with that, invented the actor model. Actors essentially allow the developer to reason about concurrent operations without the need for explicit locking or thread management. In fact, Actors were largely popularised by the Erlang programming language in the mid-eighties, where Erlang actors allowed telecoms companies such as Ericsson to build highly fault-tolerant, concurrent systems, that achieve extreme levels of availability – famously achieving “nine nines”: 99.9999999% annual uptime. In fact, Erlang is still highly popular in the TelCo sector even today, and many of your phone calls, SMS and Facebook chat IMs all use Erlang actors as they wind their way across the interweb.

So what is the actor model? Well, primarily actors are a mechanism for encapsulating both state and behaviour into a single, consolidated item. Each actor within an application can only see its own state, and the only way to communicate with other actors is to send immutable “messages”. Unlike the earlier example in the introduction that involved shared state (the foo variable), and resulted in blocking synchronisation between threads, actors are inherently non-blocking. They are free to send messages to other actors asynchronously and continue working on something else - each actor is entirely independent.

In terms of the underlying actor implementation, actors have what is known as a “mailbox”. In the same way that a postman places letters through a physical mailbox, those letters collect on top of each other one by one, with the oldest letter received being at the bottom of the pile. Actors operate a similar mechanism: when an actor instance receives a message in its mailbox, if its not doing anything else it will action the message, but if its currently busy, the message just sits there until the actor gets to it. Likewise, if an actor has no messages in its mailbox, it will consume a very small amount of system resources, and won’t block any applications threads: these qualities make actors a much easier abstraction to deal with and lend themselves to writing concurrent (and often distributed) software.

Actors by example

Enough chatter, let’s look at an example of using actors. The samples below make use of a toolkit called Akka, which is an actor and concurrency toolkit for both Scala, and the wider polyglot JVM ecosystem. I’ll be using Scala here, which is a hybrid-functional programming language. In short that means that values are immutable, and applications are typically constructed from small building blocks (functions). Scala also sports a range of language features that allow for concise, accurate programming not available in other languages. However, the principals of these samples can be easily replicated both in imperative languages like C# or Java, and also in languages such as Erlang.

The most basic actor implementation one could make with Akka would look like this:

import akka.actor._

class MyActor extends Actor {
  def receive = {
    case i: Int => println("received an integer!")
    case _      => println("received unknown message")
  }
}

This receive method (technically a partial function) defines the actors behaviour. That is to say, the actions the actor will take upon receipt of different messages. With the receiving behaviour defined, lets send this actor a message:

import akka.actor._

// boot the actor system
val system = ActorSystem("MySystem")
val sample = system.actorOf(Props[MyActor])

// send the actor message asynchronously 
sample ! 1234

Besides the bootstrapping, the ! method (called “bang”) is used to indicate a fire and forget (async) message send to sample actor instance, which, as you saw from the earlier code listing will output a message to the console. Clearly there are more interesting things to do than print messages to the console window, but you get the idea. At no point did the developer have to specify any concrete details about threading, locking or other finite details. Akka allows you fine control over thread allocation if you want it, but relatively default settings will see you through to about 20 million messages a second, with Akka being able to reach 50 million messages a second with some light configuration. This is staggering performance for a single commodity server. With all that being said, this example is of course very trivial, so lets talk about what else actors (and Akka) can do…

It turns out that the conceptual model of actors is a very convenient fit for a range of problems commonly found in business, and Akka ships with a range of tools that make solving these problems in a robust, and correct way nice and simple. Whilst this blog post is too short to talk about all of Akka’s features, one that is no doubt of interest to most readers is fault tolerance.

Supervisor Hierarchies

Think about the way many people write programs: how many times have you written a call to an external system, or to the database, and just assumed that it would work? Such practices are widespread, and Akka is entirely based around the opposite idea: failure is embraced, and assumed to happen. With this frame of reference one can design systems that can recover from general faults gracefully, rather than having no sensible provision for error. One of the main strategies for providing such functionality is something known as Supervisor Hierarchies. As the name suggests, actors can be “supervised” by another actor that takes action in the case of failure. There are a couple of different strategies in Akka that facilitate this structure:

All for one: If the supervisor is monitoring several actors, all of those actors under the supervisor are restarted in the event that one has a failure.
One for one: If the supervisor is monitoring multiple actors, when one actor has a failure, it will be restarted in isolation, and the other actors will be unawares that the failure occurred.

Simple enough. Let’s see supervision in action:

import akka.actor.Actor

class Example extends Actor {
  import akka.actor.OneForOneStrategy
  import akka.actor.SupervisorStrategy._
  import akka.util.duration._ 
  
  override val supervisorStrategy = OneForOneStrategy(
    maxNrOfRetries = 10, withinTimeRange = 1 minute){
      case _: NullPointerException => Resume
      case _: Exception => Restart
    }
  
  def receive = {
    case msg => ....
  }
}

You can see from the listing that the structure of the actor is exactly the same as the trivial example earlier – its just augmented with a customisation of the supervisionStratagy. This customisation allows you to specify different behaviour upon different errors arising. This is exceedingly convenient, and very easy to implement.

Conclusion

The actor model is a simple abstraction that facilities easier development of robust, concurrent software. This article only scratches the surface of what’s available, but there is a lot of cutting edge work being done right now in the area of concurrency and distributed systems. If this article has peaked your interest, there has never been a better time to get your hands dirty and try something new!

HTML5 Event Sources with Scala Spray

2012-03-17T10:00:00+00:00

With the onset of HTML5 modern browsers (read: everyone but IE) implemented the W3C’s Server-Sent Events feature. This essentially allows you to push events from server to client without any additional hacks like long polling, or additional protocols like Web Sockets. This is exceedingly neat as it allows you to do stream-events with a very tiny API and minimal effort on that part of the developer.

Recently i’ve been doing a fair bit with the Scala Spray toolkit when building HTTP services, so I thought i’d take it for a spin and see how trivial it would be to implement SSE. Turns out it was pretty simple. Here’s a demo:

As mentioned in the demo video, most of the SSE tutorials you’ll find online don’t talk about streaming events, they just talk about having an event source HTTP resource for which the browser will consume events. The massive omission here is that if the server just closes the connection like any regular request, then you essentially have to wait for the retry period to pass before the browser will then automatically reconnect. This is not really that great as it just gives you a more convenient API for polling (how 90s!). On the one hand, if you have a low volume of events, or a bit of lag between updates doesn’t hurt then it may not be an issue, but in the main i’d imagine that most people wanting a stream of events would want exactly that, a stream. This is achieved by using chunking the response server side, and the content flushed to the client with the SSE’s mandatory “data:” prefix will then be operated on, giving you a handy stream of data. Of course, you’ll have to close that connection at some point, but if you set the retry latency to a low value then you’ll just move from stream to stream with very little lag (as per the example video).

In the broad sense, this kind of approach is useful in a wide set of circumstances to improve the user experience. I’ve rambled on about this before, but now with SSE there is a generic way of achieving it without needing a framework or platform that explicitly makes comet-style programming easy: its broadly accessible for all toolkits.

The code for the example application I demoed in the video can be found on github. Enjoy.

Using dust.js with Scala SBT

2011-12-26T19:54:59+00:00

If you’ve been living under a rock for the past few weeks you might have missed LinkedIn bloging about their usage of dust.js. This idea of client-side templating is certainly a very interesting one, and something that is in all likelihood very useful for many large-scale applications currently in production. dust.js is quite nice in that it has no dependencies on any javascript library such as JQuery or YUI. This is really rather handy is it means no prescription on your style of implementation.

However, whilst the dust.js documentation has a wide range of examples demonstrating how to use the templating engine, what it does not have is any examples of how one might actually wrap it up and use it. So, with that I wanted to just write up a quick post to explain some of the things that might not be immediately obvious. Before getting to the meat of the article, it’s worth bearing in mind some terminology:

Template: The source dust.js template
Compiled template: A pre-compiled version of the template; essentially some generated JavaScript that is created at build-time (it’s also possible to create the JavaScript at runtime, but this is heavily discouraged)
Context: Each template can only be rendered with a context. More regularly we can think of this as the data that populates the template variables

In short, the developers writes a source template which is then compiled to JavaScript at build time and then rendered at runtime with a given context; which is likely a dynamic JSON payload.

When I first go going with dust.js I was unsure how to handle this build-time compilation. Frankly, the support for everything that isn’t JavaScript or Node was (and is still mostly) non-existent. This was problematic as I certainly didn’t want to be compiling templates on each and every request for a given HTML page. With that, I set about writing an SBT plugin for dust.js template compilation. This then allows you to have SBT build the dust.js templates into JavaScript that you can simply reference directly from your HTML (more on this later). The plugin itself takes templates it finds in src/main/dust and compiles them to the resource managed target whenever you invoke the dust task from the SBT shell. It’s also super easy to redirect the compilation output to the webapp directory during development so that you can simply use the local Jetty server to tryout your app:

(resourceManaged in (Compile, DustKeys.dust)) <<= (
      sourceDirectory in Compile){
    _ / "webapp" / "js" / "view"
}

But I digress. For more information about the plugin, see it’s README file in the github repo.

After making my own tooling for SBT to work with dust.js, I then got to thinking about how one would actually wrap the dust.js rendering system. Curiously, the examples I found online suggest making an AJAX call to fetch the context, or template data. This strikes me as quite sub-optimal as it would mean creating the following requests:

fetching the general html for the page in the first instance
another request for the dust.js runtime
another request for the dust template
and then another for the context data

All that, just to render one template and add unnecessary lag to the page load whilst the ancillary request for context data takes place (after the initial page load has already completed!). For general page rendering, this seems somewhat bizarre. That is to say, requesting context data via AJAX for say, a single page application could well be a reasonable use-case, but for general pages that render “in full”, it seems silly.

Assuming you’re just rendering regular page content, we can make the separation between things one would want to load and cache locally in the browser, and things that need to be dynamic. For the most part, the context is the only thing you’d want to be dynamic and the more general plumbing code for rendering would probably be reusable throughout your rendering call sites. Consider a simple sample:

The dust template (lets call it home.dust in src/main/dust):

<div>
  <h2>Hey {name}</h2>
</div>

This template will then be compiled to home.js and placed in the resource managed output location (as per your build config). The other thing to probably think about is if had several pre-compiled template files needed on a given page it would make sense to merge and minify them into a single JS file. Alternatively, if you were building a single-page application it might be nice to use a client-side loading framework like LABjs to pull in the template files lazily (i.e. as you needed them). Getting back tot eh example, and considering the aforementioned, lets assume a HTML page that loads the DOM skeleton and the static libraries needed for rendering:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
  <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
  <script type="text/javascript" src="js/dust.min.js"></script>
  <script type="text/javascript" src="js/view/home.js"></script>
</head>
<body>
  <h1>Home</h1>
</body>
</html>

Now, as it stands this page would display the heading “Home” and load the dust library along with the compiled template; otherwise it is essentially static. In order to render the dust template one would have to add something like the following to the page <head>:

<script type="text/javascript">
//<![CDATA[
dust.render("home", { name: "Fred" }, function(err, out) {
  document.getElementById('whatever').innerHTML = out;
});
//]]>
</script>

This simple bit of JS would render the content into the element with the “whatever” ID attribute. Now then, this will work, and prove the concept for sure. What it does not do however is provide a dynamic context, which almost certainly most users would want to do. In this way, I can see why people might see that it made sense to make an AJAX request to a pure JSON backend service to get the context dynamically, but as above, this would be a bit slow and cumbersome. Instead, I think it would probably make more sense to dynamically deliver a call to a rendering function with the desired context. Let’s qualify that: when the page loads, the template has been loaded and so has the dust library, so all that is actually required is have a call site that starts the rendering. I think this need can quite nicely be served by a server-side helper function. In essence, when the page is loaded, the server generates a <script> tag. Here’s an example:

<body>
  ...
  <script type="text/javascript" src="/view/home"></script>
</body>

Which in turn would deliver something like:

draw("home", { name: "Whatever" });

That is, where the draw function wraps the dust rendering system. This is just an example though as the point is that this could be any kind of JavaScript. For instance, it could be a backbone.js view or something along those lines: you deliver the rendering code along with the context. This at least seems more optimal than making a fuckton of JavaScript requests for context objects as other blogs suggest.

In the next post i’ll be showing you how to combine all the latest hipster web programming tools like dust.js, CoffeeScript and Less with a Scala backend using Unfiltered to deliver the dynamic template contexts.

The Year in Photos, 2011

2011-12-17T20:55:19+00:00

This is a fairly personal post, so i’m sorry if you are a regular reader of the Scala articles here - those will resume from the next post!

The Years Events

As is becoming customary, I like to try and review the year nearly past as we look forward to the delights of the next. This blog is the second in a yearly series to make a conscious effort to note what the year was made up of and recall those awesome memories made.

Alpe d’huez, France

Most years I try to get some skiing in with friends, and this year we’d decided to go to France rather than my beloved Italy. Whilst the snow could have been better, we had a great time. I took this picture at the top of the highest gondola lift…

Bathoniean Sunsets

I’m fortunate to live in a beautiful area of the UK, and spring time is quite idyllic here. As the winter thaw gives way to things growing and spouting, the sun once again starts to bathe the evenings with glorious sunsets. This is one such evening looking out over the garden.

QConn London, Scala Community Dinner

March saw QConn return to London for a whole week and bring with it a heard of geeks from all over the globe. Many of my friends were visiting for the conference and with that decided that it would be a good idea to get the Scala folks together for some food, wine and all-round good times. Having people like James and Debasish was awesome fun. Can’t wait to do something similar next year during Scala days. James, we’re still waiting for “scala minus minus” ;-)

California Route 1: Pacific Highway

For anyone who’s been reading this blog over the past years, you may recall that last year I was stranded in mainland Europe during the volcano eruption whilst attending ScalaDays 2010. After rounding up a bunch of stranded folks for dinner, I ended up meeting Jon-Anders - making a long story short, we became good friends and one day decided that what we should do is drive the length of the pacific highway in a convertible Mustang. After a hastily organised flight to america we were all set. What ensued was an utterly hilarious time away where we had arranged nothing in advance and just took things as they came. From huge trees in Redwood country…

…to the winding roads and stunning coastline of BigSur and southern california:

The trip was a complete riot and finishing it off with spending time at Xerox PARC was a real high for me. It was literally like walking the halls of technology fame.

Early Summer Thistles

Getting back from Cali was somewhat of a come down, but I often go for hikes in and around where I live and was out one day and noticed a small set of thistles just growing on this open exposed hill face. Pretty nice, especially with the city backdrop.

Canada, Up-state New York and Niagara Falls.

In summer I was sent to Canada for a meeting (technically just across the boarder in Upstate New York) but this was a pretty awesome experience. Having booked a “Small European Saloon Car” before traveling, me and my colleague were surprised to be presented with a 4.0 litre land cruiser. Clearly “small” has some definition in Canada that i’m not familiar with. Anyway… it was fun to drive around in a ~~tank~~ truck for a few days and enjoy the delights of Toronto and New York scenery:

Of course, being in the area it only made sense to swing by Niagara falls on the way back to the airport. It was my first time here and damn, it’s far, far bigger than you might imagine. It really is a stunning natural spectacle.

End of Summer: Harvest Time

Back in England for the end of the summer is yet another lovely time. The UK was drenched in sunshine for the last few days (weeks?) of August which meant all the farms nearby were busy making the most harvesting their crops and preparing the land for winter. Being out on a walk I picked up this picture just after the field had been harvested:

JavaZone and M.I.T Arctic

Come September it was time once again for the most awesome conference of the yearly calendar: JavaZone! I love this conference; its always well organised, has great talks and really wicked parties. This year was no exception to that rule and proved to again be most excellent. This year we were also joined by some of our american friends, which just added to the delight. Following the conference, some of us head north from Oslo to the M.I.T Arctic lab for a few days of geekery and walking in the tundra. This was a really special highlight of the year and felt like a somewhat magical trip for everyone. Here’s a small sample of what we got up too (not all pictures taken by me, credits where credit appropriate):

(photo credit: Andreas Røe)

And don’t forget the somewhat successful fishing in the fjords!!

It really was an amazing trip. If you’ve never visited Norway, you should, its a stunning country.

Lift in Action goes to production

So we’re nearly at the end of the year, so I should really mention Lift in Action going to print. This was a huge, huge milestone for me personally and concluded a two year unit of work. I should also mention that during the writing of this blog post I had originally neglected to even mention the book and that a friend had to remind me to mention it… I’m sure the psychologists would have something to say about that, but anyway!…

Documenting the Difficulties of Documentation

2011-12-11T20:11:50+00:00

Over the years i’ve been involved in a wide-range of diverse open source projects. Some large. Some small. Some very obscure. But every single one has at one point or another, had a “problem” with documentation. Many of you reading this will no doubt have had a similar experience at some point in your programming career. Perhaps you were a project creator wondering how best to communicate the inherent awesome of something you’ve just created, or perhaps you were that eager n00b trying to get to grips with some new technology you came across on github - in either case, you have to create or consume “documentation”.

Now then, before we go any further I do have somewhat of a problem with the overloaded meaning of the term “documentation”. The widely accepted definition of documentation is the following:

Doc·u·men·ta·tion noun - Material that provides official information or evidence or that serves as a record.

Evidence that serves as a record? That sounds awfully vague doesn’t it. I’d like to suggest that this is, to all intent purposes, too vague. More specifically it is often an ambiguous umbrella term for what people are actually referring too. It’s this ambiguity that causes all manner of problems for software projects and their (would-be)users.

Disambiguating Documentation Components

There are many excellent projects in the Scala ecosystem, and many suffer this apparent “lack of documentation”. Curiously though, each project seems to suffer this affliction in its own specific way… at least, this is often what you might witness on the mailing-list of any project you might care to pick at random. Most projects have some thread or other in the archives of their mailing lists where someone has complained about their documentation, or distinct lack of whatever it is they were looking for.

With this in mind, i’d like to now illustrate what I think are the key aspects of the wider “documentation problem”, and disambiguate their respective intentions.

Tutorials and Introductions

The first thrust of documentation i’d like to define is that all-important introductory material people will need when coming to your project in the first instance. This is quite probably the most difficult type of text to write, particularly if the subject matter is highly technical in nature. It’s typically difficult for the following three reasons:

Assumptions - Assuming the reader fully understands a topic, line of code, operator or anything else must be one of the most common issues. In introductory texts and tutorials its highly likely that the reader will not be in possession of the implicit knowledge that’s relevant to properly grok the topic at hand.
Accessibility - “Joe Developer” can initially be easily scared by large words, complex-sounding terms that originate from academic theories, and terse writing styles. It is so vitally important that your tutorials and introduction texts are accessible. For the most part this may well mean that you have to sweep over some of the finer details, or forego some more abstract possibilities in order to effectively get the point across to newcomers. Ironically, it is often this simplification or frivolity with the facts that programmers struggle with when taking up the pen (ok - keyboard, but you know what I mean).
Authorship - Writing is hard. Be honest with yourself and recognise that you may not be any good at it. During the writing of Lift in Action I had to throw away a whole bunch of manuscript which was either too technical or just plain rubbish. Having a professional team of editors who could help me (re)learn about writing was really key… but i’m aware that this is hardly practical for the common case. The fact is that most of us have not written long documents since high-school or college, and it is incredibly difficult. Don’t forget this when writing your project introduction and tutorials - if you can’t do it the proper justice it deserves, then embrace the fact that us humans all have different skills and find someone who can plug the required gaps. Having your texts reviewed honestly by your peers is also another useful strategy to ensure what you’re writing is actually any good.

Examples and Explanations

The second branch of the documentation umbrella is somewhat of an extension to the first, but I decided to make it separate because its use-case feels distinctly different. With this in mind i’ll add the caveat that yes, examples often form parts of tutorials (and later, references), but in and of themselves you wouldn’t use verbose introductory-style writing within an example. In my mind at least, the primary difference is that example/explanatory documentation is tightly coupled with the code to which it relates. That is to say that the text is more often than not sitting next to the code itself in the comments, which usually means there are certain conventions to follow with regard to syntax and so forth (e.g. ScalaDoc, Wiki markup etc). Critically though, the tone of voice in the explanatory text when compared to that of the introductory & tutorial manuscripts is much shorter, more concise and really focusing on the line-by-line, blow-by-blow goings on of the code. More generally, its reasonable to suggest that examples are about illustrating concepts, and this is where the writing style really differentiates itself from the other branches listed in this article.

Finally, if you’re going to go to the trouble of writing examples and explanations, make sure the code actually works! Nothing is more frustrating to find an example that simply does not do what it should because the code either doesn’t compile or doesn’t run correctly.

Reference materials

Reference material is your last line of defence before forcing users to delve into the source code. Consider the type of person might be using a reference, or what their goal might be? I’d propose that when someone is looking at a reference, they know what they are looking for: they need something specific. It’s probably also fair to assume that before arriving at the reference they will have read tutorials and examples, so there is a degree of implied understanding. Reference materials are usually heavy on details and light on fluffy writing style, which allows the author to be far more technical, and satisfy the aforementioned need for presenting the exact facts.

Unlike the other aspects of documentation writing, references can have the tendency to become quite large; even for mid-sized projects. With this in mind you should take care to refactor the organisation and layout of the reference with each major change and strives for a reference that is logically ordered and consistent throughout. If your reference is massive, then you should seriously consider having a decent search function in addition to a logical layout.

Source Code

Your last line of documentation defence is the source code. That might seem odd, as i’m not talking about comments or ScalaDoc (or similar in your language of choice). I’m talking about types (sorry dynamically typed people!). Type annotations can be extremely useful when reading code and concisely communicating the result or intention of a particular item. With Scala for example, consider these two lines:

// actual code irrelevant
val foo = whatever.map(...).flatMap(...).foldLeft(...).map(...)
val foo: Option[Int] = whatever.map(...).flatMap(...).foldLeft(...).map(...)

Having the simple type annotation frees me from having to mull over the code in order to understand its result; its right there in the type annotation. When moving code between teams, or people, having the ability to simple scan complex blocks of code and understand it is a huge win (IMHO). Sure, make use of type inference where the value is obvious (e.g. simple assignment etc), but where you think something might not be directly understood explicitly annotating can serve as effective documentation.

…In any case, writing readable and well documented code could easily be the subject of a whole other blog post (or indeed, academic paper), so we’ll put a pin in that subject and move swiftly along…

In my mind at least, when your general users complain about documentation, they are typically complaining about one of first three branches of this documentation umbrella.

Isn’t all this a lot of work?

I won’t lie to you, good reader: this will take a lot of time and dedication to do. To do it well, will take more time and a fuckton more effort. However, if you want your software or project to be used by people other than yourself, then it is imperative the documentation is well structured, and exhibiting some - if not most - of the traits in this article. It’s also important that you realise that it will, in all likelihood, be “expensive” in terms of time; this is nearly unavoidable, but it will make your project more approachable and more usable.

Interestingly one thing that you often see are projects that try to distill the documentation effort by promoting community authorship, which when considering what a social activity programming has become in recent times, does not seem like such a crazy idea. The reality however is somewhat different to the ideal: people are often keen to submit bugs and patches, but those same keen people are still typically reluctant to contribute documentation. One could speculate that writing documentation was too tedious, or that perhaps it was not as fun as doing the coding and people didn’t want to spend time in that area… the reason is actually irrelevant, as the result is always the same: without a small core of dedicated people who write, maintain and constantly improve that wiki the whole documentation effort will fail. To qualify that, i’m not saying that community-powered documentation never works, as that clearly isn’t the case. I am however suggesting that by-and-large for most communities it simply does not operate effectively, and this has an overall negative impact on the project as a whole.

Who’s doing it right?

I’m not going to gratify this article with pointing out projects that are “doing it wrong”, as frankly most projects are making a hash of their documentation; irrespective of language or community. I do however want to highlight a couple of projects that are setting an excellent example:

Akka - The Akka team are making a superb job of documenting their project, even with extensive changes to the codebase they are very effective when it comes to ensuring the docs are up-to-date and covering all new or refactored features. The documentation is nearly exclusively maintained by the core team of programmers.
JQuery - Very different to Akka, and in a different community, JQuery has been very effective in delivering core reference materials that allow (and encourage) the wider community to write tutorials, introductions and other helpful articles. JQuery also makes extensive use of illustrative examples, and its good documentation is probably one of the reasons for its apparent ubiquity on the web today.

Both of these projects exhibit dedication on the part of the coders, who are typically the ones authoring the core reference materials and bulk of the ancillary texts. Learn from these projects and others like them. We can all do a better job of documenting our projects, myself included. Say no to undocumented projects, and the next time you’re throwing something on github, take the time to write some documentation… even if its a long README people will thank you for it.

Lift in Action Finally Completed

2011-11-06T17:10:14+00:00

So here we are. The end finally happened. Lift in Action was sent to print last week, representing the conclusion of 20 months of work. Without doubt this has been one of the largest and most difficult projects i’ve ever undertaken. With this in mind I wanted to thank everyone who read the MEAP edition, contributed fixes, asked questions, issued pull requests and everyone who generally helped me with the project! Without your input the book wouldn’t be what it is, and I hope that it serves to be a useful reference for learning and making the most of Lift.

The print copies should be coming out in a week or so and depending where you purchased, your copy should be arriving in the coming weeks.

Thanks again for all the support and kind words during the writing; i’m now looking forward to getting my life back! :-D

Using Apache Shiro with Lift

2011-08-23T15:42:24+01:00

As it stands, Lift only has its proto* traits for user management, and that system has its limitations and you will ultimately end up replacing it in any non-trivial application as your needs change and you need to grow. Whilst this is what those traits are designed for (quick start, short haul), you typically end up rolling your own system for users etc when using Lift, and this can often be somewhat cumbersome or not particularly easy to do well. As this whole user management piece is so often requested, I figured that i’d write a plugin library for Lift.

Apache Shiro is a Java security framework (formally known as JSecurity) and it comes with a fairly abstract set of classes for building systems that have the familiar users, roles and permissions setup. Pretty much most applications these days have some notion of users, customers or some other subject that you care about and might want to conduct access control around. This is exactly what Shiro is designed for, and it ships with out of the box inter-operation with ActiveDirectory and other such repositories commonly found in the enterprise space for managing user data.

Part of the reason that other security frameworks never really took to Lift (or vice-versa) is that Lift has its own mech for managing resource ACLs and it never made sense to separate that into a different servlet filter and somehow munge that together: its not 1990. Fortunately Shiro was fairly easy to integrate with Lift in such a way that it allows you to simply augment your existing SiteMap setup, template markup and even dispatch resources. Currently this integration project is in early stages, and you can find the source code here: github.com/timperrett/lift-shiro

Example

Here’s a quick walkthrough of the various ways you can use the integration within your project. Firstly, lets assume you only want to display a section of content to authenticated users:

<lift:has_role name="admin">
  <p>This content is only available for admins</p>
</lift:has_role>

There are a range of authentication snippets that allow you to define who sees what within your templates, checkout the documentation for more on that. Nextup, what if you want to block access to an page entirely if the user is not authenticated? Just add the following to your SiteMap:

...
Menu("Home") / "index" >> RequireAuthentication
...

By default RequireAuthentication will redirect unauthenticated users back to the URL defined in Shiro.loginURL. Likewise, you can specify whole resources to require a particular role or permission:

...
Menu("Role Test") / "restricted" >> RequireAuthentication >> HasRole("admin")
...

Clearly the SiteMap functionality is implemented as LocParam, so you can implement them within your own Loc types, or simply use them declaratively within the regular SiteMap usage.

This whole integration project wraps the Shiro types, so you only need to configure shiro.ini in the root of your classpath and enter the appropriate realm information as per the regular Shiro documentation, then away you go: password files… active directory… whatever you want.

As above, this project is still early stage, but it does indeed work. I’m currently looking for feedback, so if you have some thoughts or things that would be cool to see, then please checkout the project on Github and fork away.

System Scripting with Scala

2011-08-01T14:46:07+01:00

For the longest time I have used Ruby as my pocket knife for system scripting… you know, just knocking up small little executable files that run helpful tasks or automate yawnful processes. I like Ruby for this kind of thing and it works. Recently a coworker in marketing wanted some automation for something relatively simple and instead of using Ruby I thought i’d just knock it together with Scala and in doing so I came across a neat little thing with the Scala scripting support.

Assuming you have Scala installed, you can execute bash statements and pass them directly to your Scala code. This can be pretty handy as there are certain things that are particularly annoying to do from Scala (and more broadly with Java) like finding out where exactly you are executing too on the file system. Often if you use the good ol’ protection domain trick it will give you a location in /var/tmp, as opposed to the real location of the script. Consider the following:

#!/bin/sh
SCRIPT="$(cd "${0%/*}" 2>/dev/null; echo "$PWD"/"${0##*/}")"
DIR=`dirname "${SCRIPT}"}`
exec scala $0 $DIR $SCRIPT
::!#

import java.io.File

object App {
  def main(args: Array[String]): Unit = {
    val Array(directory,script) = args.map(new File(_).getAbsolutePath)
    println("Executing '%s' in directory '%s'".format(script, directory))
  }
}

Notice the base code between #!/bin/sh and ::!#. This allows you to execute bash script (or whatever script you want) before evaluating this file as a Scala script. This can be pretty handy for certain tasks when doing system scripting :-)

Assume you saved this file as “thing”, you can then execute it like any other script: ./thing

Enjoy.

Using SOAP with Scala

2011-07-26T13:44:14+01:00

It seems I have inadvertently become “that guy who does SOAP with Scala”. Given this illustrious position it seemed prudent to get around to putting up an article that explains how to use the SOAP support now present in “Scalaxb”:http://scalaxb.org/.

Scalaxb is a nice tool for working with XSD Schema from Scala, and recently WSDL support was added. In short this means that you can use native Scala types (such as OptionT) for interacting with SOAP methods.

Prerequisites

Before you get going, its necessary to install Scalaxb, which itself requires “Conscript”:https://github.com/n8han/conscript/, so install that first. With ConScript in place, install Scalaxb by running:

cs eed3si9n/scalaxb

This will install Scalaxb and make it available on your $PATH. You can verify this by running the scalaxb command.

Generating Contracts

With Scalaxb installed and your WSDL to hand, you can run something like:

scalaxb -p eu.getintheloop.sample \ 
        -d src/main/scala \
        src/main/wsdl/weather.wsdl

This will generate a range of .scala files pertaining to the WSDL contract you passed as the last argument to the scalaxb tool. In addition this sample includes the -p to specify your own package and also the -d argument to place the output files into my Scala source directory. In this particular case, I’m using “this SOAP endpoint”:http://www.deeptraining.com/webservices/weather.asmx?wsdl, and you can freely do so too for the sake of this example.

Understanding Generated Classes

Scalaxb generates three separate components:

SOAP protocol classes: These are the ancillary classes that are used to operate the SOAP protocol for things like the message envelope.
Default transport implementation: The HTTP driver to actually POST the SOAP message to the endpoint URI. This default implementation is based on the blocking HTTP “Scala Dispatch”:http://dispatch.databinder.net/Dispatch.html
Your service contracts: This is the only service specific part of the generated files. These relate to your specific service whilst the other two parts are completely decoupled and reusable over implementations / services you might have.

Enough talking, lets look at some code.

Usage

The Scalaxb SOAP mechanism makes heavy use of the cake pattern, and thus allows you to mix and match different components as your situation dictates. Here’s the default usage pattern:

package eu.getintheloop.sample

import scalaxb._

object Main {
  def main(args: Array[String]){
    
    val remote = new WeatherSoap12s 
       with SoapClients 
       with DispatchHttpClients {}
    
    println {
      remote.service.getWeather(Some("New York"))
    }
  }
}

Note specifically that by combining the weather service contracts with the SOAP protocol classes and the transport implementation, the result is a working driver from which you can directly call the SOAP methods.

The result of the service call itself is a Either[Fault[_], Option[T]], so if you want to get at the actual value you can just simply fold on the Either.

Suggestions for Customisation

If you have multiple services to interact with then it can be nice to cake together all your service implementations so that you just have a single client that lazily loads the appropriate service classes as needed. Additionally, you could also replace the transport implementation with something asynchronous and based on futures… that also is fun.

The sample code for this example can be found at: “https://github.com/timperrett/scalaxb-soap-example”:https://github.com/timperrett/scalaxb-soap-example

Enjoy.

Running SproutCore from within Lift

2011-05-25T09:13:21+01:00

If you want to run “SproutCore”:http://sproutcore.com from within your Lift application there are a couple of configurations you need to ensure you apply within your Boot.scala in order to actually make it work as you might anticipate.

Firstly you need to ensure you set the HTML parser to use HTML5 and not XHTML, otherwise your templates will explode in extraordinary fashion when using the built template file as created by SproutCore build tools:

<code>LiftRules.htmlProperties.default.set((r: Req) => 
  new Html5Properties(r.userAgent))
</code>

With that in place you need to do the equivalent of implementing a symlink for those setups that use the filesystem (a la PHP, Rails etc). Within Lift this is accomplished by using a stateless rewrite:

<code>LiftRules.statelessRewrite.append {
  case RewriteRequest(ParsePath("index" :: Nil, "", true, false),_,_) => {
       RewriteResponse("static" :: "todos" :: "en" :: "1.0" :: "index" :: Nil)
  }
}
</code>

This code tells Lift to rewrite the root path (/) to /static/todos/en//index template. Also be sure to implement your SiteMap so that only / is accessible and not the full (direct) SproutCore template path. If I do much more stuff with SproutCore I may well end up making an SBT plugin that automatically builds the updated JS files and copies them to your src/main/webapp path… not sure yet, we’ll see. Eitherway, the rewriting is certainly a good candidate for stuffing into a LocParam and making it reusable based upon some project configuration. For example:

<code>Menu("Home") / "index" << UsesSproutCore </code>

For more information like this checkout my book on Lift launching this quarter on Manning Publications.

SBT gets CloudBees support (and fast deploys!)

2011-04-19T10:33:13+01:00

About 18 months ago a company came out of nowhere called Stax Networks who were offering the ability to host JVM-based applications in the cloud, for FREE. This was too awesome not to look into and I subsequently made a plugin for SBT that automatically deployed your WAR files with a simple command.

Stax themselves were recently taken over by “CloudBees”:http://www.cloudbees.com and from what I can see the service has done nothing but become more awesome since that has happened. Now that take over is bedding in, the API has changed somewhat so I thought i’d rewrite my Stax plugin to work with CloudBees instead and take advantage of the ability to delta WAR files and only publish the updates.

Speedy Deploys

The way this works is the build tool creates a WAR file like it would do normally, but as this is usually fairly large it can be exceedingly annoying to have to keep waiting for the deployment to take place. Rather than wait, the build tool calculates the difference between the deployed artefact and the one you just created, and subsequently only deploys the difference (or delta): essentially reducing your deploy time from 15 mins or more to a few seconds.

Gimmeh It

In order to get started with using the SBT plugin, you will of course need a CloudBees account and upon registering you need to grab the key and secret from grandcentral.cloudbees.com, which should look something like:

These values represent the API Key and API Secret that CloudBees will use to verify your deployment rights. Once you have these two values, you can do one of two things in order to have them recognised by the SBT:

Enter them when the plugin prompts you; this will be on everytime you run a deployment to the cloud so is potentially a little sub-optiomal.
Create the properties file $HOME/.bees/bees.config so that you only need to define them once per computer. This properties file needs to be a key-value pair which should look something like this:

bees.api.key=XXXXXXXXXX bees.api.secret=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX=

Whichever route you choose to specify that information, you then only need to define the plugin information in any given project. Specifically, in the Plugins.scala file define the following:

import sbt._
  class Plugins(info: ProjectInfo) extends PluginDefinition(info) {
    lazy val cloudbees = "eu.getintheloop" % "sbt-cloudbees-plugin" % "0.2.7"
    lazy val sonatypeRepo = "sonatype.repo" 
      at "https://oss.sonatype.org/content/groups/public"
  }

Add the plugin to your SBT project like so:

  import sbt._
  class YourProject(info: ProjectInfo) extends DefaultWebProject(info) 
    with bees.RunCloudPlugin {
    ....
    override def beesUsername = Some("youruser")
    override def beesApplicationId = Some("whatever")
  }

Again, if you would prefer to enter these values when you deploy your application then you can of course just enter the appropriate values when prompted. Now your all configured and good to go, there are two commands you can run with this plugin:

Get a list of your configured applications: bees-applist
Deploy your application bees-deploy

Upon running bees-deploy for the first time there will of course be a wait whilst the first version is deployed, but all your subsequent deployments should be much, much quicker. CloudBees is a rather awesome service and i’d highly recommend checking it out for fast, easy deployment of your JVM web applications.

Find the source code for the CloudBees SBT plugin here

Color: You Are Your Device

2011-04-18T14:23:53+01:00

If you’ve been living in a cave for the past months, you may not have seen “Colour”:http://www.color.com/. These guys “raised $41m in investment”:http://www.businessinsider.com/color-deal-2011-3 which in and of itself is seriously impressive. After that hit the headlines, I decided that just like everyone else i’d grab me a chilled bottle of tech koolade and try it out.

Now, I want to say here that my first impression of Colour was, well, underwhelming. When you’re alone, its useless. However, when you’re at an event, or gig, or somewhere else with a number of other people it was actually pretty awesome. Being able to collect a whole set of photos and events from a range of vantage points was quite awesome… It’s like event snaps, but on steroids because you automatically have everyone else’s.

There is however a rather huge “but” that has just reared its ugly head today. Colour identifies devices not people. This small but subtle difference is somewhat of a killer for Colour. Case in point: At the weekend I dropped my iPhone 4 by accident and totally destroyed the screen… this meant I had to get a replacement, which I did this morning. Upon reloading Colour from my backup, I no longer have all the pictures and groups from my old device because it thinks i’m someone “new”.

Guys, really, $41m and you cant make a login? I actually really liked the app, but having it not come with you when you change device is just such a killer. People these days have a really high turn over of mobile devices, so it simply doesn’t make sense to bind to the device… the service is meant to be about people and the photos they take, so perhaps it would be a super great idea to stop using the device ID as a route to identify people.

It generally feels like this could have been a really cool app, but its just lacking in some key areas.

End rant.

Using Type Classes for Lift Snippet Binding

2011-04-11T14:56:59+01:00

Back in 2009 I was inspired by “this article”:http://logji.blogspot.com/2009/10/composable-bindings-in-lift-part-ii.html by “Kris Nuttycombe”:https://profiles.google.com/kris.nuttycombe/about which demonstrated how to use implicit conversions to declare composable, reusable Lift snippet bindings.

At the time that article was written, Scala was still in the 2.7.x series and Lift only had the bind() helpers. As I write this today, Lift has its new CSS-style transformers and the Scala language has moved on quite considerably - especially with regard to the handling of implicits. With this in mind I wanted to show how one could make a similar setup but by leveraging Scala context bounds and Lift’s CSS-style transformers.

Use Case

Consider that you were building a system that displayed products. It’s quite feasible that everywhere you wanted to display said product a same or similar binding would be required. A classic point in case is the product listing vs the checkout; the latter would only require a few of the product aspects to be displayed but there would be enough commonality with the listing usage as to not want to repeat yourself.

For the sake of this article Product will be represented like so:

<code>case class Product(name: String, price: Long)
</code>

Binding

In order to make something implicitly bindable, that is to let the compiler know when it should apply the in-scope binding we need to make a bit of infrastructure to facility this:

<code>import net.liftweb.util.CssSel

object Bindings {
  class Bind[T](what: T){
    def bind(implicit bind: DataBinding[T]) = bind(what)
  }
  type DataBinding[T] = T => CssSel
  implicit def asCssSelector[T](in : T): Bind[T] = new Bind[T](in)
}</code>

Here the Bindings object contains a converter class call BindT whos method bind takes an implicit parameter of DataBindingT. In more lay terms, given T (ultimately, Product type) apply an implicit conversion from T => CssSel, where CssSel is the internal Lift type for CSS transformers. Finally, the asCssSelector method does the actual conversion to a BindT instance just like any regular implicit conversion.

So all that remains is to implement a binding for Product and show how you can use that in your snippet classes:

<code>import Bindings._
import net.liftweb.util.Helpers._

object ProductBinding extends DataBinding[Product]{
  def apply(product: Product) = 
    ".name" #> product.name
}

class MySnippet {
  private implicit val binder = ProductBinding
  
  def render: CssSel = 
    Product("whatever", 1234L).bind
}</code>

In this example, ProductBinding defines the generic binding that one requires for Product usage within the system. In your code, this would certainly be a lot more fully featured, but for the sake of example it just binds the name. Within the MySnippet class one can then simply call Product("whatever", 1234L).bind and thats all that needs doing. In order to put the implicit binding in-scope, the ProductBinding singleton is just assigned to a value. This could have just as easily been held somewhere else in a common binding object and then imported to save the clutter, but here it is explicitly defined for the snippet.

There is an alternative syntax which you might prefer for the snippet implementation that uses context bounds like so:

<code>def render[Product : Bind] = 
  Product("whatever", 1234L).bind
</code>

Importantly, with both implementation styles not that you can also add snippet-specific additions to the generic binding. For example:

<code>def render: CssSel = 
    Product("whatever", 1234L).bind & 
    ".thing" #> "example"
</code>

Hopefully you found this useful. For more information like this, please pick up a copy of “Lift in Action”:http://affiliate.manning.com/idevaffiliate.php?id=1138_235

HTTP Dispatch Guards in Lift using PartialFunction

2011-02-23T13:06:46+00:00

There are many little known features within the Lift code base and one that i’ve been meaning to blog about for quite some time is the ability ot use partial functions as guards for HTTP dispatch.

Let’s assume that your system requires some state to be set, like a session variable to ensure the user has logged in before they can access other parts of the service. The following example shows you you can implement a simple partial function to act as a guard.

What’s the goal?

The aim is to be able to define your services like so:

LiftRules.dispatch.append(withAuthentication guard Service)

This gives you a fairly readable API that even with little knowledge of the underlying API you can see that you must have an authenticated session. In this super simple case, I’ll just implement authentication as a Boolean in a SessionVar, but you get the point.

The Service

Consider this service definition:

import net.liftweb.http.rest.RestHelper
import net.liftweb.http.{PlainTextResponse}

object Service extends RestHelper { 
  serve {
    case "protected" :: Nil Get _ => 
      PlainTextResponse("Ohh, secret")
  }
}

This is a super simple service using HTTP dispatch in Lift. Let’s then define the dummy “login” url. In practice this would actually be some kind of bonified authentication but for this example it will suffice.

import net.liftweb.http.rest.RestHelper
import net.liftweb.http.{SessionVar,OkResponse}

object Security extends RestHelper {
  serve {
    case "login" :: Nil Get _ => 
      LoggedIn(true)
      OkResponse()
  }
}

object LoggedIn extends SessionVar(false)

As you can see, simply hitting the URL /login will set the session to be logged in. Completely useless for production, but it serves my means here.

The Boot Configuration

Finally, now you have the services setup its time to write it all together within the Boot class.

import net.liftweb.http.{LiftRules,Req}
import net.liftweb.util.Helpers._ 
import sample.lib.{Security,Service,LoggedIn}

class Boot {
  def boot {
    val withAuthentication: PartialFunction[Req, Unit] = { 
      case _ if LoggedIn.is => 
    }
    
    LiftRules.dispatch.append(Security)
    LiftRules.dispatch.append(withAuthentication guard Service)
  }
}

The key here is the import of Helpers._ to provide the implicit functions to enable guard to be called on the withAuthentication partial function, and the withAuthentication value simply passes all cases to an if statement that accesses the value of the SessionVar. Pretty simple, eh?

Enjoy. All the code for this sample can be found “here”:https://github.com/timperrett/lift-dispatch-guard

A Long Road To The Final Chapter

2011-02-13T20:44:54+00:00

Today is a landmark. I’ve been working on “Lift in Action”:http://manning.com/perrett/ for one year and one month, and finally, this evening, I’ve reached the last chapter. The manuscript alone is currently sitting at around 120,000 words and represents the single largest body of work I have ever completed in my entire life. To that end felt the need to share this jubilant moment with you, dear reader.

Chapter 6 is the only remaining part of the book and I have now started writing it! If you have bought the MEAP and and given feedback via the forum and emails, I thank you - your feedback thus far has been utterly invaluable. I have many corrections and revisions to include based on your comments and they are coming (promise!), so you should see them appear in the MEAP over the coming weeks accompanied by the as-yet unreleased chapters.

It’s been a seriously long road thus far but the end is finally in sight! Cheers to all my readers, friends and family for their support and contributions!

Memoization of Lift localeCalculator

2011-01-30T17:53:10+00:00

Within Lift each request has its locale calculated from scratch. Whilst this is fine with the default setting of reading from the accept headers on the incoming request, when you want to perhaps conduct something more complex you can inadvertently add quite an accidental overhead to the processing of each request. You can however ensure that the localeCalculator is only executed once by memoizing the value of the locale for the whole request cycle.

The following code memoizes the locale computation into a variable that will exist for the full lifetime of a page request ensuring that you do not take the overhead of processing the locale a bunch of times.

gist id=803049

This is obviously a basic implementation that carries a smalloverhead, but the principal here is the same irrespective of what it is you are actually doing :-)

Find great tips like this and more within “Lift in Action”:http://affiliate.manning.com/idevaffiliate.php?id=1138_235

The Year in Photos, 2010

2010-12-28T12:22:00+00:00

Wow, the 28th of December. This year has utterly flown by! What a year its been; i’ve done a lot and experienced a whole set of new countries, met some awesome people at different conferences and generally had a great time! With this in mind, I thought I would just share some of my favourite moments from 2010 as we look forward to 2011.

New Years Day, 2010

The 2010 new years day was a crisp but cold day and I decided to go for a walk along the canal and managed to snap some great pictures of the bridges and people from the barges.

Livingo, Italy

In early January I went skiing with some friends to the beautiful mountain village of Livingo, Italy. I absolutely love Italy and this was one of my favourite shots from the holiday. Everyone is super friendly, the food is amazing and the skiing is great. What more could you want?

Sestiere, Italy

After having been persuaded by my “friend”:http://twitter.com/#!/mikeedwards83 that going on another ski holiday would be a good idea, we ended up in the Olympic resort of Sestiere. Unfortunately I had a nasty accident on the first day and received a broken hand. Amazing scenery though!

Scala Days, Switzerland

At long last Scala Days 2010 was upon us! This was easily one of the best conferences of 2010, anywhere. Really great people and going to EPFL was like a complete pilgrimage.

As some of you may remember though, there was a the small matter of the icelandic volcano eruption which made flying impossible. An adventure then ensued as we all tried to get back to our mother countries by whatever means possible. For me that meant trains, hitchhiking and ferries! I took this one as we steamed away from France and I can tell you now, i’ve never been more relieved to be on a ferry! It was a truly epic adventure.

Road Trip!

A friend of mine was getting married in June and his best man has the rather genius idea of doing a Top Gear-style rally up to the most northern point in the great british isles… there was however a catch, we had to buy the most crappy car we could lay our hands on for no more than £300, drive up there and back, and then sell it. This was one of the most hilarious weekends of 2010 and with all the breakdowns and poor weather it was just awesome as we drove a car that was older than me 1500 miles over four days. Epic.

Beautiful June

This time last year I moved into a new house that has the most amazing wildlife in the garden; all year round there are things growing or living there. Being a country boy at heart, this has been really enjoyable and I snapped this flower on a fine summers day in the garden.

Florence, Italy

As the summer came, work was taking its toll and the need to visit Italy again was becoming irresistible. So, one Monday evening I booked a flight to florence the next day and just got on a plane and relaxed in the tuscan country side for a week of serious culture-vultur’ing. Florence is an amazing city and if you ever get the chance to visit, either with a loved one, or solo, its amazing. I took this picture from piazza michelangelo, the highest lookout in Florence. The bridge in the scene is the famous Ponté Vechio. L’amo Italia.

Hyderabad & Mumbai, India

“Viktor” and Sandra came to visit me in the summer, and that was just awesome. Whilst they were here work decided that it would be a great idea to send me to India the following week! I reluctantly accepted the assignment and started to arrange visas etc, as India was not somewhere that really appealed to me travel-wise. How wrong could I have been… India was AMAZING. Every single corner of every view you could take is crammed with vibrant colour and people with smiling faces. Its a truly humbling country and a must-visit place for any westerner. The two pictures below are from some time spent sight-seeing; this first one is Charminar, an ancient lookout post.

The second one here was a moment captured at the top of Golconda Fort; these two lovers had been looking out over the landscape for ages; and what a view it is looking down on the ~7million inhabitants of Hyderabad.

Israel

As November came, I was once again sent to Israel to work on a research project. This trip was specifically memorable thanks to “Gal”:http://twitter.com/#!/gktheelder and “Erik”:http://twitter.com/#!/erikzaadi. This view was taken over the Herzliya beach early evening. Israel is awesome!

Snowpocalypse!

With December came snow. Lots and lots of snow! This picture was taken about a week ago, and reminds me why I love living in the UK. Its like a winter wonderland!!

Distributing Critical State With ContainerVar

2010-12-10T09:31:15+00:00

Lift 2.2 brings a lot of new features to Lift, and one of them is ContainerVar. I previously blogged about it but did not have time then to make a screencast or sample. Now I have done both :-)

This and many more techniques are discussed in my book, “Lift in Action”:http://affiliate.manning.com/idevaffiliate.php?id=1138_235.

Find source code here: https://github.com/timperrett/lift-terracotta-example

Implicit REST Services with Lift

2010-11-23T17:53:20+00:00

Do you ever have those days, when you wake up and become unhappy with the “code you wrote last year”:http://blog.getintheloop.eu/2010/03/29/multi-format-rest-services-with-lift/ ? Today was one of those days (technically speaking it was yesterday, as I went to bed thinking this, but still). There simply must be a better way. So, im going to blame this post on “Ross”:http://twitter.com/dridus and “Jon-Anders”:http://twitter.com/jteigen . Those two are truly to blame for my interest in type-classes and compiler voodoo… anyway, I digress…

DISCLAIMER: The following post makes heavy use of Scala implicit conversions and type system foo. Its not big, or clever, but it is fun. Before someone points it out, yes, there is a far simpler route to implement something similar. This was ultimately an experiment :-)

The Goal

I write a lot of services. These services do a lot of different things and more often than not need to be published in a bunch of formats. In this way, its often easy to end up with one of two things:

Accidental complexity
Code duplication or blurred lines between logic process and media representation

In this post i’m going to show you one possible method of avoiding the second problem, but, subjectively, the first problem may seem better or worse depending on your familiarity with the Scala type system.

Plan of Attack

Ok, so how are we going to do this? Well, as you may or may not know, Scala has this awesomeness called “implicit conversions”:http://www.codecommit.com/blog/scala/scala-for-java-refugees-part-6 - these can be used to get the compiler to convert type A to type B when it only has A, but requires B. This is useful because the Scala compiler will use the appropriate implicit conversion that it finds within the given lexical scope; essentially meaning that you can “inject” different behaviour at different points during compile time. With this in mind, its a concept that matches quite closes to multi-format services. That is, I have type A from my logic or domain process and I need type B for the representation and delivery of a service.

We will be making a very small sample application that

The Book Store

Before we get into the utter type-madness, let us define the basic domain classes we’ll be using:

gist id=712032

I think you’ll agree this is super basic. Basically its just a fake bookstore that has a super limited stock of only Harry potter and Scala books. Strange, but true. Anyway…

Define the Service Types

We’ll need some traits to glue this stuff together and provide the right syntax. ReturnAs[A, B] - This structure is basically used to mediate each format from its input to its output; in and of itself it does not do a lot. The companion object here is actually where the good stuff is: the implicit method f2ReturnAs takes a function A => B and converts it to a ReturnAs type. This is going to be handy when it comes to writing the implementation as we’ll be able to say Book => LiftResponse etc.

The next structure is the Return[A] type and its nested type As[B]. The Return type will be used as an explicit helper within the service implementations, so the API would be Return() via the companion object apply method. In addition to the single parameter, this method also has an implicit parameter of: implicit f: ReturnAs[A, B] which will be made available to use from within the implementation call via “Scala context bounds”:http://stackoverflow.com/questions/2982276/what-is-a-context-bound-in-scala. This context bound will be expanded to plain type parameter T, together with an implicit parameter so that we can “recapture” the correct return type within the implementation.

gist id=712060

Format Conversions

With the plumbing in place, one needs to actually provide implementation of the various A => B type conversions. In this exceedingly small sample, im only going to be working with ListBook, so to keep this sample as simple as possible these are the implicit conversions for making an XML and plain text service:

gist id=712162

I wont labour the details too much here as this is pretty standard LiftResponse stuff that you can find in my other articles. That being said, there is an interesting point of note, and that is the type signatures:

Return[List[Book]]#As[XmlResponse]

Here we use a type projection to specify the return type - this is pretty sick I think you’ll agree.

Service Implementation

With the media type conversions in place, the only thing that is actually left is within the service creation element is the writing of the method implementations.

gist id=712152

So this is important. This is where we use the context bound that we talked about earlier comes into play in expanding to a type parameter and an implicit parameter which allows us to write these rather nice looking methods wrapped in a the Return objects apply method without needing to specify any specific types.

HTTP wire-up

Last but certainly not least is writing this awesomeness up into the HTTP implementation. To do that we are simply going to use Lift’s REST DSL that needs a return type that is some subtype of LiftResponse. Of course, here we have those so we get a rather nice looking implementation:

gist id=712177

To test this out, all you need to do is visit /bookshop/books to get the text representation and /bookshop/books.xml to get the XML version. Hope you’ve enjoyed this article, you can find full source available to download from here

Find more information like this within my book, Lift in Action

Meet Lift’s ContainerVar

2010-11-18T21:41:46+00:00

For those of you who have been following or working with Lift for sometime, you will know that our implementation of sessions provides a strongly-typed variable holder, but had a limitation in that it was not serializable across application servers using familiar scaling technologies like Terracotta. This is completely fine for the vast majority of use cases and having the extra type-safety and flexibility that SessionVar brings is valuable.

However, there are situations and organizations that require scaling via their existing Java infrastructure and up until now Lift was unable to service that. Well, that is no longer the case as Lift has a new variable buddy called ContainerVar to sit alongside RequestVar, SessionVar and TransientRequestVar.

What’s the bobby?

Firstly, its important to understand that ContainerVar is not a replacement in any way-shape-or-form for SessionVar. It is an accompaniment to provide people who want to use a session store that is backed by the container. This does however come with several limitations in that you are then restricted to storing session data in strings, ints and so on that are moderately basic types which are serializable.

How’s it work?

Glad you asked.

object MySnippetCompanion {
  object mySessionVar extends ContainerVar[String]("hello")
}

As you can see, this is just like the other state variables within Lift. All the plumbing is abstracted away from you, so all you need to do is ensure your container sessions are configured correctly.

I was hoping to make a sample application using “HazzleCast”:http://www.hazelcast.com/ or Terracotta but alas I need to get on with writing “Lift in Action”:http://manning.com/perrett/ so you good reader can learn more about the deep workings of Lift.

Distributed Data Networks with GaianDB

2010-10-31T15:37:00+00:00

This weekend has been a rather fascinating one. It was BathCamp 2010 where I gave a couple of different talks on Scala and NoSQL and also listened to others talk about new and interesting tech. One of those talks was by Dale Lane from IBM about their AlphaWorks project GaianDB

Firstly, a very interesting talk by Dale. I have monitored the NoSQL space for the past 18 months with a close eye. The way in which we are building applications and the scale to which we as developers have to take said apps is really reaching new, dizzying heights - these are exciting times. With that said, GaianDB was a new one on me, and very out of left field.

So what is this thing?

Well, I must say that the “DB” part of the name feels a little wrong, as Gaian is essentially a way of treating lots of heterogeneous distributed data stores as one, coherent unit that can be queryered and accessed from any part of the “network”. IBM claim that Gaian was:

biologically inspired in that it strives to minimize network diameter and maximize connections to the most fit nodes. GaianDB advocates a flexible “store locally, query anywhere” (SLQA) paradigm.

This strikes me as pretty interesting for several reasons:

1. Today is the age of the supposed “cloud” where centralised data storage and large scale applications rule. It takes a brave team to buck that trend and go for a strategy that is different.

2. Being connected all the time, anywhere, with anything is, alas, still a dream. There are many business situations and organisations that are physically unable to be connected all the time perhaps because it simply is not practical for either cost or network performance reasons.

3. Whilst technically possible to hold large volumes of data centrally, for a lot of telemetry-style applications this may not actually be the best route.

4. With more and more people/things becoming “connected”, having a system that can be both independent and collectively connected seems… interesting.

Here’s an example of Gaian nodes within a visualisation system:

Some Figures (from IBM)

Given 1000 Gaian network nodes, and one million rows of data the following was recorded:

1. Query Time - We are able to query all 1000 nodes in about 1/8 second. The results show that the query time grows logarithmically - in other words as you add more and more databases, the increase in query time slows down, providing excellent scaling. The way that a Gaian Network is grown from individual nodes automatically ensures this behaviour.

2. Fetch Time - We are able to fetch 1 million rows of data in under 5 seconds. The fetch time is proportional to the amount of data returned so that if you fetch twice the data it takes twice as long regardless of which of the 1000 nodes the data resides in. The Gaian Database actively pre-fetches the data from all the nodes to achieve this scalability

3. Concurrent Queries - I injected queries from up to 40 nodes at the same time, the Gaian Database showed that it could handle these queries robustly with a modest increase in the query time due to running out of available processor time on our test platform.

Tentative Conclusion

For some unbeknownst reason, I do find this sort of thing really interesting. I think there are a bunch of use cases where this type of workflow could be come useful - I mean, its essentially like skynet… small pieces of tech that can comunicate to other pieces of near-by tech and also be commanded as a large whole. To that end, I’ll be keeping an eye on this project to see how things evolve over time and pondering some other use cases for such a technology. Who knows, perhaps we’ll see some kind of interactive advertising that merges RFID and smart telemetry over similar data networks to taylor customer experiences or something.

Using Lift’s AutoComplete Widget

2010-10-13T20:08:00+01:00

In this 5 min tutorial I am going to cover how to implement Lift’s AutoComplete widget. I assume that you already have a Lift project setup with SBT.

Widget Dependency

First, add the widget dependency in your Project.scala:

val widgets = "net.liftweb" %% "lift-widgets" % "2.1"

Configure your Boot class

Widgets generally need to be wired into the ResourceServer in order to serve their CSS and JavaScript etc from the widgets JAR, to the client. This is, for the most part, done by calling “init” on the widget companion object.

  import net.liftweb.widgets.autocomplete.AutoComplete

  class Boot {
    def boot {
      ...
      AutoComplete.init
      ...
    }
  }

Example Snippet

In order to implement this snippet, simply do something along the lines of:

import scala.xml.NodeSeq
import net.liftweb.util.Helpers._
import net.liftweb.widgets.autocomplete.AutoComplete

class AutoCompleteSample {
  private val data = List(
    "Timothy","Derek","Ross","Tyler",
    "Indrajit","Harry","Greg","Debby")

  def sample(xhtml: NodeSeq): NodeSeq = 
    bind("f", xhtml, 
      "find_name" -> AutoComplete("", (current,limit) =>
        data.filter(_.toLowerCase.startsWith(current.toLowerCase)),
        value => println("Submitted: " + value))
    )
}

This snippet simply calls the built in AutoComplete factory which ultimately returns a NodeSeq that is bound to the namespace “f” and the element “find_name”.

Markup

All that remains is to call that in your template with something like:

  <lift:auto_complete_sample.sample form="post">
    <p>Type a name here: <f:find_name></f:find_name></p>
  </lift:auto_complete_sample.sample>

Enjoy

Calling AppleScript from Scala

2010-09-22T12:28:00+01:00

It occoured to me today that it would be a doddle to control parts of OSX from the Scala REPL. Check this out below, the script itself simply says “Hello World” out loud but it would be easy to substitute in something more meaningful.

scala> import javax.script._
import javax.script._

scala> val script = "say \"Hello World\"" 
script: java.lang.String = say "Hello World" 

scala> val mgr = new ScriptEngineManager
mgr: javax.script.ScriptEngineManager = 
  javax.script.ScriptEngineManager@8094cc7

scala> val engine = mgr.getEngineByName("AppleScript")
engine: javax.script.ScriptEngine = 
  apple.applescript.AppleScriptEngine@16cc8a48

scala> val context = engine.getContext                
context: javax.script.ScriptContext =
   javax.script.SimpleScriptContext@2278e185

scala> val bindings = context.getBindings(ScriptContext.ENGINE_SCOPE)
bindings: javax.script.Bindings = 
  javax.script.SimpleBindings@65712a80

scala> engine.eval(script, context)
res0: java.lang.Object = null

That’s all there is too it. Scalatastic!

JavaZone 2010: Adventures with Monads and Beer

2010-09-10T09:51:00+01:00

I have just arrived back from JavaZone and wow, what a great conference. Personally, I see this as being the best conference in Europe at the moment - the approach the organisers take to making zero profit and ploughing all the attendee’s fees right back into the conference is simply awe-inspiring! There really is no where quite like it on the EU conference circuit… for example, where else would you see a 3m high Java Duke that when you hug it, subsequently vends free beer… amazing.

Scala in the Nordics

Scala was very very well represented. This was my first time up in that part of the world and wow, what a community they have. Everyone is super smart and really interested in new technology; Scala in particular. On the second night of the conference the local “ScalaBin” held a meeting / pre-party and it was extremely well attended: much beer was consumed and some great conversations had.

Some Photos

Jonas Bon

Native2Ascii plugin for SBT

2010-08-18T09:32:00+01:00

As I have been doing quite a lot of localisation work recently, I thought it prudent to port the native2ascii tool over to SBT so that I didn’t constantly have to keep using it manually on the command line. If you are interested, you can grab the source code from here

If order to get started with the plugin add it to your Plugins.scala within your project:

  import sbt._
  class Plugins(info: ProjectInfo) extends PluginDefinition(info) {
    val n2a = "eu.getintheloop" % "sbt-native2ascii-plugin" % "0.1.0" 
  }

…and then mix the Native2Ascii trait into your project:

      class SampleProject(info: ProjectInfo) 
    extends DefaultWebProject(info) 
    with Native2AsciiPlugin {
    ...
  }
</code>

Ensure you have all your localization .txt files in src/main/i18n and by default they will be translated into your application resources folder in src/main/resources.

Then, from your SBT prompt just hit:

> native2ascii

All being well you should then see something like:

      [info] == native2ascii ==
  [info] Encoding '.txt' to '.properties' file(s) in src/main/resources
  [info] Translation complete.
  [info] == native2ascii ==
</code>

Scala: It’s Good for Your Health

2010-07-15T18:36:00+01:00

I was just stood in my kitchen washing the dishes and reflecting on the past four years coding Scala and being heavily involved with the community; seeing it grow and enjoying the relationships that have been forged through that time. I have learnt more during my time in this community than probably any other I have ever been involved with.

To date, I am yet to encounter someone in the Scala and Lift communities who isn’t above average intellectually and super friendly. Pretty much everyone, without exception, is very clever and this generates amazingly stimulating discussions, no matter what the topic. I have been exceedingly fortunate to work with the likes of David, Marius, Jonas and Viktor and have them provoke new ideas and create shared experiences with as we challenge our own thinking and continue to improve both Lift and Akka.

Finally, I have been lucky enough to forge some great friendships that have ultimately been very rewarding. Moreover, it makes me wonder what my life might be like had I never gotten involved with Lift? The likelihood of making those same friendships would be slim, and I doubt I would be a published author either!

All this makes me conclude that Scala has had a very, very positive impact on my life as a whole. So, if you are feeling under the weather? Perhaps you should take some Scala ;-)

Controlling Schemifier naming conventions with MapperRules

2010-06-27T16:27:00+01:00

I was recently asked how to stop Lift’s Mapper from flattening the naming of columns during its “schemification” process. Well, there is a little known configuration object in the net.liftweb.mapper package called MapperRules.

MapperRules contains a whole bunch of stuff that you can use to control the way that Mapper names stuff that it creates and how it will try and inspect your database. If you prefer (like me) to have columns and tables like first_name rather than firstname, then add the following to Boot.scala:

class Boot {
  def boot {
    ...
    MapperRules.tableName =  (_,name) => StringHelpers.snakify(name)
    MapperRules.columnName = (_,name) => StringHelpers.snakify(name)
    ...  
  } 
}

Thats it!

Lift In Action – The comprehensive guide to Lift

2010-06-23T19:41:00+01:00

So I have been pretty lax of late with keeping this blog up to date. There is a very good reason for that, and that is I am currently working on a book about Lift - If you haven’t heard, Lift In Action will be available on Manning Publications in Q1 of 2011 and is currently available as an eBook via the Manning website. Its generally eating a lot of my time right now so I haven’t had a chance to post about all the cool new stuff that is making its way into Lift every week - hopefully I will soon get time to write about the awesome new Record persistence system that we are all very excited about as we are currently integrating the extremely awesome Squeryl as a backend for dealing with RDBMS.

By all means, go and checkout the book and let me know what you think as it develops in the author online forums and I will get back to you ASAP. Enjoy!

Inline Markup Validation with Lift

2010-05-01T15:22:00+01:00

Continuing with my series of articles about Lift, here’s another hidden gem that not many people know about. Lift can automatically validate your markup so that any validation errors or warnings are high lighted to you as part of the development process. Pretty neat. Here’s what you need to do to enable it:

Configure Boot

  class Boot {
    def boot {
    ......   
      LiftRules.xhtmlValidator = Full(StrictXHTML1_0Validator)
    }
  }

Where of course you can replace StrictXHTML1_0Validator with any one of:

  TransitionalXHTML1_0Validator
  StrictXHTML1_0Validator

  // or subclass:
  GenericValidtor
  // to provide your own custom implementation

Find this and more great tips in my forthcoming book, Lift in Action

Enjoy!

Understanding Lift’s Box[T] monad

2010-04-16T09:26:00+01:00

So im at Scala Days 2010 right now, and during dinner last night their seemed to be some misconceptions about what Lift’s BoxT monad actually does and how it differs from OptionT. Now, im not computer scientist, so im not going to be discussing the algebraic validity of BoxT, not shall I be calling it a “tri state option”…

Why bother boxing values?

Have you ever thrown a NullPointerException? Frankly, I don’t know any developers that haven’t had this happen to them at some point because of something they hadn’t considered during development… in short, a NPE is something that was caused by an unexpected series of events causing your application to explode in a variety of ways. This is not good.

Consider a scenario where I want to do something with a value returned from a database; its not uncommon to see programs where one assumes the database query got the correct result, then do some operation on it. Well, news flash, its highly plausible that the database query might not go as you had expected and return some other result….

…Boxes to the rescue! As the name might suggest, a Box is something that you can put stuff in, take stuff out of, and also find empty. Much as you would do with a real life box. Lets illustrate with a simple example:

    import net.liftweb.common.{Box,Full,Empty,Failure,ParamFailure}

scala> val x: Box[String] = Empty
x: net.liftweb.common.Box[String] = Empty

scala> val y = Full("Thing")
y: net.liftweb.common.Full[java.lang.String] = Full(Thing)

</code>

We have two simple examples here that assign new boxes to vals, and as you can see, x is assigned as a Box[String], yet its actual value is Empty which is a sub type of Box. The second assignment to y has the same type signature, but this time it is “Full” with a value; in this instance “Thing”.

What about exceptions?

Lift has a helper method in net.liftweb.util.Helpers called “tryo” that is a specilized control structure so that if you can execute code in a block that returns a Box’ed value irrespective of what you did in the block and how its execution went. If your using Lift webkit then you already have access to these utility methods, however if you just want to stay light and are not using webkit then the definition of tryo looks like:

      def tryo[T](
      ignore: List[Class[_]], 
      onError: Box[Throwable => Unit])
      (f: => T): Box[T] = {
    try {
      Full(f)
    } catch {
      case c if ignore.exists(_.isAssignableFrom(c.getClass)) => 
        onError.foreach(_(c)); Empty
      case c if (ignore == null || ignore.isEmpty) => 
        onError.foreach(_(c)); Failure(c.getMessage, Full(c), Empty)
    }
  }
</code>

There are some other overloads for syntax sugar, but essentially it lets you do:

      tryo {
    // your code here
  }

</code>

So lets assume we had some remote API to invoke, and it could not connect, or blow up in some way you hadn’t planned, what would happy given a tryo block / wrap? Consider:

    scala> tryo("www".toInt) 
res4: net.liftweb.common.Box[Int] = Failure(
  For input string: "www",
  Full(java.lang.NumberFormatException: For input string: "www"),
  Empty)

</code>

As “www” isnt an Int, it unsurprisingly cannot be converted to one, so it blows up with java.lang.NumberFormatException. However, using tryo (boxed values) we are left with a special Box subtype called Failure. This lets us handle the error in a concise way rather than needing to check all the possible outcomes or worry about some awful try-catch block.

scala> tryo("www".toInt) openOr 1234                       
res7: Int = 1234

scala> tryo("www".toInt).map(_.toString).openOr("Invalid Strings")
res8: java.lang.String = Invalid Strings

scala> for(x <- tryo("www".toInt)) yield x
res11: net.liftweb.common.Box[Int] = Failure(
  For input string: "www",
  Full(java.lang.NumberFormatException: For input string: "www"),
  Empty)

So you can see there several ways to interact with Box, Full and Failure subtypes, providing defaults inline and mapping the results etc.

Handling empty values?

But what if we need more information or the value we are looking for isnt a failure, its just the value is Empty (or None for scala Option)? For example, lets assume we were looking for a request parameter to a REST API or similar… rather than throwing a HTTP 500 error with no reason, it would be nice to give the user a much more granular reason. Consider getting a request paramater using Lift’s S.param method.

scala> S.param("id") ?~ "You must supply an ID parameter" 
res17: net.liftweb.common.Failure = 
Failure(You must supply an ID parameter,Empty,Empty)

The ?~ method allows you to supply an error message and convert the Empty into a Failure. This is an incredibly helpful paradigm when used with for comprehensions.

Chaining boxes

Lets assume that we have a situation where by we could recive a value from several places, or we need to “fall through” to a specific result based on trying several values in a sequence… Box supports this by way of the “or” operand.

scala> val x = Empty
x: net.liftweb.common.Empty.type = Empty

scala> val y = tryo("qqq".toInt) 
y: net.liftweb.common.Box[Int] = 
  Failure(For input string: "qqq",
  Full(java.lang.NumberFormatException: For input string: "qqq"),
  Empty)

scala> x or y or Full("Default")
res18: net.liftweb.common.Box[Any] = Full(Default)

Box has a bunch of features I haven’t covered here, but I hope this helps you understand the rational of this specialised option-esq type.

Asynchronous REST Services with Lift

2010-04-02T09:55:00+01:00

Note: REST and many other topics discussed within my upcomming book Lift in Action, more information will shortly be published on manning.com

This one is hot off the press - Lift 2.0-SNAPSHOT now has the ability to do notification style service workflows - essentially comet for REST.

So how does this work? Well, following on from my other post on REST services with Lift we are still using a DispatchPF to setup the service, but we now use the statefull dispatcher LiftRules.dispatch rather than statelessDispatchTable and from inside our resource definition we call S.respondAsync. Here’s a complete example:

  LiftRules.dispatch.append {
    case Req("example" :: rest, _, _) => {
      S.respondAsync {
        Thread.sleep(20000) // some computation here
        Full(XmlResponse(<async>response</async>))
      }
    }
  }

The call to S.respondAsync executes the function you pass to it on another thread and if the container you are using supports suspend / resume idiom Lift will pass control to the container and no threads are blocked during its execution; just return a BoxLiftResponse like normal and Lift will handle notification to listening clients. Sweet.

Multi-format REST services with Lift

2010-03-29T17:00:00+01:00

Note: This subject is discussed in much greater detail within my upcomming book Lift in Action, more information will shortly be published on manning.com

The Goal

So what are we actually trying to achieve here? We’ll use the twitter.com API as an example as its a good illustration in this instance. Consider the following URLs:

http://api.twitter.com/users/timperrett.xml

and

http://api.twitter.com/users/timperrett.json

Its the exact same content, but just presented in different formats. This is very nice, and also very helpful as often you might be in a situation where you need to syndicate a single API from multiple devices or platforms where toolkits may be limited. For example, working with JSON is a lot easier from Java script than it is XML (the latter is of course possible its just more verbose). In the examples above, Twitter are using the format component of the URL following the ”.” to determine the content type the user wants - this is of course one route, and some services may want to add the additional stipulation of a specified accepts and content-type headers in the requests so the server is sure its serving the right content.

The interesting problem here is that if you have the same content, you really only want one point of access for the logic. You simply dont want to be maintaining two set of logic - only the actual code that handles differences in presentation should alter between calls right?

Implementation Pattern

Typically, when I want to create web services in my Lift application I break out my dispatchers into separate objects related to their concerns. This way it keeps things nice and tidy. So, in our example, i’ll be wiring up the following services:

  LiftRules.statelessDispatchTable.append {
    case Req("apps" :: Nil, "json", GetRequest) => 
      ApplicationServices.json.list
    case Req("apps" :: Nil, "xml", GetRequest) => 
      ApplicationServices.xml.list
  }

For those not familiar with Lift’s dispatching mechanism this will expose the following URLs:

/apps.json
/apps.xml

All pretty simple so far. As you can see from the code listing above, I have broken my application services out into a singleton object called ApplicationServices which has fields for each content type im going to be presenting. Without further ado lets take a look at the definition of the apps listing service:

object ApplicationServices extends Loggable {
  .....
  protected def _list(mediaAction: List[String] => LiftResponse) =
    () => tryo(mediaAction(Applications.list.map(_.name)))
  .....
}

Ok so this is pretty cool. Im using function passing to let each subsequent definition define how it builds the LiftResponse sub-type; in this instance that’ll be JsonResponse for JSON and XmlResponse for XML.

JSON Listing

def list = _list((apps) => 
  JsonResponse(compact(JsonAST.render(("applications" -> apps)))))

XML Listing

def list = _list((apps) => XmlResponse({
  <applications>
    {apps.map(a => <application name={a} />)}
  </applications>
}))

So you can see that each presentation version is only dealing with how it needs to create the response… its not re-computing the list of applications or such.

Syntax Sugar

Within the overall definition I define the formats as inner-objects and assign them to public fields in order to achieve the nice ApplicationServices.xml.list notation:

object ApplicationServices extends Loggable {
  val json = Json
  val xml = Xml

  protected def _list(mediaAction: List[String] => LiftResponse) =
    () => tryo(mediaAction(Applications.list.map(_.name)))

  object Json {
    // definitions here
  }

  object Xml {
    // definitions here
  }
}

Round Up

Thats pretty much all there is too it. You can leverage Scala’s extremely powerful features to create your web services and keep code LOC to a minimum. Over and out.

Gracefully handling 404s with Lift

2010-02-21T14:07:00+00:00

In my continuing effort to improve the wider knowledge of Lift, here’s another great nugget that shows you how to provide a 404 handler powered from a Lift template

class Boot {
  def boot {
    // ...other stuff...

    LiftRules.uriNotFound.prepend(NamedPF("404handler"){
      case (req,failure) => 
       NotFoundAsTemplate(ParsePath(List("404"),"html",false,false))
    })

  }
}

And I have a 404.html file in the ./webapp/404.html with the contents:

<lift:surround with="default" at="content">
  <h2>Ooops!</h2>
  <p>Wow, we totally couldn't find your page. Sorry about that.</p>
</lift:surround>

Then whenever a user browses to a URL that is not supported by your application, you can give them a friendly (and nicely designed) webpage rather than some technical message about 404s.

Note: The only cavet is not to use the <lift:menu /> snippet within your 404 template because Lift is unable to calculate the SiteMap when its a URL that is not accounted for within that scheme.

UPDATE The above note is now redundant because of commit by Marius on issue 376

Using Jetty 7 continuations to power Lift comet support

2010-02-18T17:07:00+00:00

Recently, Marius added an abstraction layer within Lift so that we could support different comet implementations over different servlet containers. Out of the box, we provide two both Jetty 6 and Jetty 7 APIs. This API is however not very well documented so I thought I would just write a short post on how to use it. In your Boot.scala file you need to put something like:

import net.liftweb.http._
import provider.servlet.containers.Jetty7AsyncProvider

class Boot {
  def boot {
    // ...other stuff here...

    LiftRules.servletAsyncProvider = (req) => 
        new Jetty7AsyncProvider(req)

  }
}

Thats pretty much all you need to do in order to swap comet implementations! If you want to use something else, like Resin etc, all you need to is write the correct provider and away you go.

Consistent hashing and encryption between Lift and .NET

2009-11-11T14:50:00+00:00

Its quite often that I need to pass tokens between Scala (Java also) and .NET - this instantly provides a whole raft of issues because of how the algorithms are built under the hood in the respective platforms. This short post details how you can generate consistent base64 hashes over both platforms (this can be helpful for security purposes etc)

The Scala way:

  val input = "sample" 
  val encoding = "UTF-8" 

  base64Encode(
    hexDigest(
      input.getBytes(encoding)
    ).getBytes(encoding)
  )

The .NET way

  using System.Security.Cryptography;

  string input = "sample";

  SHA1CryptoServiceProvider shaHasher = 
       new SHA1CryptoServiceProvider();
  byte[] data = shaHasher.ComputeHash(
    Encoding.UTF8.GetBytes(input)
  );

  StringBuilder sBuilder = new StringBuilder();
  for (int i = 0; i < data.Length; i++){
    sBuilder.Append(data[i].ToString("x2"));
  }

  string hash = sBuilder.ToString();

  Convert.ToBase64String(
    Encoding.UTF8.GetBytes(hash)
  );

UPDATE After I posted this code originally, it was pointed out that it would be good / helpful to show encryption as well as hashing; so here goes:

The Scala way:

  val key: Array[Byte] = "qwehfjgtufkgurifghfkdjfg".getBytes("UTF-8")
  tripleDESEncrypt("sample", key)

The .NET way:

  using System.Security.Cryptography;

  string key = "qwehfjgtufkgurifghfkdjfg";
  string toEnc = "sample";

  TripleDESCryptoServiceProvider DES = 
       new TripleDESCryptoServiceProvider();

  byte[] bytKey = Encoding.UTF8.GetBytes(key);

  DES.Mode = CipherMode.ECB;
  DES.Key = bytKey;
  DES.Padding = PaddingMode.PKCS7;

  ICryptoTransform DESEncrypt = DES.CreateEncryptor();

  byte[] buffer = Encoding.UTF8.GetBytes(toEnc);

  string base64string = Convert.ToBase64String(
    DESEncrypt.TransformFinalBlock(buffer, 0, buffer.Length)
  );

  Response.Write(base64string);

Maybe this will help someone avoid the pain I have endured with this! lol.

Conversational Marketing with Scala and Lift

2009-10-16T17:10:00+01:00

As most of my readers are technical (yes, thats you there) I wont labour the marketing peice, but if your interested, David wrote a really nice article explaining his concept about what happens when social media meets cross-media; find it over here

A cunning scheme indeed

So, from the off, this was going to be an interesting campaign to put together because there were numerous issues, including:

* Lots of concurrent (and asynchronous) read / write ops to Twitter API

* Backend communication to XMPie cluster

* Push production status back down to the browser

* Downloading of avatar images automatically from twitter and insert into XMPie asset source

* Doing any of this in the given time frame!?

This project was going to have a sizeable set of works attached no doubt to manage the interplay between XMPie systems and the Twitter API.

Technology Sandwich

To implement this campaign, I ended up using a whole bunch of different technologies… The implementing stack eventually ended up utilising the following:

* Lift

* Dispatch

* Scala Actors

* XMPie uProduce

* XMPie ICP

* Microsoft SQL Server

In earlier drafts of this article I tried to explain the interplay between components, but simply could not find a good way to articulate the meaning. To that end, I hope diagram gets the structure across sufficiently!:

As you can see - quite a number of software components here! I’ve been working with Scala and XMPie uProduce for quite a while now and have built up some very nice abstractions that completely hide calling of services under the hood.

This entire campaign was implemented based on top of ICP, using my Scala abstraction along with the awesome power of Scala Actors, scheduling and some generally very cool libraries like Dispatch

Questions

A few people asked me to write this post, but as im not sure what specifically you might have been interested in, if you want to know more about how its put together, what does what etc then I would gladly take questions! Just leave a comment at the bottom and i’ll get back to you…

Replacing Checkpoint SecureClient with IPSecuritas on Snow Leopard

2009-08-31T21:57:00+01:00

So its that time again - another awesome Apple update to the best operating system in the world; for most, its a joyous time, one of new beginnings and wonderment…. for some, alas this is just an ideal as all there proprietary software comes crumbling down under a new kernel.

Friday was one of these momentous days, and yes, my world came crumbling down. For those who don’t read my blog often I work remotely 85% of my working weeks as the company I work for are in another country so having a secure and speedy VPN is critical to actually getting paid and doing some work. After upgrading OSX 10.5 to Snow Leopard my Checkpoint SecureClient completely stopped working - this appears to be the plight of many users out in the interweb so I thought id write up this guide how to use IPSecuritas (as it rocks) which is infinitely better than the default checkpoint client (which sucks major ass!).

Why should I care about IPSecuritas?

This is simple - basically (as above) it rocks and has the following great features:

Its speedy.
Automatic connection recovery
Password persistance
Can talk to a bunch of different Firewall types… no more vendor tie-ins
Great OSX integration
Automatic connection upon login (instantly connected to VPN!)
Oh, did I mention its fast?

Removing Checkpoint SecureClient

OK, so now we’ve established that SecureClient is evil, lets remove it. Helpfully, checkpoint took the time to provde a shell script in the install directory to do just this. Open a Terminal window (Applications > Utilities > Terminal) and type the following:

# i'm not 100% sure on the names, as im writing this from memory, 
# but just have a poke around and you'll find what I mean.
timperrett$ cd /opt/C (press tab for auto-complete then return key)
timperrett$ open Uninstall.command

Type “yes” when prompted - the script will then go about removing all the various components. If your thinking of skipping this step, your free to, but remember than this installation is completely broken and wont ever run under Snow Leopard as it appears to be tied to the 9.x Kernel present in 10.5… so you might as well clean up and keep tidy.

Setting up IPSecuritas

If you havent already, download IPSecuritas from here - open the DMG and drag the application to your Applications directory. Once there, double click the application to load it for the first time - you’ll need to enter your Administrator password then the application will install a daemon onto your system and configure itself. Once completed, take my advice and reboot your system - upon reboot you should see a new menu item that looks like a broken wire (below, dont worry about “XMPie”, thats just what I decided to call my profile)…

Choose the “Open IPSecuritas” menu item - and up should spring the main GUI. You now need to configure a connection - this is what you will use to connect to your VPN endpoint (clue’s in the name!). Making this connection is however a rather technical process for most users so im going to post screen shots of my configuration at every stage so that you can make something similar (yours may not be identical - it really depends on the setup implemented on the firewall; however, what I detail uses common place defaults).

So the two boxes in red are the important ones. For Remote IPSec Device you need to fill in the domain name or IP address of your firewall / vpn endpoint; this is organisation specific so i’ve removed mine. Secondly, Network address - this is the base IP range to which you want to connect to; again, organisation specific and yours will likely be different - if you don’t know, check with an IT administrator.

These are standard security options needed to work with the checkpoint vpn - because I work long hours connected to the VPN, i’ve set it to timeout after 10 hours (essentially it never cuts me off and im in charge… sweet).

Again, this is standard checkpoint stuff so just copy the configurations as is - you don’t need to know whats what.

So this is an interested panel - the two boxes ive marked in red need to be filled with your username for the VPN connection; you should have this already with that which you were using with the Checkpoint SecureClient system. In this example (and the checkpoint default) its using Hybrid RSA, but a lot of organisations use XAuth RSA etc that involve certificates etc… if you need this, just use the certificate manager and configure accordingly. I’ve also set it to remember my password so that I dont have to keep entering it - depending on your outlook, this is a good/bad thing. Personally, I think its a great timesaver!

Depending on your setup, you may want to configure specilized DNS servers - you might want this if you have servers that you wish you access with a UNC style such as:

<code>\\somefileshare</code>

Without specifying those DNS, your VPN will try to use external DNS and you just wont get what you want (or expect). I’ve removed mine for security reasons, but it should be fairly simple to figure out what you need to enter.

So this is the killer panel that confuses most people - you have some fairly finite control over the connection parameters - if your using Checkpoint VPN-1, just do as I have (unless your using another authorisation mechanism) and you should succeed!

Once you have all that, your good to go - just close the window, and click “Start” on the main IPSeceritas window then provided all went well you should get a green light next to the connection name - in my instance, i called the connection “office”. If you are having issues, seek help from your IT administrator as it might be a configuration issue - with VPN’s your client much EXACTLY match what the endpoint is configured to… any miss-match at all will result in failure; however if you wish to debug the issue yourself, bring up the connection log from the top bar menu and you’ll be able to see exactly what is going on under the hood (if you need a boat load of wire information, set the logging level in preferences to DEBUG and then restart/reboot IPSecuritas)

Enjoy, and good luck.

How To: Extensive Localization with the Liftweb Framework

2009-07-26T22:06:00+01:00

One of the best things about Lift is its amazingly flexible template and resource localization system. This article discusses the mechanisms that you can use to localize your application.

Overview

Out of the box, Lift gives you the following options - items 1 and 2 require zero boiler plate, whilst the 3rd option gives you the flexibility to extend the localization however you need.

Text localization from property bundles
Full template localization
Custom resource bundle provider hook

We will now take a look at each localization option in turn, in relative detail stopping to investigate how they work.

Assumed environment

For the length of this article we’ll assume that we have the need to localize into English (our default language), French, German and Hebrew. These languages have the following locale codes:

		Language 
		Code

English

en_GB

French

fr_FR

German

de_DE

Hebrew

he_IL

These are standard ISO codes used by the Java localization system, irrespective of Lift et al. As a bit of background for those of you not familiar with locale codes and there purpose, you can see that the first part of the locale code denotes the spoken language - for example, en is English - and the second part after the underscore is the country code. This is amazingly helpful for languages like English and Arabic which are spoken in many countries as it gives you a specific language and then culture on which you can account for language nuances etc. Salutations are a classic one - UK English might have a salutation of “Good afternoon” whilst Australian English might have “Good ay’“.

In order to tell Lift that this application will be localized into several languages, we have to set a customized localeCalculator. In our application it will look like this:

  def localeCalculator(request : Box[HttpServletRequest]): Locale = 
  request.flatMap(r => {
  def workOutLocale: Box[java.util.Locale] = 
  S.findCookie(localeCookieName) match {
    case Full(cookie) => cookie.getValue()
    case _ => Full(LiftRules.defaultLocaleCalculator(request))
  }
  tryo(r.getParameter("locale")) match {
    case Full(null) => workOutLocale
    case Empty => workOutLocale
    case Failure(_,_,_) => workOutLocale
    case Full(selectedLocale) => {
      setLocale(selectedLocale)
      selectedLocale
    }
  }
}).openOr(java.util.Locale.getDefault())

Add this function someplace in your application - here we’ll assume its just in the Boot class, then set it in LiftRules like so:

LiftRules.localeCalculator = localeCalculator _

Text localization from property bundles

So, given our working languages we have several issues at hand and several strategies we could choose. Lets start with the most basic form of localization that will be familiar with most users of the java platform… properties files loaded as a ResourceBundle.

So lets assume that we have our translations already completed, we just need to put them into key-value pairs in properties files located in:

`${project.basedir}/src/main/resources/mybundlename_*locale*`

So for us, that looks like:

mybundlename_en_GB.properties
mybundlename_fr_FR.properties
mybundlename_de_DE.properties
mybundlename_he_IL.properties

This is pretty standard stuff that is well documented in the javadocs from sun. So, you need to know how to specify value keys within lift. There are two use cases, one is in your code, and the other is in your HTML templates.

In code:

S.?("mykey")

In template:

<lift:loc locid="mykey">Default Text</lift:loc>

This is all well and good, but there are use cases that simple key/value replacement doesnt take care of - with our use case languages we have a great example here:

Hebrew is a language that is written right-to-left (RTL) so generally the content / css / markup will be quite different
Within the languages that are left-to-right (LTR), French and English have a comparable number of character tokens, but german is typically more verbose and content takes up more room.

So, to deal with such cases Lift brings you comprehensive template localization which we’ll now discuss.

Template file localization

Given this conundrum about language direction and content length lift can assert different template names. For example, lets assume that in your webapp directory you have a file called index.html - based on the LiftRules.localeCalculator Locale that is returned, it can choose the right template. With the problems we face here, we might have:

index_en_GB.html
index_en_AU.html
index_de.html
index_he.html

This would then yield different content templates for German and Hebrew, and gives us two distinctly different english templates for UK English and Australian English… for arguments sake the imagery in the two templates could be different because Australian culture is somewhat more casual than compared to England.

This exact same scheme also applies for CSS resources if you do not need the possibly side effect of code duplication in this system.

Custom resource bundle provider hook

One of the driving mantras of Lift is that everything, and we mean everything has sensible defaults, but you can hook right into the core lift lifecycle and add your own stuff. Localization is no different and it is of course a common idiom to need to load localization content from a database backed cache. I wont delve into exactly how you handle the caching or similar (that my friends is up to you) but this is how you pass the special resource bundles into Lift’s cycle:

LiftRules.resourceBundleFactories.prepend { 
  case (basename, locale) if localeAvalible_?(locale) => 
      CacheResourceBundle(locale)
  case _ => CacheResourceBundle(new Locale("en","GB"))
}

In the above example, localeAvalible_? checks if this locale is available (from a value in my application) and of course, CacheResourceBundle is a subclass of ResourceBundle and has the following signature:

case class CacheResourceBundle(loc: Locale) extends ResourceBundle

Conclusion

Thats pretty much it folks - by way of a mix of these schemes its possible to build up a very rich localization methodology which works for pretty much all locale needs. In this example we have discussed language lengths, RTL languages and given you all the code you need to get going with localization in Lift - go forth and localize!

Lift AMQP with RabbitMQ and Scala – Tutorial and Screencast

2009-05-22T11:39:00+01:00

Admitidally, every time i’ve ever looked at the Lift codebase in TextMate, i’ve always wondered what the hell the “lift-amqp” module was for. Well, curiosity may have killed the cat but it was not enough of deterrent to me finding out and building a sample AMQP / RabbitMQ application!

AMQP you say?

Yup, thats right. AMQP - Advanced Message Queuing Protocol to the lay’ man not familiar with the acronym. Essentially AMQP is an open standard for enterprise messaging. There are several implementations of the standard, namely, Open AMQ and RabbitMQ - the latter is newer and generally considered to be the defacto implementation. Rabbit MQ is written in ERLang and has clients for Java, .NET, Ruby etc etc.

So where does Lift fit in?

Strictly speaking, it doesn’t. It just so happens that Steve J was working on the Lift team some time ago and wrote a nifty Scala wrapper around the standard (and highly mutable) Java client implementation of Rabbit MQ. To all intense purpose, “lift-amqp” can be used in any Scala application and has no other dependencies within the Lift framework.

Hand-wavy overview

Lets get down to business… I spent quite a lot of time just looking at the source code and scala-docs for lift-amqp and for the longest while couldn’t figure out what went where. Being a fairly visual person, I’ve drawn up a diagram that explains the implementation and how you (yes, you reading this) would implement your application:

A working example?

So rather than try to post lots of pictures - I thought just putting up a screen-cast explaining the components would make more sense :-)

The client listener

So from the diagram above that explains the code layout - you can see that one of the first things you need to do is subclass AMQPDispatcher. The below is my example code that connects to the RabbitMQ broker and firstly declares a queue with the appropriate parameters. If you wondering why when the 2nd client connected in my example things didn’t explode and complain about more than one exchange existing with the same name, thats because RabbitMQ is clever enough to know that if the queue / exchange already exists then it just Noop’s the requests. Its a nice feature.

class DemonstrationSerializedAMQPDispatcherTueueName: String, factory: ConnectionFactory, host: String, port: Int) extends AMQPDispatcherTactory, host, port) { override def configure(channel: Channel) { val ticket = channel.accessRequest(“/data”) channel.exchangeDeclare(ticket, “mult”, “fanout”) channel.queueDeclare(ticket, queueName) channel.queueBind(ticket, queueName, “mult”, “example.”) channel.basicConsume(ticket, queueName, false, new SerializedConsumer(channel, this)) } }

So, your probably thinking that this was not the class that I created an instance of during the screencast - and you’d be right. Essentially we still need a wrapper class to mask our dispatcher; the listener proper.

The purpose of the listener is to establish a connection to the broker, pass the right credentials etc. Within the listener we create an instance of the dispatcher and start its actor. We also then just have an inner class that we use for example purposes (another actor) that prints out the messages you saw in the terminal window. In reality, this would likely be passing messages to some other, external actor which handled the business process. For a full code listing see the end of this article.

The postman…

So now that you know how the example listens for messages, lets see the posting code…

  val params = new ConnectionParameters
  params.setUsername("guest")
  params.setPassword("guest")
  params.setVirtualHost("/")
  params.setRequestedHeartbeat(0)
  val factory = new ConnectionFactory(params)
  // Create a new instance of the string sender.
  // This sender will send messages to the "mult" exchange with a 
  // routing key of "routeroute" 
  val amqp = new StringAMQPSender(
    factory, "macbookpro", 5672, "mult", "example.demo" 
  )
  amqp.start

  /**
   * Salute the rabbit!
   */
  def salute = amqp ! AMQPMessage("hey there!")
}

Little explanation needed here - essentially the string sender is just an actor that knows how to respond to AMQPMessage…. it couldn’t be simpler! I hope this has proved helpful and a good overview of the lift-amqp module.

If you would like to download the source code, you can get it from here

XMPie Marketing Console for iPhone

2009-05-20T09:54:00+01:00

Its been a long time since I posted my original application for iPhone - iDashboard - to control XMPie uProduce on the move. I’ve been busy with a bunch of other cool stuff and in the run up to the annual XMPie Users Group meeting in Las Vegas decided to make Marketing Console for iPhone!

The application syndicates reporting and analysis information about your campaigns and lets you see the latest up-to-date charting / figures wherever you are. Currently, the application is using an internal R&D; build of Marketing Console; so right now you’ll have to wait before you can monitor your own campaigns on the move!

If you have feedback, just leave a comment at the bottom of the article. Enjoy…

Running Rabbit MQ on Mac OSX – Solving {badrpc,nodedown}

2009-05-17T23:06:00+01:00

Im currently exploring Rabbit MQ and had a few issues getting up and running reliably on Mac OSX. The problem wasted so much of my own free time that I thought it would be a good idea to post about it and perhaps it might help others in the future.

The Problem

The broken boots normally as the rabbitmq user defined in the system - however, when trying to connect to it using rabbitmqctl you get the following error (repeatedly):

macbookpro:~ timperrett$ sudo rabbitmqctl status
Status of node rabbit@macbookpro ...
{badrpc,nodedown}
...done.

The Solution

After many hours dabbling, and checking, checking again, rechecking my user and permissions setup, I found that it was actually to do with the way in which ERlang networks. Essentially, I was running the broker on:

rabbit@macbookpro

However, whilst I could ping the host “macbookpro” from terminal, it appears that Rabbit MQ needed it defined in the /etc/hosts file in order to work correctly.

Both strange and annoying, perhaps this will save someone some time!

URL Rewriting with the Lift Framework

2009-05-03T20:39:00+01:00

With my on-going effort to write more documentation and articles for Lift I’ve decided to write a walk through of Lifts dispatching and rewriting mechanisms.

Before we start this discussion, its important that you know (and understand) the importance of partial functions in scala. If your not familiar, check out this article - it should fill you in on all the particulars.

Application Boot

If your not familiar with Lift or are new, you should understand that anything of consequence that changes the application environment must in someway hook into the boot-up cycle - the default looks something like this:

class Boot {
  def boot {
    // where lift will look for snippets
    LiftRules.addToPackages("eu.getintheloop.tutorial")
  }
}

Ok, so not a great deal going on there… this is however a bare-bones lift boot class. In our case, we want to add a rewrite so that the following mappings take place:

/product/some-product-link

/** maps onto **/

./webapp/product-display.html

Its important to note that rewriting is just that, its not used for redirects or any other such activity - its 100% for URI translation and interpretation.

Adding a Rewrite

Pretty much all of application configuration within Lift is done through the LiftRules object - its the central place for configuration PFs and operation vars. So, how do we use it? Well, first add the following to the top of your Boot.scala file.

import _root_.net.liftweb.http.LiftRules

Next, add the following code below the snippet package definition:

LiftRules.rewrite.prepend(NamedPF("ProductExampleRewrite") {
  case RewriteRequest(
      ParsePath("product" :: product :: Nil, _, _,_), _, _) => 
    RewriteResponse(
      "product-display" :: Nil, Map("product" -> product)
  )
})

Rewrite matching in detail

So lets step through this part by part… we already know about the LiftRules object and what its for, and one of its var properties is “rewrite” - a RuleSeq - and has the notion of both prepending values, and appending them. In this case, we prepend this rewrite rule, meaning that it will execute before any other rewrite rules previously added to the rewrite var.

List[String]

/**
 * options for RewriteRequest
 */
case class RewriteRequest(
  val path : ParsePath, 
  val requestType : RequestType, 
  val httpRequest : HttpServletRequest
) 

/**
 * options for ParsePath
 */
case class ParsePath(
  val partPath : List[String], 
  val suffix : String, 
  val absolute : Boolean, 
  val endSlash : Boolean
) 

/**
 * options for requestType object
 */

GetRequest
PostRequest
PutRequest
DeleteRequest

The first argument in the RewriteRequest is a ParsePath, this is one of the primary URI matching mechanisms and enables you to define detailed paramaters on which request to match and which to ignore. Lets take a look at some various RewriteRequest examples:

/**
 * example 1.
 * matches: GET /some/demo/path
 */

RewriteRequest(
  ParsePath("some" :: "demo" :: "path" :: Nil, "", true, false), 
  GetRequest, _
)

/**
 * example 2.
 * matches: PUT /some/image.png
 */

RewriteRequest(
  ParsePath("some" :: "image" :: Nil, "png", true, false), 
  PutRequest, _
)

/**
 * example 3.
 * matches: * /some/demo/
 */

RewriteRequest(
  ParsePath("some" :: "demo" :: "index", "", true, true), _, _
)

/**
 * example 4.
 * matches: GET /product/[item]/details
 */

RewriteRequest(
  ParsePath("product" :: item :: "details" :: Nil, "", true, true), 
    GetRequest, _
)

So far we’ve seen how to configure the incoming request - to complete the picture, lets now take our request handling and couple that up with matching the response and mapping parameters so they are then available in the rest of our code via S.param.

The RewriteRequest object has several overload apply methods to keep the verbosity of your code to a minimum - namely, these overloads are:

def apply(path : ParsePath, params : Map[String, String])
def apply(path : List[String])
def apply(path : List[String], suffix : String)
def apply(path : List[String], params : Map[String, String])

We can see that this gives us a bunch of flexibility depending on our needs - straight rewrite, rewrite with params etc etc. Lets take a look at some examples:

/**
 * rewrite to a html file in webapp/show.html with no params
 */

RewriteResponse("show" :: Nil)

/**
 * rewrite to a html file in webapp/show.html with 
 * a paramater called "product" from
 * a parameter placeholder called product
 */

RewriteResponse("show" :: Nil, Map("product" -> product))

/**
 * rewrite to a html file in webapp/example.pdf with 
 * no params, but a pdf suffix
 */

RewriteResponse("example" :: Nil, "pdf")

We have now looked at both requests and response - so moving back to our original example I hope you can see how it now works. One thing we have no explored is how to access the various parameters you might configure in your snippets / other application code. The answer my friends, is simple:

S.param("product").openOr("fail over product")

By now I presume you are a rewriting guru!… that might be a bit overboard, but I hope you found this guide informative and useful.

A detailed look at Lift’s view code binding system

2009-04-13T16:31:00+01:00

Overview

This stuff crops up time after time again, and people are always asking about how bind() works and what they should actually be doing with it within there lift apps. I’ve seen some fairly ugly abuse of scala.xml._ within Lift snippets… however, not only does this tightly couple your HTML design with the snippet code behind - and thus turn into a nightmare for your designers - your missing out on some of lifts best features!

All the code snippets below presume the following imports:

import _root_.scala.xml.{NodeSeq,Text,Node,Elem}
import _root_.net.liftweb.util.{Box,Full,Empty,Helpers,Log}
import _root_.net.liftweb.util.Helpers._
import _root_.net.liftweb.http.{S,SHtml}

Also, all the methods are a part of the snippet class “Demo”.

Basic Binding

So, if we want to deal with a view control on the server side, we need to bind it to a namespace in the markup. In our first example, we’ll just create a text input and populate its value.

Scala Snippet

var tempValue: String = "" 

def exampleOne(xhtml: NodeSeq): NodeSeq = bind("example", xhtml,
  "item_one" -> SHtml.text(
   tempValue, // the "read" value
   tempValue = _), // the "write" function
  "submit" -> SHtml.submit("Submit", () => Log.info(tempValue))
)

So, lets just review quickly. We’ve made a string variable that were going to assign the value of our text field when the form is submitted. Furthermore, if you watch the console window you see that upon submission the value you entered is output to the console.

XHTML Code

<lift:demo.example_one form="post">
  <p>Sample input: <example:item_one example:id="demo_id"></example:item_one></p>
  <p><example:submit></example:submit></p>
</lift:demo.example_one>

You can see how the prefix we specified in the bind function (“example”) is now used in the markup body of the snippet. Also, note how the actual snippet call uses snake_case (“example_one”) when our method is called “exampleOne” - Lift is clever enough to know what method you are trying to use… this ensures your designers dont moan about camel cased markup!

Binding++

The previous example was illustrative, but pretty usless in real applications. Lets take a look at something that is a bit more feature full - namely, a small form that updates a database table using the Mapper persistence framework.

Before we begin lets assume that we have a model class called “User”, and it has the fields:

first_name - MappedString
last_name - MappedString
email - MappedEmail
display_email - MappedBoolean

In this fictional form, users just enter their details and a new database user row is added.

Scala Snippet

 def exampleTwo(xhtml: NodeSeq): NodeSeq = {
  var user: User = new User
  def submitHandler() = {
    if(user.saveMe){
      S.redirectTo("thank-you")
    }
  }
  bind("example",xhtml,
    "first_name" -> SHtml.text(user.first_name, user.first_name(_)),
    "last_name" -> SHtml.text(user.last_name, user.last_name(_))
    "users_email" -> SHtml.text(user.email, user.email(_)),
    "display_email" -> SHtml.checkbox(false, user.display_email(_)),
    "submit" -> SHtml.submit("Submit", submitHandler _)
  )
}

Here we have something slightly more complex - the principals are exactly the same but with more fields and a nested method to handle the submit button. In this case, the submit button just saves a new record to the database and then if successful redirects the user to a thank-you page.

XHTML Code

<lift:demo.example_two form="post">
  <p>
    <label for="first_name">
      User name <example:first_name example:id="first_name"></example:first_name>
    </label>
  </p>
  <p>
    <label for="last_name">
      Last name <example:last_name example:id="last_name"></example:last_name>
    </label>
  </p>
  <p>
    <label for="users_email">
      Email <example:users_email example:id="users_email"></example:users_email>
    </label>
  </p>
  <p>
    Display your email? <example:display_email></example:display_email>
  </p>
  <p><example:submit></example:submit></p>
</lift:demo.example_two>

You can see how we have neatly wrapped our inputs in labels (a largely view centric concern) without any changes or specific ties to the server side logic that does the database insert. This example also shows another common mistake that new-commers make… there is no crazy magic binding going on between the names of the bind placeholders and what you put in them… its purely a convention thing / coincidence that they are called the same. You are free to call them whatever you feel appropriate.

Advanced Binding

Its when you need advanced binding that lift really comes into play - lets take the oh-so common idiom of one article having many “tags”; as you are reading this on my blog i’ll assume you are familiar with the notion.

So, assuming we have a Article model object with the appropriate fields, and we have a Tag model object that has a MappedForeignKey field relating back to the Article model object. We’ll assume we have a utility method in the article object that grabs all the tags for that article - this will allow us to do something like:

article.tags.flatMap(tag => ....)

Lets get down to business:

Scala Snippet

def exampleThree(xhtml: NodeSeq): NodeSeq = { 
  // load article based on a fictional url parameter
  val article = Article.find(By(Article.link, S.param("permalink")))
  article.flatMap(a => bind("a", xhtml,
    "body" -> Text(a.body), // outputs the article body
    "tags" -> a.tags.flatMap(t => 
      bind("t", chooseTemplate("tag","list", xhtml),
        "name" -> Text(t.name)
      )
    )
  ))
}

After loading a single article based on the URL parameter passed to the snippet by some routing or query string parameter (doing this is out of scope for this article - check on the lift wiki) - we can the bind the contents of that model object and also iterate over the list of tags associated with that article.

The new stuff here is both the nested bind, and the chooseTemplate(…) method - this actually lets us tell the binding that for this nested bind, were going to be using a set template within the passed / input XHTML. Very cool - you’ll see this working in the view code below.

XHTML Code

<lift:demo.example_three>
  <p><a:body></a:body></p>
  <p>Tagged with: <a:tags> 
  <tag:list>
    | <t:name></t:name> |
  </tag:list>
</a:tags></p>
</lift:demo.example_three>

You can see how we output the normal dynamic information just as we would any other bind item, wrapping the body in a scala.xml.Text node. We then posistion the tags placeholder of the outer bind in the correct place. Following that we then define the tag template and the content inside the tag template using the “t” prefix we specified in the nested bind function within the exampleThree method def. This should be fairly self evident that the chooseTemplate method is selecting the tag:list template and using that for each itteration of the nested bind - this is way cool and extremely usfull in most applications.

This concludes the tutorial - I hope this has been useful for someone :-)

Understanding Lift’s StreamingResponse

2009-03-19T17:15:00+00:00

I was just looking at the Lift’s StreamingResponse and was a bit bemused by the structural type being used for the first parameter. After some fiddling around I realized that:

data: {def read(buf: Array[Byte]): Int}

This will actually make the method more flexible and not tied to a particular hierarchy of classes (and super classes). So, if your looking at Lift’s StreamingResponse and thinking “what the hell”, all you need to remember is that you can pass any thing into the first parameter as long as it implements the read method with the above signature. This, IMO, is majority cool. The default thing that implements this signature is java.io.InputStream, but you could of course pass anything - even your own custom classes!

A sample implementation might look like:

var data: Array[Byte] = // get your data here
val headers = 
    ("Content-type" -> "application/pdf") :: 
    ("Content-length" -> data.length.toString) :: 
    ("Content-disposition" -> "attachment; filname=download.pdf") 
    :: Nil
Full(StreamingResponse(
  new java.io.ByteArrayInputStream(data),
  () => {},
  data.length, 
  headers, Nil, 200)
)

</code>

So, to recap, structural types are awesome!

Are iPhone applications the latest marketing frontier?

2009-03-01T20:43:00+00:00

Recently I’ve noticed more and more iPhone applications being released in the run up to both film releases and real-world product launches for large purchase items such as cars. So, what do the marketing chaps want to achieve with these adventures into the world of mobile computing?

Can we summarize that the marketeers see the iPhone as a lot more than just a mobile device? I think so, otherwise they simply wouldnt bother right? The thing I simply cannot get my head around, is how, or indeed if, the marketeers look to get any ROI on what must be a sizable amount of time and investment to create these iPhone apps which are usually quite sophisticated games.

Rhino Ball

For example, lets take Rhino Ball - a pretty sophisticated 3D game in which you control a hamster in a rolling ball. Pretty amusing, but I cant say it really compelled me to go and see the film. So why bother? In times of recession and tight budgets, I simply cant fathom why they are spending money on this rather niche medium to touch a small group of people with a weak message.

Perhaps this is just a prime example of marketing depts. blowing budgets for the simple reason “its cool”.

New Internationalization Extendability in Lift

2009-02-28T00:26:00+00:00

I just committed new functionality into lift master for handling the abstraction of i18n ResourceBundles’ that are powered by custom sources - for example, database driven localization, or hooking into some translation service.

Now, you can do:

 LiftRules.resourceBundleFactories.prepend { 
   case (_, locale) if locale.getISO3Language == "eng" 
      => new MyResources
   case (_, locale) if locale.getISO3Language == "swe" 
      => new MyResources_sv
   case (_, locale) => new DBResourceLoader(locale)
 }

In this example we assume that you have fictional java.util.ResourceBundle subclasses called “MyResources” and “MyResources_sv”. Furthermore, if the ISO 639-3 code is not either “eng” or “swe”, then it attempts to load it from another fictional ResourceBundle subclass “DBResourceLoader”.

Long term, I will probably provide a ProtoDBResourceBundle or something similar, but for now, your on your own to implement the clever string and translation loading :-)

Can Open Source gain Enterprise traction through the back door?

2009-02-16T00:37:00+00:00

NB: What follows is pure opinion and based on no actual metric

As we enter the final stages of the Lift 1.0, I cant help but reflect on open source, the enterprise, and how actions taken by communities that create great projects like lift ultimately have a real and tangible benefit to companies.

For a lot of organizations, open source software is generally seen as too much of a “risk” as their is no official support for products blah blah blah. However, it appears to me at least, that more commercial grade systems are having their open APIs integrated into OSS by the communities that power them, we are indeed seeing OSS adopting enterprise, rather than the other way around. This trend appears to be growing at a pretty quick rate with tools like SAP, Oracle and Salesforce.com (all mainstays of enterprise systems) being integrated in lots of different environments and OSS - for instance, Rails integration for salesforce, oracle middleware drivers for PHP etc etc etc.

I then got to thinking - where the hell did all this integration come from? More than likely, it came from guys and girls who were hacking this stuff out with tools they would usually have access to in a enterprise environment (at the workplace). That to me appears to be a fairly sensible rational, now for the “why”…

This I fear is a little more complicated - perhaps those hackers wanted to play with it? Perhaps it was a pet project (a la google private projects)? Or maybe something else entirely? Whatever the reason, the fact remains those tools exist. Now, the really interesting thing I believe to be about this is that the next time those same developers (and perhaps their co-workers) need to complete a project, what tools might they choose to use? Something that they need to write from scratch…? or something they already made in their spare time for that OSS framework or such?

With platforms such as the JVM welcoming technologies like Scala, that still run on the same infrastructure yet can have a totally different code design, I think we might see more and more OSS “leaking in through the backdoor” into the enterprise? OK, yes, its true that most large dev environments have equally large teams to manage all the accompanying bureaucracy, but I think this type of backdoor behavior might well find traction in the SME arena and give birth to a new erea of OSS coding in the enterprise.

Whatever happens - I for one really do hope that SME and Corporate companies open their eyes to what comunity powered development has to offer.

Stop seeing the risk, and start appreciating the possibilities :-)

Over and out.

Change in XMPie 4.5+ API user command

2009-01-14T09:52:00+00:00

I have recently been working on some API automation for XMPie uProduce and after some head scratching, realized that the username paramater has now changed format.

Previously, you only needed to provide a username and password on api methods, however, in the new version it is important to include your customer name in a windows AD format. For example, if your credentials were:

Customer Name: ABC
User Name: badger
Password: 123456

You would need to present the username in your API call like so:

ABC\badger

I hope this save someone, somewhere, some time!

Now writing for Scala Blogs...

2008-12-23T17:19:00+00:00

Yesterday Jorge Ortiz invited me to join the team at scala-blogs.org - obviously I was delighted, and now I will be sharing my adventures in scala both through this blog, and on scala-blogs.

For my first article I’ll be writing about the PayPal integration I created with David Pollak for Lift.

Watch this space boys and girls, and dont forget to check out scala-blogs.org

Enabling launchers and warnings with scala-maven-plugin

2008-12-19T12:47:00+00:00

If your using the scala-maven plugin and need to enable some of the cool extra functionality then check out these pom.xml snippets:

To enable launchers which let you do

mvn scala:run

use an XML snippet like:

<configuration>
  <scalaversion>${scala.version}</scalaversion>
  <args>
    <arg>-target:jvm-1.5</arg>
    <arg>-unchecked</arg>
   </args>
   <launchers>
     <launcher>
       <id>main-launcher</id>
       <mainclass>com.myproj.TheMainClass</mainclass>
       <args>
         <arg>/some/intial/argument</arg>
       </args>
     </launcher>
  </launchers>
</configuration>

Also note the “-unchecked” compiler argument… this will then prompt you if you have any deprection warnings and suggest how you could go about fixing them.

Will WADL provide a second wind for REST uptake in enterprise?

2008-11-06T20:47:00+00:00

Any one who has ever implemented a SOAP consuming client service will appreciate that the tooling for various languages is a very welcome feather in SOAP’s cap. Without the WSDL tooling that web service developers are accustomed to, a large portion of those developers would probably struggle to use the said service as the sher verbosity of the SOAP entity payload is simply overwhelming.

REST: A idealist view of the web or serious technology?

When REST came onto the scene it appeared to be met with two very different view points. The open source communities and web purists rejoiced… a resource orientated architecture which allowed them to scrap there verbose SOAP and XML-RPC service flow. However, from what I saw first-hand, the reaction from enterprise was somewhat different; namely, one of “So now we have to implement clients by hand? Sod that!”.

For a number of developers, including myself, we could see that REST was a much better architectural style for distributed computing on the web, but with ever decreasing budgets and squeezed time scales the added overhead of creating totally bespoke services - and therefore clients - was one that proved somewhat of a bitter pill to swallow.

WADL: Here to save the day?

Recently, WADL is coming to the fore as a structured way for providing description and automated development workflow for using REST services. WADL is based around similar principles to WSDL in that it provides a contract of service between service resource and the client.

If WADL can provide a robust framework for automated tooling with REST, I really think that we’ll see REST pick up a second wind of uptake in the enterprise setting. With players like Yahoo! leading by example, how long will it be before others follow suit?

Only time will tell…

Adding JASYPT encryption to your scala JPA entity classes

2008-10-27T00:58:00+00:00

I’ve recently been looking at how best to make sensitive user data encrpytable within JPA models when in use with lift. I came accross JASYPT and it seems to be a really nice encryption tool. If you want to add it to your scala JPA models, do something like this:

package eu.getintheloop.bloglite.model

import javax.persistence._
import java.util.Date
import org.jasypt.util.password.BasicPasswordEncryptor

@Entity
@Table(){val name="users"}
class User extends BaseEntity {
  @Id
  @GeneratedValue(){val strategy = GenerationType.IDENTITY}
  @Column(){val insertable = false, val unique = true}
  var id: Long = _

  @Column{val nullable = true}
  var first_name: String = "" 

  @Column{val nullable = true}
  var last_name: String = "" 

  @Column{val unique = true, val nullable = false}
  var username: String = "" 

  @Column{val nullable = false}
  var password_hash: String = "" 

  @Column{val unique = false, val nullable = false}
  var email: String = "" 

  @Column{val unique = false, val nullable = false}
  var is_active: Boolean = false

  def password: String = this.password_hash

  def password_=(in: String) = this.password_hash = encrypt(in)

  def authenticate(in: String): Boolean = {
    new BasicPasswordEncryptor().checkPassword(in, password) 
      && is_active
  }

  private def encrypt(in: String): String = 
    new BasicPasswordEncryptor().encryptPassword(in)

}

Explictially Setting Application DocType with Lift

2008-10-01T10:02:00+01:00

As you might know, I use the lift framework a lot - I recently came across a strange issue where I couldnt set my own doctype in the layout template. Maybe this nugget of information will be usefull for someone:

ResponseInfo.docType = { 
  case _ if S.getDocType._1 => S.getDocType._2 
  case _ => Full(DocType.xhtmlStrict) 
}

/*
Avalible options are:

xhtmlTransitional
xhtmlStrict
xhtmlFrameset
xhtml11
xhtmlMobile

*/

UPDATE (Lift 2.1)

Be sure to use the following as ResponseInfo is deprecated.

LiftRules.docType.default.set(r => Full(DocType.xhtmlStrict))

Doing so will ensure you don’t get any warnings etc during compile time.

Free your mind with FreeMind

2008-09-30T23:44:00+01:00

Check out this great open source mind mapping tool called FreeMind - I recently came accross it whilst reading this article about hibernate 2nd level caching.

A true testament to the power of OSS and the great software it can produce.

World First: XMPie Dashboard Application for Apple iPhone

2008-08-26T17:57:00+01:00

Readers of my blog will know that I love macs. Absolutely anything to do with Apple interests me; to that end, it will not surprise you that I have been doing some iPhone development.

For kicks I decided to write a mobile XMPie Dashboard application which would let you control XMPie uProduce whilst out and about. Whilst this is a sheer technology demonstration, its fairly functional and a great example of how with the right knowledge and skill, absolutly anything is possible within the XMPie toolset.

Below is a quick screencast demonstrating the application in all its glory!….

Im interested in your thoughts, so please comment below if you feel so inclined…

Running Resin Alongside Microsoft IIS

2008-08-24T01:10:00+01:00

I recently thought that it would be good to run a Servlet container alongside IIS which is so well embedded in the clients that I deal with as they are mainly .NET types…

Anyone who has read my blog before will now that I have somewhat of a penchant for Resin - so once again, this was my choice of servlet container. Luckily for me, Caucho are awesome enough to supply an ISAPI filter which drops right into your IIS system, making Resin just an other ISAPI filter! Very sweet.

If you too would like to have some Java sweetness on your M$ box, then check out this walkthrough

I Want To Be A Rocket Man!

2008-08-23T15:05:00+01:00

I just came across this on DIGG. I wouldn’t usually blog about this type of thing, but its truly amazing… can you imagine if they massed produced these one day - we might be flying to work!

Hessian Kit Framework for Objective-C Released

2008-08-09T16:54:00+01:00

This sweet new cocoa framework is now available for download from Sourceforge - if your writing anything for mac that involves client <-> server interplay, then this is certainly worth a look.

Small, fast and lightweight. What more could you want?

Enterprise Social Messaging

2008-08-08T11:06:00+01:00

Programatically adding Applications to the Login Items

2008-07-28T14:27:00+01:00

Its taking some kicking around, but here is the fruit of my labour… using core foundation over the RC bridge.

require 'osx/cocoa'

loginItems = OSX::CFPreferencesCopyValue(
      "AutoLaunchedApplicationDictionary", 
      "loginwindow", 
      OSX::KCFPreferencesCurrentUser, 
      OSX::KCFPreferencesAnyHost)

application_path = File.expand_path("~/path/to/your.app")
application_hidden = false

loginItems << OSX::NSDictionary.dictionaryWithObjects_forKeys(
    [ application_path, application_hidden ], 
    [ :Path, :Hide ])

# puts loginItems

OSX::CFPreferencesSetValue(
    "AutoLaunchedApplicationDictionary", 
    loginItems, 
    "loginwindow", 
    OSX::KCFPreferencesCurrentUser, 
    OSX::KCFPreferencesAnyHost)

OSX::CFPreferencesSynchronize("loginwindow", 
        OSX::KCFPreferencesCurrentUser, 
        OSX::KCFPreferencesAnyHost)

With any luck this might help someone :-)

Checking Network Connectivity with Ruby

2008-07-25T10:53:00+01:00

I’ve just been playing around with a few things and stumbled accross a really neat ruby lib called ‘timeout’. I used it to make a quick check for network connectivity like so:

require 'timeout'
require 'socket'

begin
  timeout(10) do
    TCPSocket.new("www.rubyforge.org", 80)
  end
    puts true
rescue
  puts false
end

For so few lines thats a really neat; I thought so at least! :-)

Great guide for people interested in Scala

2008-07-17T17:33:00+01:00

I just came across a great Scala presentation from JavaOne. If your not looking at learning Scala, then this presentation will give you a whole heap of reasons to get up and do something about it.

Download the article from here

The Website Is Down!

2008-07-09T11:57:00+01:00

A friend of mine sent this to me - anyone working in IT will appreciate the funny side of this!!

Bongtastic load testing!!

2008-07-08T22:56:00+01:00

My mind escapes me at this late hour on a Tuesday eve, but Im fairly sure i’ve blogged about the httperf tool from HP Labs before.

Anyway, if I havent, thats a different story, as today, I want to point you good reader to the wonderfull tool that is Bong

httperf has somewhat cryptic command syntax, and bong is a really nice wrapper around it which make leveraging the full power of httperf a breeze.

Check out the Bong home page for more info - I’ll put up a quick tutorial when its not so late and my brain is functioning properly.

How to use Photoshop Smart Object

2008-06-28T13:02:00+01:00

I just stumbled accross this article - the subject matter certainly puts questions over the sanity of the author, but, never the less it is actually quite a good example (or tutorial if you can go that far) of how to use Smart Objects within Photoshop.

Were Moving!

2008-06-25T15:01:00+01:00

This is just a quick note; my apologies for the lack of posting lately - its not that I haven’t had anything to write about, quite the opposite… but right now im preparing for a server upgrade and I have already ported the blog data, so yes, Im being a bit lazy, but we should be all ported to the new servers in a couple of weeks.

So bear with me guys, Cheers

iPhone 3G – Roll on July 11th

2008-06-10T12:41:00+01:00

If you havent seen it already, then where the hell have you been? iPhone 3G is comming out July 11th, and, yes, like all the other Apple disciples I shall be venturing down to my local store to get one. Sweet mother of god, just look at it…. its got GPS and everything…

Amazon EC2; Could This Be The Best Computing Platform Ever?

2008-06-09T23:28:00+01:00

I’ve recently started using Amazon EC2 and put quite simply, it could well be the best computing platform the world has ever seen! Its flexible, scalable, and very competitively priced which makes it an attractive proposition for users that are currently on fixed priced virtual machine hosting.

The real differentiator with EC2 is that you are effectively building a linux machine from scratch that can be flug around there cloud to any location. So, that means you could make an image of your running machine using there AMI Tools and have your box running in East Coast America. Pretty straight forward so far. You then get wind (pun intended) that a massive hurricane is about to hit the East Coast, and you’d feel safer if your servers were far far away, right? No problem, with a simple one line command you can re-deploy the image to the West Coast cloud and everything would just pick up where it left off! Amazing!

On top of all that clever trickery, you get pretty decent control over firewall ACL’s and the choice of box configuration is pretty decent - the ‘small’ machine still has 1.7GB of RAM, which is a boat load more than most of VM’s on the market. You can even get 7.5GB and 15GB variants.

Amazon, you have out-done yourselves. Congratulations!

How To Fix Generating XMPie API Java Stubs With Metro

2008-06-06T17:39:00+01:00

As a number of you are aware, I have spent quite a bit of time wrestling with XMPie uProduce SOAP API with the CXF service framework in Java. I finally (and with quite a lot of disappointment on my part) gave up trying to get that to work. CXF just does not want to play nice and I cant seem to make the XJB bindings work correctly with it and the mish-mash WSDL coming from uProduce

Anyway… with that rant over, I tried to be objective and switched to using Metro - ok, out of the box I still had a whole heap of problems, exactly as I did with CXF. However, I have managed to wangle it!

Step 1

You will need access to an XMPie server with the API’s installed and working. For the sake of this example i just wget’d the WSDL file so the generation code was not so bloated.

Step 2

You’ll need an XSD XJB binding file which looks like this:

<bindings xmlns:xjc="http://java.sun.com/xml/ns/jaxb/xjc" xmlns="http://java.sun.com/xml/ns/jaxb" xmlns:xsd="http://www.w3.org/2001/XMLSchema" version="2.0">

  <globalbindings>
    <xjc:simple></xjc:simple>
  </globalbindings>

  <bindings scd="~xsd:complexType">
    <class name="ComplexTypeType"></class>
  </bindings>

  <bindings scd="~xsd:simpleType">
    <class name="SimpleTypeType"></class>
  </bindings>

  <bindings scd="~xsd:group">
    <class name="GroupType"></class>
  </bindings>

  <bindings scd="~xsd:attributeGroup">
    <class name="AttributeGroupType"></class>
  </bindings>

  <bindings scd="~xsd:element">
    <class name="ElementType"></class>
  </bindings>

  <bindings scd="~xsd:attribute">
    <class name="attributeType"></class>
  </bindings>
</bindings>

Save this as xsd.xjb (or whatever you like, just make sure the name is reflected in the command below)

Step 3

Actually generate the code! Your command might look different, but heres mine:

  wsimport -b http://www.w3.org/2001/XMLSchema.xsd \
    -b src/xjb/xsd.xjb \
    -keep \
    -s src/java \
    -d target \
    -p com.xmpie.wsapi.icp \
    src/wsdl/InteractiveCampaign_SSP.wsdl

You should then see something like the following in your finder (or other OS file browser if not on mac):

These are the class files created for the InteractiveCampaign WSDL - all 128 of them!

Anyway, I have run out of time for today, but will try and get another blog up soon about how to use this in a client application from a java CLI.

Creo Darwin Set To Hot Up VDP Competition

2008-06-06T14:54:00+01:00

Whilst at Drupa I was lucky enough to get a back room showing of what Creo will be releasing in Q4 of 2008 and into 2009.

The first impressions are really great - the native UI (Cocoa on OSX and .NET windowing on WinTel) means that no matter what your platform you get widgets that you are familiar with and work seamlessly with the operating system. That really is quite a differentiator when compared to competing products, and one that I am sure a lot of people would welcome as even from personal experience, quite a number of other solutions are using SWNG, or, god forbid, AWT in Java which are clunky and just not up to the modern users expectations.

Plugins

One of the things that really is very good about Darwin is the plugin system they have devised. Whilst I currently have no information on what inner complexities one might face with implementing your own plugin, the architecture I was shown looked very open, and very friendly - which is a massive plus point in my eyes.

Performance

Darwin was awesomely quick too - speed and beauty in the same package! Eliot Harper has blogged some very nice performance testing between XMPie uDirect and Creo Darwin which can be found here

Integration

Creo have changed the way in which the Darwin working environment is persisted - its now a separate DVJ file. This means that when working with InSite, the system can automagically create the darwin files for you. I’d go out on a limb and say that if InSite is able to create those files, then any other system you might want to write would also be able to generate DVJ files - this could really create some interesting options for mash-up style workflows!

What About The Cons?

Ok, so all products have there cons - thats life - at the moment Darwin only supports flat file data-sources. Perhaps that will change over time, but we’ll see I guess. Release is quite some time off so who knows what those guys might come up with!

Kudos Creo , this really is great work.

Another interesting article about Darwin and InSite can be found here

ProJet 3D Printer for Engineering Prototypes

2008-06-06T14:26:00+01:00

This post is a bit off of the usual code focused articles I write, but having just arrived back from Drupa I feel that I just need to write a quick post about this as I really did think it was awesome!

The product in question is 3D Systems ProJet

No word of a lie, this machine can “print” 3D models of things via an InkJet process - it really is amazing the fine, granular control they have over how it works its magic. Like I said, very off topic, but freaking awesome at the same time!!

Is Facebook lowering user expectations?

2008-05-28T07:05:00+01:00

How many times have you been using Facebook and received the “Sorry, an error has occurred. We’re working on getting this fixed as soon as we can.” message?

This appears to happen fairly often (relativly speaking) from what I can see. I wonder if this lowers peoples expectations of what uptime they should expect from the big web 2.0 platforms? For instance, I dont ever remember getting such a message on YouTube… but then again a whole lot more people use facebook than use YouTube…

I guess my question is this… Is it acceptable to be having public downtime on such sites/services that are becoming such a part of everyday society? I would say no, its not. Were told that facebook has somewhere in the region of 10,000 servers! So goodness knows why with 10K servers they cant get a decent amount of uptime for most users.

Zero Turnaround Release Open Source Version of JavaRebel Just for Scala

2008-05-21T09:57:00+01:00

Its been in the wings for some time now but the announcement has finally been made public. This is great news as it means that when were coding in Scala the classes in your web container will automatically be updated without the need to restart the container and redeploy the WAR.

For more on this check out the announcement

Using Embedded Resin as a Development Container

2008-05-19T08:24:00+01:00

I recently read this article and realised that I had never blogged about the resin plugin that works perfectly with lift for development and deployment.

Heres what you need to add to your pom.xml

    <build>
   ...
   <finalname>${project.name}</finalname>
   ...
    
      <plugin>
        <groupid>com.caucho</groupid>
        <artifactid>resin-maven-plugin</artifactid>
        <version>3.1.6</version>
        <configuration>
          <contextpath>/</contextpath>
        </configuration>
      </plugin>
    ...
</build>

<pluginrepositories>
...
<pluginrepository>
  <id>caucho</id>
  <name>Caucho</name>
  <url>http://caucho.com/m2</url>
</pluginrepository>
...
</pluginrepositories>

Note, the final name var is important, as thats what resin uses to point at. Dont worry tho, as lift already has a name element so provided you haven’t deleted it, your laughing.

Next, to boot the server, just do:

    mvn resin:run

That should be it! Enjoy!

Exposing Lift Components via Hessian/Burlap with Resin Container

2008-05-05T19:51:00+01:00

Right, as you know, I use the lift framework a lot these days. I also use Resin as my choice container. So, i recently got to thinking, wouldn’t it be cool if i could just bosh together a simple way to expose controllers in lift via the groovy protocols made available by Resin - well, I have here an example of just such a thing.

Lift Side

Ok, before anyone has a poke at this code… it was a very quick and dirty example, so its not ideal, and obviously your code would be a whole lot more complex, but this article is just about proving concept.

I set up the object that I want to expose via remoting. Critically, there would be nothing stopping me using the persistance built into lift, or anything else available to java or scala for that matter. The key is the traits… as they are like java interfaces (broadly speaking), they let resin know what methods it can call.

package com.timperrett.resin.example.controller

trait HelloTrait {
  def hello: String
}
class HelloWorld extends Object with HelloTrait {
  def hello = "Hello from lift" 
}

Resin Configuration

Next, we add a bit of config to resin so that it knows how/where to expose the service. Critically, we could have used Hessian, Burlap or CXF (for SOAP) and all by being pushed out of the container with zero extra effort.

 <web-app root-directory="webapps/lift-remoting-example" id="/lift-remoting-example">
  <servlet-mapping url-pattern="/api/hello-world" servlet-class="com.timperrett.resin.example.controller.HelloWorld">
    <protocol uri="hessian:"></protocol>
  </servlet-mapping>
</web-app>

Client Side (in another language!)

To proove the point, I thought I would just make a simple client in another language (namely ruby):

require 'rubygems'
require 'hessian'

url = http://127.0.0.1:8080/lift-remoting-example/api/hello-world
client = Hessian::HessianClient.new(url)
puts client.hello

Again, a very simple example, but one that proves the point. You should then see “Hello from lift” output in the terminal window.

Project Files

You can download my example from here

If you found this article interesting, or have played with this yourself also, then by all means leave a comment or get in touch.

Hessian and Burlap for API transit

2008-05-02T13:59:00+01:00

Web Services for API? Mehhh, haven’t we heard all this before?

Well, frankly, yes, you probably have. But you might not have had the full story. Im not talking about SOAP or other such protocols that require a serious amount of time at remoting-protacol-fat-camp… Today I would like to take some time to talk about the Burlap and Hessian protocols. For the astute readers, yes, those are the same ones that I was banging on about in a previous post when talking about hot technology for 2008. They are both slick remoting systems that run on Resin (they probably can be run on other JEE servers too, but frankly I haven’t tried), and according the chaps at Caucho:

“Hessian and Burlap are compact binary and XML protocols for applications needing performance without protocol complexity. Hessian is a small binary protocol. Burlap is a matching XML protocol. Providing a web service is as simple as creating a servlet.”

Sounds pretty cool doesn’t it? Well they are. I have been playing with Resin running these protocols and evaluating them as methods for creating robust API which 3rd parties can easily integrate with. Before reflecting on what I have experienced during these trials with Burlap/Hessian I would have instantly said that Hessian was better, but then I got to reflecting on use in the world: is it really practical to use binary remoting protocols for services that the 3rd party user does not know well? I think about the times in which I have been developing clients to services I don’t know (or didnt build for that matter) and I do usually end up looking at the message body when trying to troubleshoot any issues that might arise - and Hessian would be totally unworkable in that instance.

Anyway, to conclude, if you or a partner company are working in tandem and have a good working relationship then sure, its got to be Hessian all the way; its fast, its light weight… its all good. However if your exposing services for developers at a 3rd party, then you should really consider Burlap, as it could make someone else’s life so much easier. On the other hand, you could just be greedy and use Resin to expose Hessian, Burlap and SOAP!

Flash goes open source!

2008-05-01T23:48:00+01:00

It appears that flash has finally gone open source - craziness all round from Adobe at the moment.

http://aralbalkan.com/1332

Papervision 3D Trailer

2008-04-28T10:08:00+01:00

I appreciate this is a little late in terms of the papervision bandwagon, but I only came across this on the tube this morning. A truly awesome display of what the latest version of flash + papervision is capable of:

Geek Honors: Joins Scala Lift Core Team

2008-04-24T23:17:00+01:00

My appologies; I hardly ever blow my own trumpet on this blog, but I just cant let this one pass without a good old toot of the proverbial horn.

Today I joined the lift committers. That might not sound much, but when you think that lift will be the next Ruby on Rails size phenomenon to hit the development community, thats pretty dang awesome.

I shall toot no more - respect to David and the rest of the other guys on lift core… you freaking rock!

Accessing OSX Network Interfaces From Wireshark X11

2008-04-23T11:57:00+01:00

Just a short post for people who might not be sure how to get Wireshark to read the interfaces from your mac…

I had to run the applications as root:wheel to let the X11 interface read the network interface devices. To do the same, just run:

<code>sudo /Applications/Wireshark.app/Contents/MacOS/Wireshark</code>

Then, when prompted, just enter your password (provided you are a machine administrator that is). Wireshark X11 should then boot up and work without problem.

If your on 10.4 you will need to install X11 beforehand

NGINX (Engine-X) Rewrite Rules For CakePHP

2008-04-17T18:59:00+01:00

I’ve been doing some work with NGINX of late and anyone familiar with CakePHP will know that it ships out of the box with Apache .htaccess files to make sure that the URL’s are devoid of there query string.

Anyway, enough talk, if you want to host cakephp on NGINX, you’ll need to use a vhost like so:

    server { 
        listen       80;
        server_name  somedomain.com;
        access_log  /var/www/logs/somedomain.access.log  main;
        error_log   /var/www/logs/somedomain.error.log info;
        rewrite_log on;

        # rewrite rules for cakephp
        location / {
          root   /var/www/sites/somedomain.com/current;
          index  index.php index.html;

          # If the file exists as a static file serve it 
          # directly without running all
          # the other rewite tests on it
          if (-f $request_filename) { 
            break; 
          }
          if (!-f $request_filename) {
            rewrite ^/(.+)$ /index.php?url=$1 last;
            break;
          }
        }

        location ~ \.php$ {
          fastcgi_pass 127.0.0.1:9000;
          fastcgi_index index.php;
          fastcgi_param SCRIPT_FILENAME \
          /var/www/sites/somedomain.com/current$fastcgi_script_name;
          include fastcgi_params;
        }
    }

Apache CXF Graduates From The Incubator

2008-04-16T23:12:00+01:00

Today was the day that Apache CXF graduated out of the incubator and will now become a full bonafied apache project. This is such brilliant news as it means that CXF should replace Apache Axis and Xfire as the defaco Java services framework.

Congratulations to Dan Kulp and the rest of the CXF team - all the sterling work has paid off after about 20+ months of rocking development.

I wish them all the best for the future - CXF will be a big hit!

Tutorial: Getting Started with Apache CXF

2008-04-12T12:53:00+01:00

If you are looking to lean CXF, then you should really read this:

http://www.jroller.com/gmazza/entry/using_strikeiron_s_super_data

It was one of the best written tutorials I managed to find and covers both Maven and Ant compilation.

Kudos Glen!

Java Is Not A Dirty Word…

2008-04-11T11:01:00+01:00

Strong title I know… It feels like thats how Java is regarded these days - Ruby and Python are becoming the golden boys of programming choice and poor ol’ java is being left out in the cold because people only remember it as having crap costing/license models where you the developer needed to pay for everything.

Well, I have good news for you my friends, Java should no longer be a dirty word! The inertia that Ruby on Rails had in the Java community did cause quite a lot of controversy its true, but its almost like after the initial xenophobia had worn off they sat up and thought “you know what, these guys might actually be onto something here”. I recently got into using Maven 2, and god, my good god, its a million times better than Ant, which just sucked so badly I cant explain! For anyone who is familiar with Rake, Maven is like Rake, but with all the package management stuff built right in - it does still have a certain level of entry required, but it just simplifies the process of created projects and managing dependencies in a very streamlined way.

The language itself has evolved quite a lot too - the new annotations are used heavily be JEE apps and frameworks - take CXF as an example; it will eventually replace XFire and Axis2 as the premier SOAP/WebService framework for Java, its a very very powerful tool and pretty easy to use (relatively speaking)

Anyway, I digress… my point is that if, like me, you left Java because the last time you worked with it you were slogging it out with 1.4 and Ant hell, then perhaps its time you took another look? Java has had significant investment as a platform, and there is a hell of a lot of good code out there for doing pretty much anything you can think of. In all honesty, one of the main reasons I have come back to JEE and the JVM in general is deployment - the JVM is very robust, and the containers built of top of it like Resin and Glassfish are being used in very high-concurrency environments and the deployment method via JAR or WAR is just so much more robust than on a platform like Ruby - and thats coming from having spent nearly the past 3 years solidly coding Ruby and doing all manner of deployments!

Dont write Java off - its an amazing platform, so what if its not as easy to get started as with other languages; think long-term, its a language that will grow with you rather than one you might someday reach the ceiling of - or just find incredibly annoying when the application dispatchers crash for a past time ;-)

Over and out

One Click Installer for Apache Maven 2.0.8 on Mac OSX

2008-04-10T10:06:00+01:00

I have put together an all-in-one installer for Apache Maven 2.0.8 and Scala 2.7.0-final! Just download the installer, be it on 10.5 or 10.4, just download the right distro for your operating system and then follow the installation procedure.

The installer will then automatically append all the right shell variables to your $PATH so they will be freshly available from the terminal - splendid.

You can find the all in one maven installer for OSX here

Any problems feel free to get in touch.

New Download Section on timperrett.com

2008-04-09T18:24:00+01:00

Well, I finally pulled my finger out and put together a download section on my main site - I have included a couple of different packages right now, but I should be adding stuff up there on a fairly regular basis so stay tuned listeners!

You can find my new download section here

How to use rails 2.0.2 without a database

2008-04-09T14:53:00+01:00

Step 1

Run the rake task to freeze rails 2.0.2 into the project so your not reading from gems:

<code>rake rails:freeze:gems</code>

Step 2

Remove the database.yml file from RAILS_ROOT/config/

Step 3

Change line 21 in RAILS_ROOT/config/enviroment.rb to:

<code>config.frameworks -= [ :active_record ]</code>

Bobs your uncle, that should be it!

Cross Platform Language Mashups Are The Way Forward

2008-04-05T16:11:00+01:00

With new cross-language mash ups appearing all the time with things like JRuby and Jython it makes me wonder where all this is going? Will we have J2EE apps deployed alongside dynamic languages all within the same container, all capable of leveraging the awesome JVM runtime? If so, then wow, thats an awesome proposition. Pretty much every language you can think of is now deployable within a java container - its even possible to deploy .net applications via mono!

Perhaps people will start to standardize on the JVM as a platform - all this constant fighting over runtimes, platforms and environments makes work for the developer and architect extremely difficult; we want to keep up with the latest developments, but it just makes it so difficult to do that with all this chopping and changing.

Anyway, things I think are going to be seriously hot this year are:

Resin Application Server

Lift Framework and Scala

JRuby

Hessian Binary Remoting

We’ll see how things play out over the next few months, but whatever happens, its going to be exciting to see how all these technologies interact with each other - which fizzle out and which go on to achieve wonderful things.

Updated XMPie / Merb Extension for Merb 0.9.2

2008-04-03T17:23:00+01:00

Just a quick note that I have refactored the code base for the XMPie ICP extension for Merb 0.9.2 and added some extra cool things that are inherited by the ICP subclasses.

Now, its possible to add validation into the model subclass. Lets say we have a feild defined in our uPlan called ‘email’ and ‘user_name’. I need the ‘user_name’ field to be required, which you can now do like so:

class Visitor < Xmpie::Icp::Base
  validates_presence_of :user_name
end

# in your controller...
@visitor.valid? # => true if has a user_name, otherwise false

All exciting stuff! When XMPie make ICP a more viable persistance tier, then this will no doubt be the quickest and slickest way to implement front ends in. Furthermore, with the Engine Yard team working on mod_rubinus then we should see ruby web application performance finally get where it needs to be.

Extending Adobe CS3 Applications

2008-03-31T16:28:00+01:00

Of late ive been doing some investigation into extending InDesign with plugins and such. Traditional Adobe plugins needed to be written in C++ but now with the addition of the CS3 SDK its possible to write JSXB’s from which you can extend the UI and all sorts of other good stuff!

I’ve been playing around with it, and true, its not as quick as C++, but its so much vastly more accessible the trade off does not seem like a major crime. If you want to try it for yourself, check out the SDK here

Capistrano recipe for Merb 0.5.3 in production

2008-03-14T12:38:00+00:00

Further to my article about God, and using it for watching Merb it seemed like a good idea that I also post my capistrano recipe so people can get the complete picture :)

# MIT License
# 
# Copyright (c) 2008, Tim Perrett
# 
# Permission is hereby granted, free of charge, to any person 
# obtaining a copy of this software and associated 
# documentation files (the "Software"), to deal in the Software
# without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, 
# and/or sell copies of the Software, and to permit persons 
# to whom the Software is furnished to do so, subject to the 
# following conditions:
# 
# The above copyright notice and this permission notice shall 
# be included in all copies or substantial portions of the Software.
# 
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF 
# ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED 
# TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A 
# PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT 
# SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR 
# ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN 
# ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE 
# OR OTHER DEALINGS IN THE SOFTWARE.

set :application, "domain.com" 

set :repository, "svn://yourrepo.com/path/to/repo" 

# Set your SVN and SSH User
set :user, "sshuser" 
set :svn_user, "svnuser" 
#Set the full path to your application on the server
set :deploy_to, "/path/to/your/#{application}" 

# define your servers
role :app, "domain.com" 
role :web, "domain.com" 
role :db, "domain.com", :primary => true

desc "Link in the production extras" 
task :after_update_code do
  run "ln -nfs #{shared_path}/log #{release_path}/log" 
end

desc "Merb it up with" 
deploy.task :restart do
  # you need to add restart tasks for each port you plan 
  # to run merb on. Whilst this is a little long winded, 
  # it will ensure uptime compared to the sledge hammer
  # than is #{current_path}/script/stop_merb
  run "(cd #{current_path}; merb -k 4006); sleep 1; \
  #{current_path}/script/merb -u timperrett -G timperrett \
  -M #{current_path}/config/merb.yml -p 4006 \
  -e production -d" 
  run "(cd #{current_path}; merb -k 4007); sleep 1; \
  #{current_path}/script/merb -u timperrett -G timperrett \
  -M #{current_path}/config/merb.yml -p 4007 \
  -e production -d" 
end

Any problems, let me know :)

Monitoring Merb Processes in the Wild with God

2008-03-13T15:04:00+00:00

Before you start reading, this post only applies for Merb 0.5.3, any newer version is totally untested

When you take Merb out into the wild, it does, unfortunatly, suffer a lot of the same problems as the mongrel handler than runs Rails.

There is however a saviour out there - God - To clarify, im not talking about the man upstairs; rather the process monitoring tool which rocks at restarting bloated mongrel processes on nix based OS.

The Merb handler does not have any way of restarting a running cluster, so you physically have to stop, then start a new merb process. This is somewhat out of sync with how the god process handling and restarting works in that you define start, stop and critically, restart paramaters. To get around this we have to use a hacky sleep then start method - its not ideal, but hey, it works :)

Rather than letting merb handle the process forking, what were going to do is let God handle the writing of Pids and managing of the process. Ok, less of all this talk and lets take a look at some configuration code for the God configuration.

# MIT License
# 
# Copyright (c) 2008, Tim Perrett
# 
# Permission is hereby granted, free of charge, to any person 
# obtaining a copy of this software and associated 
# documentation files (the "Software"), to deal in the Software
# without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, 
# and/or sell copies of the Software, and to permit persons 
# to whom the Software is furnished to do so, subject to the 
# following conditions:
# 
# The above copyright notice and this permission notice shall 
# be included in all copies or substantial portions of the Software.
# 
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF 
# ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED 
# TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A 
# PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT 
# SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR 
# ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN 
# ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE 
# OR OTHER DEALINGS IN THE SOFTWARE.

SITE_LOCATION = "/var/www/sites/domain.com" 
MERB_ROOT = "#{SITE_LOCATION}/current" 
MERB_CONFIG = "#{SITE_LOCATION}/shared/config/merb.yml" 
MERB_ENVIROMENT = 'production'
PROCESS_USER = 'timperrett'
PROCESS_GROUP = 'timperrett'

%w{4006 4007}.each do |port|  
  God.watch do |w|

    pid_path = File.join(MERB_ROOT, 'log',"merb.#{port}.pid")

    w.name = "merb-#{port}" 
    w.interval = 30.seconds # default
    w.start = "#{MERB_ROOT}/script/merb -u #{PROCESS_USER} \
    -G #{PROCESS_GROUP} -M #{MERB_CONFIG} -p #{port} \
    -e #{MERB_ENVIROMENT} -d" 
    w.log = "/home/timperrett/godmerb.log" 
    w.stop = "(cd #{MERB_ROOT}; merb -k #{port})" 
    w.restart = "(cd #{MERB_ROOT}; merb -k #{port}); sleep 1; \
    #{MERB_ROOT}/script/merb -u #{PROCESS_USER} -G \
    #{PROCESS_GROUP} -M #{MERB_CONFIG} -p #{port} \
    -e #{MERB_ENVIROMENT} -d" 
    w.start_grace = 5.seconds
    w.restart_grace = 20.seconds
    w.pid_file = File.join(MERB_ROOT, "log/merb.#{port}.pid")
    # w.group = "merbs" 
    w.behavior(:clean_pid_file)

    w.start_if do |start|
      start.condition(:process_running) do |c|
        c.interval = 10.seconds
        c.running = false
      end
    end

    w.restart_if do |restart|
      restart.condition(:memory_usage) do |c|
        c.above = 51.megabytes
        c.times = [3, 5] # 3 out of 5 intervals
      end

      restart.condition(:cpu_usage) do |c|
        c.above = 50.percent
        c.times = 5
      end
    end

    w.lifecycle do |on|
      on.condition(:flapping) do |c|
        c.to_state = [:start, :restart]
        c.times = 5
        c.within = 5.minute
        c.transition = :unmonitored
        c.retry_in = 5.minutes
        c.retry_times = 5
        c.retry_within = 2.hours
      end
    end

  end
end

Im not sure that this is ideal, but it certainly seems to work for me and importantly keeps the site running without problems - which is a dang sight better than them becoming unresponsive!

I hope this might help someone, somewhere….

Choosing a style of personalised URL

2008-03-13T10:45:00+00:00

I think that pretty much all the ‘Personalised URL’ jobs that I have ever needed to do have always been of the directory style, e.g:

www.domain.com/mylovelyusername

However, it strikes me that a better and more programatically favourable format is the sub-domain pattern, e.g:

mylovelyusername.domain.com

In this instance you don’t have to worry about holding information in sessions, or setting up anything else complicated to make sure that they visit that URL first - it doesn’t matter with the sub-domain pattern as you always have there unique identifier to hand in the URL. Im not wholly sure why it is that more people don’t see the benefit of doing this - if you ask me it ‘feels’ more personal that a one time visit to a directory style URL.

Colloquy: An Awesome OSX IRC Client

2008-02-22T21:06:00+00:00

After having used Limechat IRC, I just discovered Colloquy and its awesome! Really slick interface - loving there work. Help yourself out and download it!

Personalised video’s with XMPie Version 4

2008-02-19T11:26:00+00:00

Check out these awesome personalized video’s which its now possible to produce using XMPie version 4!

Kudos to Eliot as he has been talking about personalized video for some time now and its finally been brought into the core version 4 product.

Check these awesome samples from the recent XUG meet in Vegas…

Working MySQL 5 OLEDB Connectivity for XMPie Server

2008-02-15T13:30:00+00:00

Well, after a long stint of having no decent MySQL support (as uProduce only supports up to version 4 out of the box. Other than my ODBC monkey patch there has been no version 5 support), I have managed to find a working, and critically viable, solution for using MySQL 5 with XMPie uProduce.

cherrycitysoftware.com sell (not that you can even call it that, its only $10) a little mysql driver that with a little bit of work you can make work with uProduce and uPlan - its nice to have MySQL back!

You can use a DSN with generic DB like this:

<code>Provider=OleMySql.MySqlSource.1;Password=YOURPASSWORD;Persist 
SecurityInfo=True;User ID=YOURUSERNAME;Data Source=YOURDBHOST;
Location=YOURDBHOST;Mode=Read;Trace="";Initial Catalog=INITIALDATABASE</code>

Initial results are great - its working cleanly and quickly - good times with MySQL!

XSLT using Ruby

2008-01-31T19:10:00+00:00

Short Story

libxsl-ruby is a nightmare to install. If you can indeed get it to install at all. I couldnt get the bloody thing to install at all on my 10.5 OSX box; and thats with ALL the required libs installed. I get the impression that its not very well maintained anymore and the project following has dropped off.

I turned to ruby-xslt - based on REXML, so yes, its slower. If your looking for speed then persist with the libxsl ruby path, but otherwise, save yourself a lot of time and just use this. Its vastly simpler!

Long(er) Story

Libxsl sucks. Strong statment I know. I would say im a fairly techincal chap, and ive just spent, nay, wasted, the last 3 - 4 hours trying to work out why libxsl-ruby wont install. Thats a bit of a bugger indeed. Who knows what the problem is, but ultimately its less portable due to reliance of the Gnome XML parsing libs. Granted there very fast, but thats no help at all if you cant even install them.

REXML vs Libxml is a non-contest. We all know libxml is significantly faster - if you need that speed then i strongly advise you to continue with libxml; it will pay dividends in the end. If however your not going to need very fast parsing, then really, save yourself a lot of wasted time and use REXML.

REXML is bundled with the ruby distribution, so its also a lot more portable that libxml2. Anyway, if your even still reading, stop now, and bosh this into your terminal:

<code>sudo gem install ruby-xslt</code>

Job done.

Flash: Go To And Learn

2008-01-29T14:18:00+00:00

I stumbled upon this the other day; it looks to be a really really great resource if your wanting to get up to date with all the latest flash goings on!

For the rubber band flicking child in us all

2008-01-20T23:32:00+00:00

Santa, this Christmas I would like a rubber band gatling gun….

Deploying Ruby WebGen with Capistrano

2008-01-11T17:47:00+00:00

WebGen is a groovy little tool that makes building HTML sites a lot simpler, and makes maintenance a lot more efficient via clever templating. I wont labour the pro’s and con’s of the project here, I’ll just get on with explaining how to deploy version webgen sites using our good friend Capistrano.

First off, install Capistrano:

<code>$ sudo gem install capistrano -y</code>

Next, we need to prepar our webgen dir with all the relevant deployment information. All these commands presume you have ‘cd’ to the webgen project directory Capistrano provides a simple way to do this

<code>$ mkdir config
$ capify .</code>

Next we need to setup our new deploy.rb file, mine looks a little something like this:

set :application, "yourapplication.com" 
set :repository,  "svn://yourhost/path/to/webgen/site/output" 

# If you aren't deploying to /u/apps/#{application} on the
# target servers (which is the default), you can specify 
# the actual location via the :deploy_to variable:
set :deploy_to, "/var/www/sites/#{application}" 

# If you aren't using Subversion to manage your 
# source code, specify your SCM below:
# set :scm, :subversion
set :user, "deploy" 
set :svn_user, "badger" 

role :app, "yourapplication.com" 
role :web, "yourapplication.com" 
role :db,  "yourapplication.com", :primary => true

# Overrides the default task supplied by cap, as this 
# is a static site there are no mongrel's et al 
# that need restarting
desc "Deploy webgen site" 
deploy.task :restart do
  puts "Using webgen, so nothing to restart" 
end

# also, because we have configured logging via our HTTP 
# server (e.g. nginx, apache etc) we have no need for this 
# part of the deployment phase to we again, just override it
# with an arbitrary message
desc "Override default cap behavior for finalize_update" 
deploy.task :finalize_update do
  puts "We dont need no stinking logs..." 
end

Now, all being well, we need to setup the server to respond to cap:

<code>$ cap deploy:setup</code>

Then we should be able to deploy our project with no problems!

<code>$ cap deploy</code>

I hope this was of use to someone, somewhere, sometime :)

Intergrated support for text messaging from merb

2008-01-05T01:37:00+00:00

Well, I got board - what can I say!

If you have a signup.to account you can enjoy really seamless intergration with merb for sending messages :)

All you need to to send a SMS is write something like this in one of your controllers:

<code>send_sms(ExampleMessenger, 
               :some_method, 
               { :from => "02UK", 
                 :to => "44123456789" })</code>

Thats pretty dang sweet; its also all very MVC, using its own messaging controllers (complete with generators) and ERB files for the message content. Lovely.

If anyone gets use out of it, I would really love to hear about it - download the gems here

Is Rails no longer the flavor of the month?

2008-01-05T00:11:00+00:00

Firstly, I read this blog post by Zed Shaw. Quite frankly I was disgusted that someone who has given so much to our community has been treated so poorly by prominent figures within our very inner circles and the major vendors. I must point out I do not know Zed, Ive never met him, nor will I likely ever meet him, but yet he helped me many times way back in the early days of mongrel, as im sure he did with countless others - and for that, he gets my respects and applause.

His blog posts about the problems with Rails echoed conversations I have had multiple times of late with various other developers and architects. People feel rails is bloated and has a lot of inefficiencies - which is true, and whilst 2.1 is leaner, it could still do with a good going over at fat camp… That sounds bad I admit, but I am not ignorant of the fact that we do have an awful lot to thank Rails (and DHH especially) for - the ideas behind Rails have changed the way the whole development community look at there working practices from start to finish. The shear magnitude of that is mind-blowing.

Personally, Rails has changed the way I look at code, the way I think and more - this blog is even powered by rails. Despite all that, when I build a Rails project now, I cant help but feel, well, a little bit dirty. Initially in my Rails career I loved the way that the bulk of the complexity was abstracted far far away down the rabbit hole - thats not because I was an incompetent coder - it was just new and exciting, and seemed like such a great idea at the time (like so many other things…). Now however, I feel like we need more; more control, more transparency in our frameworks and tools. Less black boxes and more turner-prize type glass ones!

I ask you the reader one thing: Is it time we said farewell to Rails? Or is it a case of the grass is greener on the other side?

Some of you reading this might be thinking that this is all well and good, but what the hell are we going to use to build our projects. Well, I have been putting a lot of time into the Merb framework recently, and it is, categorically, awesome, not to mention thread safe ;) Working with it feels just, well, better. Ezra has crafted a great framework that has taken heed of Rails’ shortcomings, that my friends can only be a good thing…

Anyway, I’ll be posting some Merb tutorials up here soon, so I hope you will give it a try and let me know how you get on.

Happy coding guys, and farewell Rails, its been emotional!

Implementing XMPie ICP with my new ICP extension framework

2007-12-28T11:18:00+00:00

Well its taken a while to put together, but I am finally reaching the point at which its really useable and sweet to work with. I would love to hear peoples thoughts on this - are you already using the DW controls? If so, what are your thoughts on this new style of working? Is it too limited by way of being too techie?

Its important to note that this framework is 100% cross platform any only requires Ruby, and the RubyGems package management; this screen-cast however was created on OSX Leopard.

MAKE SURE YOU VIEW FULL SCREEN TO READ THE TEXT

Enjoy!

MetaProgramming with Ruby

2007-12-22T13:18:00+00:00

Of late of been writing some nice gems makings heavy use of meta programming. If anyone is looking at doing something similar, I would high recommend you check out these resources:

http://www.infoq.com/presentations/metaprogramming-ruby

http://www.rubyplus.org/

The talk about self and its meaning was extremely interesting - its really surprising how flexible it can be in the right hands :)

Finally got hold of new Revel8 KTP boards

2007-12-14T19:13:00+00:00

Well, its been a while, the but the guys at Exclaim Skiboards came through and I know have in my possession a slick pair of Revel8 KTP skiboards!!! Were off to Tamworth snow dome tomorrow to try them out - god I really cant wait! Just look how hot they are….

Updated after snowdome trip….

Well, they rock. Simple. Everyone on the slope kept asking what they were and saying how they were the coolest thing they’d ever seen - it was annoying that people kept calling them “snowblades” tho; they were hastily corrected - no… their “skiboards” you muppet!!

God, im so hooked - bring on Italy!!!!

Give your database some GIS sugar!

2007-12-13T19:38:00+00:00

Well, its time for production now - remember that tutorial way back about building PostGRE on OSX? Well Ive just done it on Ubuntu Server, but from scratch, with all the GIS bells and whistles ;)

Lets tuck in…

First off, rather than me labour the point (anyone familiar with linux will breeze this part) check out this really usefull post - its got all the things you need for doing the actual compilation and install.

There are however a few Ubuntu related “gotchas”. Firstly, after you install GEOS, both ldconfig doesnt yet know about the new libs you’ve just installed at /usr/local/lib. To make it read the new libs you’ve gotta create a new file called ld.so.conf and stick it in /etc - fill it with the following:

<code>/lib/
/usr/lib/
/usr/local/lib/
/usr/X11R6/lib/</code>

Then to make it read in the new config, do this:

<code>/sbin/ldconfig -f /etc/ld.so.conf</code>

To check that its now reading your GEOS libs, run the following, and make sure you see a similar output (depending on the version of GEOS you installed)

<code>$ /sbin/ldconfig -p | grep geos
    libgeos_c.so.1 (libc6,x86-64) => /usr/local/lib/libgeos_c.so.1
    libgeos_c.so (libc6,x86-64) => /usr/local/lib/libgeos_c.so
    libgeos.so.2 (libc6,x86-64) => /usr/lib/libgeos.so.2
    libgeos.so (libc6,x86-64) => /usr/lib/libgeos.so</code>

Another helping sir?

OK - were good to go on that front. The key thing now is that we need to be able to add GIS functions and type handlers to any given database. Provided you set up the ldconfig stuff correctly, doing:

<code>su -- postgres /usr/local/pgsql/bin/psql \
-f /usr/local/pgsql/share/lwpostgis.sql \
-d yourdbnamegoeshere</code>

If this didnt work, you have a problem with your ldconfig! There are fixes out there which advise changing the lwpostgis.sql script, but in my personal opinion, thats not fixing the problem, thats just dirtying a fresh install.

Next, after I had everything up and running, it was puzzling me that there was no init.d script available (the google force was not strong today), I rummaged around in the source I downloaded, and found a startup-scripts dir in the contrib directory, result. Take that script, and bosh it into /etc/init.d and do:

<code>sudo /usr/sbin/update-rc.d postgresql-8.2.5 defaults</code>

Then you have a fully working PostGRE install with GIS sugar!!!

Hopefully this might help someone one day :)

Symbian applications you need in your life

2007-12-10T17:16:00+00:00

Recently I found out that their is a swath of stuff avalible to install on your mobile device if your running Symbian OS.

My personal favorite is Google maps, which works amazingly well - check out an online demo of it here

There is also a GMail app, which works really nicely - More info on that can be found here

Also, for those of you who might want to restart your remote box on the go, there is a mobile SSH client that runs on an mobile with a JVM. Fly over to XK72 to check out MidpSSH client; it even lets you record macros for repeats tasks! Magic!

Launch of XMPie google group

2007-12-06T20:54:00+00:00

Well, its finally happened - an independent, open, and crucially free forum for discussing XMPie technology in a global community.

This is all pretty new but I would love to see some good conversations bubble up on there in the not too distant future - if you would like to sign up get yourself over to this page ASAP!

We are hoping that this will prove a useful and valuable resource for XMPie discussion - I hope to see as many of you on list as possible!!

Wouldn’t you love to be as good as this?

2007-11-04T11:24:00+00:00

I came across this on YouTube - awesome, simply awesome :)

Obligatory post about leopard OS X

2007-11-04T11:19:00+00:00

I know, I know, I know - I just couldnt help myself but I needed to post about Lepoard!

It was a smooth upgrade on the whole, bar some funny issue with the DNS that couldnt resolve some external items; switching to openDNS worked like a charm.

Anyway, I wont bang on about time machine et al as thats been done to death!

How to install PostgreSQL with PostGIS spatial on OSX

2007-10-26T13:32:00+01:00

Recently spatial calculations have fascinated me. Hardly what most people would consider fascinating, but hey!

Most of the modern enterprise (I use that term loosely as someone is bound to comment complaining that some of this list are not ‘enterprise’ enough) RDBMS have spatial GIS extensions. This allows them to conduct exteamly complex calculations about size, posistion and location in a three dimensional way - pretty freaking cool! Common databases that have GIS extensions are:

PostGRE - PostGIS

Oracle - Oracle spatial

DB2 - IBM Spatial

MySQL - MySQL spatial entensions

there are lots more… just google for your specific backend.

Anyway… for this article we’ll focus on PostGRE PostGIS, and running it on OSX.

Step 1

You will need to (download a whole bunch of frameworks from here)http://www.kyngchaos.com/software/unixport/frameworks and run the installer. That will give you some of the base libs and frameworks that PostGIS requires; such as GEOS, GDAL etc etc

Step 2

Add a new user on your system and make sure its short name is postgres - this is the user you will run the database server as. See below from my box:

Step 3

Download the latest PostGRE installer from here. This will load up PostGRE in /usr/local/pgsql; while your there i’d install the startup item so you dont have to load the server manually evertime you restart your mac.

Once installed you need to do some editing of the conf files. For me, I just needed a local development server, so I loaded up /usr/local/pgsql/data/postgresql.conf and changed the line

#listen_addresses = ‘‘**

listen_addresses = ‘‘**

This will ensure that our server binds to all the interfaces the box has. If you want to be more specific, just enter the IP of your machine.

Next, you need to edit the pg_hba.conf file to add a generic host like so:

host all all 0.0.0.0/0 md5

This lets anyone from any IP/subnet connect as any user. I must stress that this was for development purposes so you might want to be a bit more explicit for a production enviroment.

Step 4

Now you have PostGRE installed and configured, run the PostGIS installer - this will PostGIS to your install. We now need to create a new database and add the spatial extensions to it so that our querys will work… but before we can do that we’ll need to enable PL/SQL like so:

<code>CREATE LANGUAGE 'plpgsql' HANDLER plpgsql_call_handler LANCOMPILER 'PL/pgSQL'</code>

Then, from PgAdmin3, open the query tool, and browse to /usr/local/pgsql/share/lwpostgis.sql - run that query, and dont worry about the output (unless its an error!) and refresh the view in PgAdmin3, you should then see the functions list gone from 0 for 600+ - thats all your spatial extensions loaded up ready for use.

That should be it - check out Boston GIS for more on GIS querys and so on - its exteamly complex and out of scope for this article. Happy geocoding people….

Making XMPie work with PostGRE SQL 8.3 backend

2007-10-19T01:39:00+01:00

For all the good things about XMPie, one of my niggles has been the lack of support for PostGRE SQL. I guess thats pretty understanderble; most of there client base use SQL server and other such MS products to some degree. There are times when id really like to use PostGre SQL as the backend and it can be a bit annoying. Anyway…thats another story…. for now were just going to look at implementing the OLEDB conectivity onto the servers to make it talk to our datasource.

Before we begin please note that it is critical to make sure that your PostgreSQL server can accept connections from your host IP. Otherwise you will get an error detailing that windows could not initialize the OLEDB provider. So, for those not familiar with PGSQL, check your postgresql.conf and pg_hba.conf….

Step 1

Ok, firstly you will need to install the OLEDB driver onto the server (and all extension serves where applicible)…

Download the OLEDB driver files from here - its currently in beta so do bear that in mind. Once downloaded, unzip it and copy libpq.dll and PgOleDb.dll to your windows\system32 folder.

Next, to register those new files as sources, go Start > Run and type

regsvr32 PGOLEDB.DLL

All being well you should see a sucsess box - if not, then i guess check your spelling against the DLL :)

Step 2

Now, provided you got your working OLEDB driver up and running in the previous step (be sure to test this with a UDL script) you should be able to add the database into the xmpie dashboard using a dsn something like this:

<code>Provider=PostgreSQL OLE DB Provider;Password=<pass>;
User ID=root;Data Source=127.0.0.1;Location=demo;Extended Properties=&#8221;&#8221;
</code>

I am going to be doing some testing to see if this is any quicker than SQL Server (fingers crossed it will be) - stay tunned readers!

Give me a shout if you dont seem to be able to make this work (for whatever reason)

Im sure Microsoft must be a synonym for crap?

2007-10-15T13:14:00+01:00

I recently have had to do come C++ coding - not my usual play ground i must admit, but never the less, something I needed to do.

My first port of call was to download the Visual C++ installer, which I did. Fine and dandy I thought, it seemed to work ok. The GUI was typical 2007 style, so yes, I couldn’t find a thing.

I then set about doing some coding, only to find that even simple stuff like

<code>#include <windows.h>
</code>

I was getting all manner of errors in the bottom console. I only then found out that without the super duper “Microsoft Platform SDK” you couldn’t actually do sweet fk all. So, to another 500mb download, on top of the painful download and installation of the IDE, this was somewhat insulting.

Microsoft, we the developers and end users of your software urge you to please plug your brain in before making these kinds of choices! I know you say VC++ is meant to only ‘give a taste' for windows C++ development, but if you want that taste to be something other than bitter resentment, make sure your products work properly out of the box!!

Do you like spear wielding spartans?

2007-10-06T14:10:00+01:00

This has to be one of the best films of the year….

Ruby SMS gateway client for sign-up.to

2007-10-05T20:44:00+01:00

Well, I finally got round to getting my own RubyForge project! Its been a while coming now, but I decided to release some of the work ive done using the (SignUpTo)http://sign-up.to SMS gateway as a gem :)

Its still very early stages, as i only needed something rough for the project I was working on, however i put it out anyway - its rough but it does work I assure you! lol

You can check the project page at (signupto.rubyforge.org)http://signupto.rubyforge.org and download the source files (here)http://rubyforge.org/frs/?group_id=4542.

Otherwise, if you wanna install via gems, just do the usual:

sudo gem install signupto

So how do I use it?

For this example, we’ll use rails, at thats probally where people might use it most often… In your boot.rb do

<code>gem 'signupto'</code>

Then require the files in your enviroment.rb

<code>require 'net/sms/signupto'</code>

That will then let you use the classes anywhere in your app :) such as

<code>account = Net::SMS::SignupTo::Account.new(‘YOUR ACCOUNT HASH', ‘CUSTOMER')
message = Net::SMS::SignupTo::Message.new(RECIPIENT_NUMBER,
'NAME OF SENDER (max 12 chars)', 
‘BODY OF MESSAGE, NON-URI encoded')</code>






<code>send = Net::SMS::SignupTo::Send.new(account, message)</code>

The SMS’s should then just fly out the door! Apologies as I know this is a very rushed first post about this, but I have places to go and people to see ;) I’ll be bolstering the unit tests and examples soon. Stay tuned people!!

Fixing random PHP crashes on IIS 6 recorded as PHP memory leaks

2007-09-30T12:50:00+01:00

I recently came accross a serious problem that seems to be relativly un-documented bar other people raising bug tickets on bugs.php.net.

If you are having problems with PHP on IIS where you get a couple page views from your application then you just get a blank screen? If so then read on….

There was nothing in the Event Log for applicatons or system, so I tried re-starting IIS; with no luck. This particular problem seemed to be happening tottaly silently with absolutly no record anywhere (php_error.log et al). I them re-started the whole box, and upon boot up I checked the Event viewer and there was an entry in the application log detailing a memory leak in “php.exe”. So, a memory leak I thought, great…

Trawling the web I then found a document detailing how windows server kills off processes it deems to be at all ‘dangerous’. It does this silently (which is unbeliviably handy) and fitted with the problems I was encountering. Going on the presumption that it was indeed killing the PHP process (or more specifically the worker threads) I turned to the application pool configuration… And broke out alll my PHP apps into a seperate application pool. When they stoped working, I could just recycle the pool and they worked again for two requests. NB: Ive seen other peoples configurations take a couple more than 2 requests so bear that in mind if your having this problem.

So, by now I knew that it was windows killing of the process causing the problems, however I wasnst sure how to fix it.

I looked at the loaded modules in my PHP.ini, and found that removing all non-essential modules solved the problem. I then brought in one by one each PHP module, and found that it was not actually the modules them selves that were leaking, but just the order in which they were loaded causing the problem.

Yes, thats very crap. I couldnt find a more elegant solution to this, and I really hope this helps someone as I must have wasted a day of my life trying to solve this bloody problem! lol

Installing Apache 2.2 on OSX with mod_python

2007-09-23T16:18:00+01:00

Following on from setting up Django on your local system for dev purposes, I thought id just do an article about how to set up apache to use mod_python for when you want to roll your Django app out to the big bad world.

Before you start, you will NEED to have completed the other post about installing Django, as you need to have the mac python installer in and configured properly for this to work.

Step 1

First of all we’ve got to get Apache 2.2 working, as Apple only ship 1.3 with OSX (I wonder if this will change in Lepord? Lets hope so!). You can download the source here.

Extract the archive and cd into the new directory. We need to configure it with some options so that we have proxying (which you might want for rails etc) and just some other candy you might find usefull at a later date.

Set the CFLAGS:

export CFLAGS=”-arch i386″

Run this configure command:

./configure \
  --prefix=/Library/Apache-2.2.6 \
  --enable-so \
  --enable-mods-shared=all \
  --with-mpm=prefork \
  --enable-dav \
  --enable-cache \
  --enable-proxy \
  --enable-shared \
  --disable-static \
  --disable-unique-id \
  --disableipv6 \
  --enable-logio \
  --enable-deflate \
  --with-ldap \
  --with-ldap-include=/usr/include \
  --with-ldap-lib=/usr/lib 
  --with-included-apr 
  --enable-ldap \
  --enable-auth-ldap \
  --enable-cgi 
  --enable-cgid \
  --enable-suexec

Let it do its business then run the usual:

make

then

make install (NB: we dont use sudo here as that way apache is compiled with permisions that let you edit the conf files and so on without needing to be an admin which is just ‘nicer’ for local dev)

Step 2

Configure your httpd.conf file as you see fit. I only really felt the need to change the following:

Change user and group to your user name (or any other one you want) Change ServerAdmin email to an appropriate one Uncomment “Include conf/extra/httpd-mpm.conf” Uncomment “Include conf/extra/httpd-default.conf” Add “NameVirtualHost ” for virtual hosting

My personal preference is then to have an “applications” folder in the apache conf dir, for which I then add a directive to httpd.conf to load in per-application virtual hosting configurations. It just keeps it nice and clean that way.

mkdir /Library/Apache-2.2.6/conf/applications

Include conf/applications in httpd.conf (in your httpd.conf)

Step 3

If you havent already, grab mod_python from here. Extract, it and cd into the directory.

You’ll then need to configure it with the following command:

<code>./configure --with-apxs=/Library/Apache-2.2.6/bin/apxs \
  --with-python=/Library/Frameworks/Python.framework/Versions/Current/bin/python</code>

Then do make and sudo make install

Provided you got no errors upon compilation, all should be well. You just need to add another line to your httpd conf:

LoadModule python_module modules/mod_python.so

Step 4

Poor yourself a nice cuppa!

Running Django on OSX

2007-09-22T18:51:00+01:00

No doubt you’ve heard about the new python web framework thats making some waves at the moment - Django.

As with most nix developers, im running OSX (10.4), and wanted to get a working Django dev environment function to have a play around with it and found that a needed to mess around with a few things to get it working, so I thought id post them up here in case anyone else found them usefull :)

Step 1

Download the python-mac installer from here and run it. That will give you a working version of the latest python bin.

Step 2

Checkout the latest version of Django source like so

cd /usr/local

svn co http://code.djangoproject.com/svn/django/trunk/ django

Step 3

If you used the python installer for mac, then the below will work, otherwise, you’ll need to just tweak the path’s a bit:

ln -s /usr/local/django/django /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/django

Step 4

Add the django source to your path:

PATH=/usr/local/django/django/bin:$PATH

Then that should be it :) However, if your a touch on the forgetfull side when using the terminal, it can be annoying having to keep putting the .py on the end of the django commands, so we’ll just quickly symlink them so if we forget to add .py, it wont slap us with a kipper

cd /usr/local/django/django

ln -s django-admin.py django-admin

Then your all done! Enjoy Django :)

Very random video...

2007-09-22T14:21:00+01:00

Everyone loves a random video…

Running Pound reverse proxy on windows server 2003

2007-09-22T12:50:00+01:00

Pound is the unix reverse proxy software popularized by Rails deployment over the past 8-12 months. Pound is a unix tool, and was never meant to run under windows - it is however possible to make it run under windows if you package it up with some key DLL files from cygwin :)

I must stress here that this is not necessarily the best thing to be doing, this post is more about how to do it if you really want too rather than a “do this, I really recommend it” type post. There are also some issues with pound and its behavior with X-FORWARDED_FOR headers, but i wont bang on about that here; refer to google for more info. Anyway… back to the post…..

Take the latest pound distro from here then compile it from within Cygwin like so

./configure –without-ssl –disable-log –disable-dynscale

then

make all

Then you should be left with a working binary of pound within your build directory. In order to then make this work in a standalone fashion (I use the term standalone loosely as its still using the cygwin environment) you need to package several DLL files from cygwin:

cygssl-0.9.8.dll (required)

cygcrypto-0.9.8.dll (required)

cygwin1.dll (required)

pound.exe (required, should be fairly obvious!)

cygrunsrv.exe (optional, include only if you want to install pound as a service)

with these files you can then run pound, and even install it as a service on the host machine. If you do want to install the service, run the following command:

cygrunsrv –install Pound –path X:\applications\pound-2.3.2\bin\pound.exe –args “-f X:\applications\pound-2.3.2\config\pound.cfg” –stdout X:\applications\pound-2.3.2\log\pound.log –stderr X:\applications\pound-2.3.2\log\pound.log

Where “X:\applications\pound-2.3.2” is the real path to where ever it is on the file system you chose to install.

If you don’t want to compile your own, then you can download my version of pound 2.3.2 for windows here. It goes without saying I take no responsibility for any problems you might run into by using my compiled version; it worked for me thats all I can say!

Making IIS play nice with Apache on windows server 2003

2007-09-21T14:55:00+01:00

Now, being a unix man through and through, I do see the irony that my first blog post is about winows; such is life!

Anyway…

Typically IIS and Apache do not work well together at all, as IIS hogs all IPs (both physical and ARP) on any given win 2003 machine which is why when trying to run anything else on port 80 on any IP can be a total nightmare.

I recently had to do an install on a win 2003 box where the client wanted to support some of their legacy ASP applications, PHP, but then allowing scope in the future to run Rails, Seam and Django applications. This means running:

* Apache with mod_python (for Django)

* Apache proxying to mongrel (for Rails)

* JBoss for Seam (EJB configuration)

* IIS for ASP/ASP.NET

IIS has no real ability to proxy to other backends (which in this instance we need due to the rails requirment) in the way Apache 2.2 does, so I chose to run Apache out front binding to one of my avalible 5 on this machine. Apache then uses proxy pass (or proxy balencer for rails) to pass PHP and ASP requests to IIS. No doubt there are people out there who would have chosen to run PHP on Apache, but due to the way the legacy applications were written, PHP and IIS were already installed so it made sense to leave it “as is” - no need to make more work for myself, especially as i only had one night to do the install!

Step 1

IIS however requires a bit of wrangling before we can get on with the install of Apache…Make sure you have your win 2003 install discs with you; as you’ll need to install the support tools (found in support > tools on the cd). It is however possible to do this without the support tools, but it involves hacking around in the registry. Once you have the support tools installed, were going to be making some changes to let IIS know what exact IP address it can bind too. If you have the support tools installed you can the use the httpcfg tool like so (note: if using RDP you might need to logout then login to get support tools on your PATH)

httpcfg set iplisten -i 555.555.555.555 (where this is your IP address)

If you do not have the support tools (or your just too lazy to dig out that disc from the depths of the server room) you can edit the following registry key:

HKEY_LOCAL_MACHINE/SYSTEM/CurrentControlSet/Services/HTTP/Paramaters in there is a paramater called ListenOnlyList from which you can edit with the list of IP’s IIS should listen to.

Once youve done that, restart IIS as usual from the MMC.

Then you can do a netstat -ano to see that IIS only binds to the IP you set. Simple.

Step 2

Next up you need to download apache. I went with apache 2.2.4 without SSL, which i grabbed from here

Run the installer, and make sure you choose on port 80 as a server for all users when installing.

Once apache is installed, you’ll need to set up port listening and IP address binding. This is a well discussed topic in the apache manuals so I wont labour the point here. See the apache docs.

I then set up some virtual hosting to hand my requests off to IIS. Granted, I could have done this with rewrite rules for .asp extensions, but I chose to proxy it on domains, due to my particular circumstances. Heres what my virtual host looks like:

<VirtualHost *:80>
  ServerName www.domain.co.uk
  ServerAlias domain.co.uk
  ProxyPreserveHost On
  ProxyPass / http://172.16.1.11:3000/
</VirtualHost>

(Where my site definition within IIS was running on port 3000)

Step 3

I then downloaded Python and installed it. And then I followed the very good instructions on setting up mod_python on google groups

Step 4

Do the usual rails install bits and bobs - this is so well document I wont pollute the blogsphere any more :)

Step 5

Java time…. Download and run the JEMS installer. At this point we only want to install JBoss on the box with an EJB configuration; it will take a couple more posts to go into the configuration required to get JBoss Seam up and running so we shant discuss that here.

Summary

Thats pretty much it. Its not too difficult, you can proxy off to any service you want to (within reason) and you can have all the latest and greatest web serving technologys happily co-exsisting!