Friday, November 30, 2012

On-Demand Java container with Jetty and xinetd

Since the addition of Java as supported language for Google AppEngine, I have been intrigued by their on-demand approach for JVMs.  Traditional JEE containers are always on and ready to receive your requests.  This is fine if you have a steady flow of requests coming in.  Otherwise the container is sitting there consuming resources (cpu, memory) while waiting for requests to come in.
In large scale deployments where efficient use of resources is key, it's not a good idea to have a large amount of application containers idling.  That is why Google has implemented a mechanism in which JVM's are started on-demand when the first request comes in.  When traffic dials back to zero, the JVM is shut down again until a new burst of requests comes in.

On-demand launch of containers can only work if the container can be started in a minimal amount of time.  Remember that the launch is triggered by an initial request.  If we don't start the container fast enough a user will be waiting for his browser to display results in vane, and in worst case scenario the connection will time out on the client side.
Our choice of application containers is hereby significantly limited.  Only few containers will launch fast enough to provide acceptable response times for the initial request.  Jetty is a good choice and it's the container that Google uses for their AppEngine.

Let's look at a basic on-demand Jetty using xinetd on Linux.  I'm assuming Redhat Linux 6, but this approach should work on most linux flavors.  You might have to tweak the configuration to match your distro.
First we tell xinetd about our service by adding a configuration file in the /etc/xinetd.d/ directory:
service jetty
    flags       = IPv4
    disable     = no

    id          = jetty
    type        = UNLISTED
    wait        = yes
    socket_type = stream

    user        = root
    group       = root
    port        = 18080

    server      = /opt/jetty8/bin/

This tells xinetd to use the script at /opt/jetty8/bin/ to handle incoming requests.  This is where we'll plug in Jetty.  One of the important lines in the xinetd configuration file for Jetty is

wait = yes

This will ensure that Jetty gets a ServerSocketChannel which it can use to processes subsequent requests to the port on which xinetd was listening on.  The general flows goes like this:
  1. xinetd sees an incoming request on port 18080 (in our case)
  2. It launches /opt/jetty8/bin/
  3. A ServerSocketChannel is available to Jetty to process the request
  1. As long as Jetty is running, it has control over the ServerSocket and can continue to process requests.

Since JDK 1.5, there's a method System.inheritedChannel which makes this integration with xinetd and the exchange of socket handlers possible.  Note that when you configure the xinetd service with wait=no, then xinetd will quickly hand a regular SocketChannel to the server process and expects it to handle that single request.  This allows for multi-threaded handling of requests if the container wouldn't support it.  This would provide more typical CGI behavior.  Support for this feature in Jetty is discussed in some more detail in the Codehaus Jira.
In our case, it's desired that Jetty continues running after the initial request since it can handle subsequent requests without the overhead of startup.  To save resources, a mechanism should be built in that stops Jetty after a period of idle time.

Some additional notes:
  • This works optimally if startup times are reduced to a minimum.  Applications you deploy in Jetty will have an impact on startup times.  Lightweight apps are key: accept payload provide a response quickly and defer other work to backend processing systems.
  • Note that all requests are sent to the same Jetty container with the same applications deployed.  I haven't taken the time to implement request routing and differentiating between Jetty instances with different apps deployed. 
  • I'm not yet convinced that xinetd is the best way to implement on-demand Java containers.  For this simple use case it seems to work beautifully.  However more research is needed to find a good way to make this work in an enterprise environment, with thousands of applications.

I've created a script which will allow you to test this mechanism quickly.  View the content here:

You can get started by launching the following command on your RHEL6 linux box (as root!):

Thursday, January 19, 2012

Thoughts on Amazon's DynamoDB

Amazon just announced their latest addition to  the AWS suite.  DynamoDB is a "fast, highly reliable and cost-effective NoSQL database service designed for internet scale applications".  I haven't had a chance try out to the service for myself, but while reading Werner Vogels blog about it, some key statements caught my attention.
 Amazon DynamoDB stores data on Solid State Drives (SSDs) and replicates it synchronously across multiple AWS Availability Zones in an AWS Region to provide built-in high availability and data durability.
SSD has become popular to speed up storage performance.  I've seen instances where SSD's are just as a caching tier in front of a somewhat traditional storage solution.  Most notable example here would be Facebook's Flashcache, a linux module which provides "a simple write back persistent block cache designed to accelerate reads and writes from slower rotational media by caching data in SSD's".  Amazon however chose to use SSD as the primary storage. Obviously this is calculate into it storage pricing, currently $1 USD per GB per month. 
What I found most interesting about this architecture detail, is that Amazon is capable of designing flexible services while still allowing themselves to make specific hardware choices, and create automation processes to support those choices.  Typically today, it is popular to standardize on commodity hardware which is relatively easy to manage and created services on top.  Having the flexibility to pick specialized hardware for a certain end user service allows for higher quality services.
 Each service encapsulates its own data and presents a hardened API for others to use.  Most importantly, direct database access to the data from outside its respective service is not allowed. This architectural pattern was a response to the scaling challenges that had challenged through its first 5 years, when direct database access was one of the major bottlenecks in scaling and operating the business.
Encapsulation is one of those things that everyone will agree with in discussion.  However, in practice it seems to be a bit more challenging.  There are excuses everywhere for getting an exception and creating tightly couple services.  Performance is often used as excuse, or just pure laziness.
Designing encapsulated services and orchestrating them in a true service oriented architecture requires discipline and planning.  Even more, I'm of the opinion that organizations should be structured around such coarse grained services.  People in those service teams should identify them selves with the services, and pride themselves in availability numbers and adoption.  It's much more than a technical decision.
Amazon's success here is good evidence that the required investment pays off.
 As we spoke to many senior engineers and service owners, we saw a clear pattern start to emerge in their explanations of why they didn't adopt Dynamo more broadly: while Dynamo gave them a system that met their reliability, performance, and scalability needs, it did nothing to reduce the operational complexity of running large database systems. …  Dynamo might have been the best technology in the world at the time but it was still software you had to run yourself. And nobody wanted to learn how to do that if they didn’t have to. Ultimately, developers wanted a service.
 The whole cloud business is sometimes reduced to being just the next hype and a lot of hot air.  However, the art of hiding complexity while creating large scale, distributed solutions is in my opinion exactly what cloud is about.  Yes, we might have been doing that to a certain degree well before 'cloud' emerged, but hey… now we have a name for it.
There's a lesson in here as well.  Since the technology allows for it, it is tempting to create private IaaS, PaaS and cloud-like solutions to be used within enterprises.  However, if those services lack a good usability strategy,  it will either prevent adoption or your operations teams will get bombarded with support tickets from people who don't understand how to use your service.  So the lesson is clear:  ease of use is equally important than delivering a high quality service.  Your users want to create business apps, and you can't expect them to be experts in the service you just created.
Consider usability as the (self-)marketing for your service.  A product with a lousy marketing strategy is often doomed to fail.
 Throughput reservations are elastic, so customers can increase or decrease the throughput capacity of a table on-demand
Throughput needs to be reserved to get the best results.  This means that as an end user, you need to have a pretty good idea of how you're going to be using the service, and what through-put you expect.  This may sound logical, but often applications are created and deployed without a good understanding of how they'll be used, or how busy they will be.
The above statement is an argument for doing your homework.  Large scale applications requires a lot of upfront analysis and planning, maybe even traffic simulations to understand what your infrastructure needs are going to be.  This reminds me of another recent article with a great punch line:  "Cloud is complex - deal with it"