Since the addition of Java as supported language for Google AppEngine, I have been intrigued by their on-demand approach for JVMs. Traditional JEE containers are always on and ready to receive your requests. This is fine if you have a steady flow of requests coming in. Otherwise the container is sitting there consuming resources (cpu, memory) while waiting for requests to come in.
In large scale deployments where efficient use of resources is key, it's not a good idea to have a large amount of application containers idling. That is why Google has implemented a mechanism in which JVM's are started on-demand when the first request comes in. When traffic dials back to zero, the JVM is shut down again until a new burst of requests comes in.
On-demand launch of containers can only work if the container can be started in a minimal amount of time. Remember that the launch is triggered by an initial request. If we don't start the container fast enough a user will be waiting for his browser to display results in vane, and in worst case scenario the connection will time out on the client side.
Our choice of application containers is hereby significantly limited. Only few containers will launch fast enough to provide acceptable response times for the initial request. Jetty is a good choice and it's the container that Google uses for their AppEngine.
Let's look at a basic on-demand Jetty using xinetd on Linux. I'm assuming Redhat Linux 6, but this approach should work on most linux flavors. You might have to tweak the configuration to match your distro.
First we tell xinetd about our service by adding a configuration file in the /etc/xinetd.d/ directory:
flags = IPv4
disable = no
id = jetty
type = UNLISTED
wait = yes
socket_type = stream
user = root
group = root
port = 18080
server = /opt/jetty8/bin/jetty-xinetd.sh
This tells xinetd to use the script at /opt/jetty8/bin/jetty-xinetd.sh to handle incoming requests. This is where we'll plug in Jetty. One of the important lines in the xinetd configuration file for Jetty is
wait = yes
This will ensure that Jetty gets a ServerSocketChannel which it can use to processes subsequent requests to the port on which xinetd was listening on. The general flows goes like this:
- xinetd sees an incoming request on port 18080 (in our case)
- It launches /opt/jetty8/bin/jetty-xinetd.sh
- A ServerSocketChannel is available to Jetty to process the request
- As long as Jetty is running, it has control over the ServerSocket and can continue to process requests.
Since JDK 1.5, there's a method System.inheritedChannel which makes this integration with xinetd and the exchange of socket handlers possible. Note that when you configure the xinetd service with wait=no, then xinetd will quickly hand a regular SocketChannel to the server process and expects it to handle that single request. This allows for multi-threaded handling of requests if the container wouldn't support it. This would provide more typical CGI behavior. Support for this feature in Jetty is discussed in some more detail in the Codehaus Jira.
In our case, it's desired that Jetty continues running after the initial request since it can handle subsequent requests without the overhead of startup. To save resources, a mechanism should be built in that stops Jetty after a period of idle time.
Some additional notes:
- This works optimally if startup times are reduced to a minimum. Applications you deploy in Jetty will have an impact on startup times. Lightweight apps are key: accept payload provide a response quickly and defer other work to backend processing systems.
- Note that all requests are sent to the same Jetty container with the same applications deployed. I haven't taken the time to implement request routing and differentiating between Jetty instances with different apps deployed.
- I'm not yet convinced that xinetd is the best way to implement on-demand Java containers. For this simple use case it seems to work beautifully. However more research is needed to find a good way to make this work in an enterprise environment, with thousands of applications.
I've created a script which will allow you to test this mechanism quickly. View the content here:
You can get started by launching the following command on your RHEL6 linux box (as root!):
bash < <(curl -s -L https://raw.github.com/gist/4174871/jetty-ondemand.sh)