2009/09/07

Performance testing: Load generator

Introduction

Performance testing is a wide term, so let me be more specific please: performance testing of a message oriented middleware from the client's point of view. A message oriented middleware can be an Enterprise Service Bus for instance. We tried several implementations - JBoss ESB, WSO2 ESB, and Mule ESB.

What does the "testing from the client's point of view" mean? It means that we don't care about internal processes of the ESB. We just care how long it takes to process our request. It reminds of integration testing in the classical V-Model.

Problem Definition

We were working in a team on the problem and after trying several load generators, we found out that there is none that is generic enough. We wanted to send various types of messages (e.g. JMS, SOAP, files...) in different ways (send all messages at once, find the maximum sustainable speed, run a long-term test...). Apache JMeter approaches the ideal but is still missing some features. Moreover, we experienced high results jitter over time. So we decided to write our own lightweight, easy-to-use load generator. Here are some basic requirements:

  • Stable results.
  • There must be a constant message inflow while received messages are being processed.
  • Messages must be generated by many concurrent clients.
  • Clients must run on remote machines not to influence results.
  • Speed should be measured on the client side for us not to affect the server.

Solution

Message Sender

First, there are different types of messages. So let's have an interface of a component that just sends a particular message.

public interface MessageSender {
  public void setProperty(String prop, String value);
  public void init(String address) throws Exception;
  public void close();
  public Serializable send(Serializable message, 
    Map properties) throws Exception;
  public MessageSender clone();
}

The component should be able to get configured with standard property/value pairs, initialize a connection to a given address, send a serializable message (e.g. a String), and close the connection. Moreover, it can clone itself for us not to have to configure each instance.

There are two important decision points. First, do you want MessageSender to be reusable - to be able to send more than one message? When you are careful enough, you should be able to achieve this. At least, we did not experience any problems with that.

Second, do you want your senders to be thread safe? This is a completely different situation, which is really hard to achieve and keep high performance. An example is sending of JMS messages. JMS session is not thread safe but it takes relatively long time to initialize. You might want to extend the interface and create a ThreadSafeMessageSender.

An example of an HTTPSender is shown below.

public class HTTPSender implements MessageSender {
  private URL url;
  private String method = "POST";

  public void setProperty(String prop, String value) {
    if ("method".equals(prop)) {
      method = value;
    }
  }

  public void init(String address) throws Exception {
    url = new URL(address);
  }

  public void close() {
  }

  public Serializable send(Serializable message, Map properties) throws Exception {
    HttpURLConnection rc = (HttpURLConnection) url.openConnection();
    
    // ... standard HttpUrlConnection usage ...

    return response;
  }

  @Override
  public MessageSender clone() {
    ...
  }
}

Message Generator

Next, there must be a message generator that uses message senders to generate messages in a given way.

public abstract class MessageGenerator {
  protected MessageSender sender;
  protected int threads = 1;

  public void setProperty(String property, String value) {
    if ("threads".equals(property)) {
      threads = Integer.valueOf(value);
    }
  }

  public void init(String address, MessageSender sender) throws Exception {
    this.sender = sender;
    this.sender.init(address);
  }

  public void close() {
    sender.close();
  }

  public abstract void generate(Serializable message, int count) throws Exception;
}

As you can see, an already configured instance of MessageSender must be passed to the MessageGenerator in the init() method. It is not a constructor because some configuration of the generator might be performed first in its descendants. This abstract generator supposes that the message sender is thread safe, because it uses just a single instance to pass all the messages through. Descendants might want to clone this instance in case it were not thread safe.

We used several new features of Java 6 to create a message generator that just sends all the messages to the server in a given number of threads. Let's first examine a sender task that is created per each thread. It uses shared AtomicInteger to keep track of actually sent messages.

public class SenderTask implements Runnable {
  private MessageSender sender; // shared sender
  private Serializable message; // message to send
  private AtomicInteger counter; // shared message counter
  private int count; //  number of messages for this thread

  public SenderTask(AtomicInteger counter, MessageSender sender, 
      String address, Serializable message, int count) 
      throws Exception {
    this.count = count;
    this.sender = sender;
    this.message = message;
    this.counter = counter;

    // expect sender's uninitialized clone
    this.sender.init(address);
  }

  @Override
  public void run() {
    try {
      for (int i = 0; i < count; i++) {
        sender.send(message, null);
        counter.incrementAndGet();
      }
    } catch (Exception e) {
      log.error(e);
    } finally {
      sender.close();
    }
  }
}

In the next listing, there is an implementation of the generate() method. No speed measurement code is included but the main goal here was to show the ExecutorService's usage.

@Override
public void generate(Serializable message, int count) 
    throws Exception {
  List<SenderTask> taskList = 
    new ArrayList<SenderTask>(threads);

  // Let tasks to initialize senders
  for (int i = 0; i < threads; i++) {
    taskList.add(new SenderTask(counter, sender.clone(), 
      address, message, perThreadMessageCount);
  }

  // Submit all tasks for completion
  ExecutorService es = Executors.newFixedThreadPool(threads);
  for (int i = 0; i < threads; i++) {
    es.submit(taskList.get(i));
  }

  // Wait for termination
  es.shutdown();
  boolean terminated = false;
  while (!terminated) {
    // output current state
    // take some sleep
    // updated 'terminated' variable
  }
}

As you can see, clones of the original sender are created for us not to have to take care of the thread safeness. Individual SenderTask instances are created in advance for all sender clones to get initialized. Then the instances are put in an ExecutorService and executed. While we are waiting for all threads to finish, we can periodically output current and overall throughput for instance.

Conclusion

This concept already demonstrated its value in real world scenarios. We were able to write various message senders and generators easily and quickly. Even though it may look simple, there are some factors you must take into consideration.

First is the warm up period of the server. You could either run the test several times and ignore a couple of first runs, or extend the generator to throw in some more messages and do not count them.

Second, always make sure that the client is able to generate messages several times faster than the server is able to process. Otherwise, you measure the client's performance, which is something you do not want probably.

WSDL 2.0: Quick reference

I wrote this blog entry mainly for me - to beat WSDL structure in my mind in a clean and nice way. Hopefully, it will help some others as well.

I would like to introduce changes between WSDL 1.1 and 2.0, and to go through the individual parts of the new WSDL version. This blog entry is based on several other on-line resources as well as books.

Introduction

WSDL stands for Web Services Description Language. It is a document written in XML. The document describes a Web service, specifies the location of the service, and the operations (or methods) the service exposes.

WSDL 2.0 was first developed as WSDL 1.2 but was renamed because of its substantial differences. Some major changes are:

  • Adding further semantics to the description language.
  • Removal of message constructs. These are specified using the XML schema type system in the types element.
  • No support for operator overloading.
  • PortTypes renamed to interfaces. Support for interface inheritance is achieved by using the extends attribute in the interface element.
  • Ports renamed to endpoints.

Conceptual Model

The description consists of two parts. In the abstract part, WSDL describes a web service in:

  • Messages it sends and receives using a type system (typically W3C XML Schema).
  • Message exchange patterns that define the sequence and cardinality of messages.
  • Operations that associates message exchange patterns with one or more messages.
  • Interfaces group these operations in a transport and wire independent manner.

In the concrete part of the description:

  • Bindings specify the transport and wire format for interfaces.
  • A service endpoint associates network address with a binding.
  • A service groups the endpoints that implement a common interface.

Structure

Following is a basic WSDL structure

<definitions targetNamespace="xs:anyURI">
<documentation /> ?
[<import /> | <include /> ] *
<types /> ?
[<interface /> | <binding /> | <service /> ] *
</definitions>

Definitions

The definitions element serves as a container.

<definitions name="StockQuote"
  targetNamespace="http://example.com/stockquote/definitions"
  xmlns:tns="http://example.com/stockquote/definitions"
  xmlns:xsd1="http://example.com/stockquote/schemas"
  xmlns:soap="http://www.w3.org/2003/11/wsdl/soap12"
  xmlns="http://www.w3.org/2003/11/wsdl">

Include

The include element helps to modularize the web service descriptions. Included documents must have the same target namespace.

Import

The concept behind the import element is very similar to that of include element, except that the imported WSDL can be in different target namespaces.

<import namespace="http://example.com/stockquote/schemas"
  location="http://example.com/stockquote/stockquoteV20.xsd"/>

Types

The types element defines the data types used by the exchanged messages. WSDL uses W3C XML Schema as its preferred schema language.

The following example refers to an imported XSD.

<types>
  <schema 
    targetNamespace="http://example.com/stockquote/definitions">
    <element name="GetLastTradePriceInput" 
      type="xsd1:TradePriceRequest"/>
    <element name="GetLastTradePriceOutput" 
      type="xsd1:TradePrice"/>
  </schema>
</types>

Interface

An interface element encloses a named set of abstract operations and the abstract messages. It can extend one or more other interfaces. Interfaces are referred to by QName in other components.

<interface name="StockQuoteInterface">
  <operation name="GetLastTradePrice" pattern="http://www.w3.org/2003/11/wsdl/in-out">
    <input message="tns:GetLastTradePriceInput"/>
    <output message="tns:GetLastTradePriceOutput"/>
  </operation>
</interface>

Binding

The binding element defines the underlying transport and wire format for messages. Each binding references to an interface.

<binding name="StockQuoteSoapBinding" 
  interface="defs:StockQuoteInterface">
  <soap:binding protocol="http://www.w3.org/2003/11/wsdl/http"/>
  <operation name="GetLastTradePrice">
    <soap:operation 
      soapAction="http://example.com/GetLastTradePrice"/>
    <input>
      <soap:body/>
    </input>
    <output>
      <soap:body/>
    </output>
  </operation>
</binding>

Service

A service element describes a set of endpoints which refer to a single network address for a binding. All other protocol specific information is contained in the binding.

<service name="StockQuoteService">
  <documentation>My stock quote service</documentation>
  <endpoint name="StockQuoteEndPoint" 
    binding="tns:StockQuoteSoapBinding">
    <soap:address location="http://example.com/stockquote"/>
  </endpoint>
</service>

Conclusion

Despite of the fact that the WSDL format might seem over-engineered to you (because it probably is), you should now be aware of its content and the meaning of individual elements.

References

[1] Article on WSDL at www.xml.com
[2] W3C School WSDL Tutorial
[3] Web Service Platform Architecture on InformIT.com

. .