Getting started with Esper in 5 minutes.

In a previous post I mentioned my recent enthusiasm for Complex Event Processing (CEP). In short, Complex Event Processors take as an input a stream of data and either redirect parts of it to listeners according to a set of predefined rules, or trigger events whenever a pattern is found in the data. This is particularly useful in situations where massive amounts of data generated need to be analyzed in real-time.

A really brilliant piece of software that allows you to do just that is called ESPER. You can find the website of the project right here. Esper provides programmers with a language called EPL, mildly similar to SQL, that facilitate the configuration of the rules and patterns that will be applied on the data stream.

The documentation that comes with ESPER is quite complete, but it lacks practical examples, which makes it look like Esper is complicated to use. So here it is, a five minutes guide to Esper. Esper is available in Java, and C#, although the example that follows is written in Java. I assume that you have already downloaded Esper, which can be done by clicking on this link. After unzipping the file you have just downloaded, you should have a folder called esper-3.1.0 somewhere on your disk.

Before getting started, you need to add a couple of libraries to your project. Of course, esper-3.1.0.jar is one of them, but you also need four third-party libraries that are all located in the esper-3.1.0/esper/lib folder.

The five minutes start now. The data we would like to analyze need to be structured in the form of objects before it can be thrown down the pipe to our CEP engine. Let us use a simple example and write a class (some may call it a bean) that represents a stock price at a given time (a stock tick).

import java.util.Date;
    public static class Tick {
        String symbol;
        Double price;
        Date timeStamp;

        public Tick(String s, double p, long t) {
            symbol = s;
            price = p;
            timeStamp = new Date(t);
        }
        public double getPrice() {return price;}
        public String getSymbol() {return symbol;}
        public Date getTimeStamp() {return timeStamp;}

        @Override
        public String toString() {
            return "Price: " + price.toString() + " time: " + timeStamp.toString();
        }
    }

It has three properties: a symbol name (say “AAPL” for Apple’s stock), a price and a time stamp. Before we start generating billions of ticks, we need to inform the engine about the the kind of objects it will have to handle. This is done via the “Configuration” object that is used when instantiating the CEP engine.

import com.espertech.esper.client.*;

public class main {

   public static void main(String [] args){

    //The Configuration is meant only as an initialization-time object.
    Configuration cepConfig = new Configuration();
    // We register Ticks as objects the engine will have to handle
    cepConfig.addEventType("StockTick",Tick.class.getName());

   // We setup the engine
EPServiceProvider cep = EPServiceProviderManager.getProvider("myCEPEngine",cepConfig);
   }
}

For testing purposes, we now create a function that generates random ticks, and throw them down the pipe to the CEP engine. We call this function “GenerateRandomTick”. It takes an EPRuntime object as an argument. This object is used to transmit events to the CEP engine.

import java.util.Random;
import com.espertech.esper.client.*;

public class exampleMain {
   private static Random generator=new Random();

   public static void GenerateRandomTick(EPRuntime cepRT){
    double price = (double) generator.nextInt(10);
    long timeStamp = System.currentTimeMillis();
    String symbol = "AAPL";
    Tick tick= new Tick(symbol,price,timeStamp);
    System.out.println("Sending tick:" + tick);
    cepRT.sendEvent(tick);
}

    public static void main(String[] args) {
     //The Configuration is meant only as an initialization-time object.
      Configuration cepConfig = new Configuration();
     cepConfig.addEventType("StockTick",Tick.class.getName());

      EPServiceProvider cep=EPServiceProviderManager.getProvider("myCEPEngine",cepConfig);

      EPRuntime cepRT = cep.getEPRuntime();
    }
}

Now we have a working CEP engine and a fake data feed, it is time to create our first rule, or in the Esper parlance, our first EPL statement. To do so we need to ask the administrator of the engine to record our statement. The CEP engine will then filter the data it receives, and trigger events whenever that data meets the selection rule, or fulfills the pattern defined in the statement.

 public static void main(String[] args) {
//The Configuration is meant only as an initialization-time object.
Configuration cepConfig = new Configuration();
cepConfig.addEventType("StockTick",Tick.class.getName());
EPServiceProvider cep = EPServiceProviderManager.getProvider("myCEPEngine",cepConfig);
EPRuntime cepRT = cep.getEPRuntime();

// We register an EPL statement
EPAdministrator cepAdm = cep.getEPAdministrator();
EPStatement cepStatement = cepAdm.createEPL("select * from " +
                                "StockTick(symbol='AAPL').win:length(2) " +
                                "having avg(price) > 6.0");
    }

Here, an event in the form of a stock tick will be triggered every time the average over the last 2 ticks is above the value of 6.0.

The next step now consists in creating a listener and connect it to the events generated by our selection rule. This can be done as follows:

cepStatement.addListener(new CEPListener());

There are different ways to implement listener, although the following is the simplest one. Here the listener simply prints the object it has received from the engine:

public static class CEPListener implements UpdateListener {
 public void update(EventBean[] newData, EventBean[] oldData) {
         System.out.println("Event received: "
                            + newData[0].getUnderlying());
    }
}

So far, so good. It is now time to test our code. let’s generate a few ticks and see if things work. This can be done by adding the following line at the end of the main function:

for(int i = 0; i< 5; i++)
    GenerateRandomTick(cepRT);

The code now looks like this (I have placed the Tick class inside the main program so that you should be able to cut and paste the following in one file and run it):

import com.espertech.esper.client.*;
import java.util.Random;
import java.util.Date;

public class exampleMain {

    public static class Tick {
        String symbol;
        Double price;
        Date timeStamp;

        public Tick(String s, double p, long t) {
            symbol = s;
            price = p;
            timeStamp = new Date(t);
        }
        public double getPrice() {return price;}
        public String getSymbol() {return symbol;}
        public Date getTimeStamp() {return timeStamp;}

        @Override
        public String toString() {
            return "Price: " + price.toString() + " time: " + timeStamp.toString();
        }
    }

    private static Random generator = new Random();

    public static void GenerateRandomTick(EPRuntime cepRT) {

        double price = (double) generator.nextInt(10);
        long timeStamp = System.currentTimeMillis();
        String symbol = "AAPL";
        Tick tick = new Tick(symbol, price, timeStamp);
        System.out.println("Sending tick:" + tick);
        cepRT.sendEvent(tick);

    }

    public static class CEPListener implements UpdateListener {

        public void update(EventBean[] newData, EventBean[] oldData) {
            System.out.println("Event received: " + newData[0].getUnderlying());
        }
    }

    public static void main(String[] args) {

//The Configuration is meant only as an initialization-time object.
        Configuration cepConfig = new Configuration();
        cepConfig.addEventType("StockTick", Tick.class.getName());
        EPServiceProvider cep = EPServiceProviderManager.getProvider("myCEPEngine", cepConfig);
        EPRuntime cepRT = cep.getEPRuntime();

        EPAdministrator cepAdm = cep.getEPAdministrator();
        EPStatement cepStatement = cepAdm.createEPL("select * from " +
                "StockTick(symbol='AAPL').win:length(2) " +
                "having avg(price) > 6.0");

        cepStatement.addListener(new CEPListener());

       // We generate a few ticks...
        for (int i = 0; i < 5; i++) {
            GenerateRandomTick(cepRT);
        }
    }
}

Output:

log4j:WARN No appenders could be found for logger (com.espertech.esper.epl.metric.MetricReportingPath).
log4j:WARN Please initialize the log4j system properly.
Sending tick:Price: 6.0 time: Tue Jul 21 01:11:15 CEST 2009
Sending tick:Price: 0.0 time: Tue Jul 21 01:11:15 CEST 2009
Sending tick:Price: 7.0 time: Tue Jul 21 01:11:15 CEST 2009
Sending tick:Price: 4.0 time: Tue Jul 21 01:11:15 CEST 2009
Sending tick:Price: 9.0 time: Tue Jul 21 01:11:15 CEST 2009
Event received: Price: 9.0 time: Tue Jul 21 01:11:15 CEST 2009

As you can see, only the last two ticks have an average above the value of 6.0, and therefore only one event was triggered by the engine. Fantastic!

Oh, you maybe worry about the two first lines. Yeah, there is still a bit of an issue to sort out. In fact, Esper uses a logging package called log4j that causes these warnings. It can be configured using a file called log4j.xml which you will find in the /etc folder of one of the example project in esper-3.1.0/examples. I don’t think that having a XML configuration file for everything is a very good idea and therefore I have used to following trick to configure the logger. Add the following imports at the beginning of your file:

import org.apache.log4j.ConsoleAppender;
import org.apache.log4j.SimpleLayout;
import org.apache.log4j.Level;
import org.apache.log4j.Logger;

and this in the main function before the rest of your code:

 public static void main(String [] args){

      SimpleLayout layout = new SimpleLayout();
      ConsoleAppender appender = new ConsoleAppender(new SimpleLayout());
      Logger.getRootLogger().addAppender(appender);
      Logger.getRootLogger().setLevel((Level) Level.WARN);
(...)

The five minutes are now over.

In my next posts, I will explore EPL statements a little bit more in depth, and provide some code to connect two engines together in order to facilitate what is called “event refinement”.

About these ads

41 responses to “Getting started with Esper in 5 minutes.

  1. It would be nice if you can provide a full solution in MS VS 2008.

    Thanks for the information

    • Sorry, I do not use Visual Studio (VS) at home (I am a linux/OS X user).

      I will provide a full solution with illustrations in netbeans 6.7, which is the IDE I use for Java. Netbeans is free, and it’s pretty good.

      • Hi, thanks for your starter example.
        I’m looking forward for ur netbeans article. Right now I’m using in it, and I found some problems when working with Esper, in particular the log4j library. Any tentative post date?

      • Hi coffeeonesugar,
        This was an excellent article for those of us that are new to esper. I’m trying to find your full solution in Netbeans. Can you point me to where you’ve posted it?
        Thanks

      • coffeeonesugar

        well, I haven’t posted anything for months now. I have some time in my hands now, so I will probably post the promised “full solution” anytime soon.

  2. Great article…having read the ESper docs and been confused, I have now seen the light and am saved. Seriously though, good clear article. Look forward to your next ones.

  3. Pingback: Esper Starter! « Burning the Midnight Oil

  4. Hey coffeeonesugar,

    This is a great heads up on how to get this stuff up and running.
    If you could notify us when your next article is available that would be great.

    Thanks,
    Mark.

  5. Great introduction! I was unable to proceed with the docs that came with Esper but this helped me get started.

    Wanted to share a point that might help future readers: there are two commons logging jars that come with Esper 3.3.0. The one in the esperio\lib\ should be set in the classpath; if you set the jar in the esper\lib folder the NoClassDefFoundError will not go away.

  6. Hello coffeeonesugar,
    This is a great article that really helped me to starting Esper.
    Now I have a question:
    Is there any visualization environment for running EPL queries, like Mysql GUI tools that runs sql queries and show errors.
    So I have a query in my program that it does not run and throws any error/exception.

    Thank you.
    Nayyereh

  7. Good Article, I have got some basic idea/concept of what exactly ESPER can do… Thanks for your effort put together into this Article.

  8. I tried the example that you have given and its working ! Although I have taken around an Hour. But now I can do all from scratch it in 5 min. Thanks.

    You can do away will all the code snippets at the starting, better to give entire code at one place and explain using line numbers.

  9. Great articles, I see it’s been a while since your last article. I hope you’ll resume writing soon, as I’m definitely looking forward to it!

  10. hello
    I just want to know that where can I get the packages
    com.espertech.esper.client.*;

    Actually I am new to Esper and when i tried to run your code, it throws an exception as expected
    So can you tell me how to run this program?

  11. Anatoli Barski

    greate thanks!!

  12. Thanks a lot for your helpful article…

  13. Thanx for a nice article. I have managed to compile the example, but when running it I guess this:
    Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory
    at com.espertech.esper.client.Configuration.(Configuration.java:39)
    at exampleTick.main(exampleTick.java:50)
    Caused by: java.lang.ClassNotFoundException: org.apache.commons.logging.LogFactory

    Any help on what might cause it?

    • Add log4j jar in your classpath. The error says it could not found the LogFactory class that is present in log4j jar.

  14. Hello,
    Congratulation for your job, you are the best teacher I ever seen in Esper CEP.
    Thank you!
    Very good job!

    Herbert

  15. i am new to esper.would u tell me how to compile and run this example.
    please help…….

  16. i am new to esper. i find this artical very much useful.thanks. i tried ur example and it was succesful. thank u.

  17. Thanks a lot…!! I did it in 30 min. :)
    I am not a computer science student and I am very new to Java and using Eclipse platform. So I needed a tutorial which is very basic. Like I don’t know stuff like “adding libraries to project” . but with help of google I could figure things out. Your tutorial helped a lot. I did it in 30 min :) thanks.

  18. Many thanks for this practical, clear guide. I have just successfully completed your tutorial using Esper 4.2 and Eclipse Helios.

    An excellent intro to get the “Esper basics” in code before moving onto the packaged examples – thanks!

  19. Good one. Helps to understand how to process the complex event through esper.

  20. thank you for the tuto ! it work very well !!!

  21. Thanks a lot for your helpful article, I write it in NEsper .Net, it works ok in c# but no event was triggered by the engine, becouse the function “Update” was not used by anyone, what can be the problem? please help!!

  22. Can we use the complex Event programming on oracle database. You have any simple example of using relational database.

  23. Thanks a lot for your article : it makes me Esper API easier to understand

  24. Simply cool stuff. Exactly what I was looking for! Thank you!

  25. it was a grate tutorial

  26. Hi … Very good intro … I’ve done CEP work in the past with Apama and now working on a project that will likely use Esper. I am trying to determine whether EPL or Java is the better approach. The Apama CEP engine [correlator] is tuned to run its EPL [monitorscipt] efficiently and I am wondering whether the same is true for Esper. Please point me to your EPL posts

    I am also interested in building dynamic queries/filters to support on-the-fly filtering using variables or perhaps an event containing the new filter parameters. I would like to hear your thoughts and seeing an example if you have one.

    Best regards,

    Fred

  27. Trucklos Morales

    Thanks for such succinct and enlightening tutorial.

  28. Thank you I was so lost :D Now I see the light :D

  29. Hi, thanks for the great Esper starter post.
    I did run into a NPE on the line:
    cepStatement.addListener(new CEPListener());

    Any ideas on why?

    Thanks again,

  30. never mind, failed to createEPL first.

  31. Thank you very much for this tutorial! Those who are, like me, just starting to use Esper with .NET, quickly found out that there are some non-trivial changes we need to make in java code to make it work in C#/NEsper.
    Below is a method Main that works in .NET. Other code above is pretty much copy-paste.

    static void Main(string[] args)
    {
    //The Configuration is meant only as an initialization-time object.
    com.espertech.esper.client.Configuration cepConfig = new com.espertech.esper.client.Configuration();
    // We register Ticks as objects the engine will have to handle
    cepConfig.AddEventType(“StockTick”, typeof(Tick).ToString());

    // We setup the engine
    EPServiceProvider cep = EPServiceProviderManager.GetProvider(“myCEPEngine”, cepConfig);
    EPRuntime cepRT = cep.EPRuntime;

    // We register an EPL statement
    EPAdministrator cepAdm = cep.EPAdministrator;
    EPStatement cepStatement = cepAdm.CreateEPL(string.Format(“select * from StockTick(Symbol=’AAPL’).win:length({0}) having avg(Price) > {1}”, 2, 6.0));
    CEPListener listener = new CEPListener();
    cepStatement.Events += (s, e) => listener.Update(e.NewEvents, e.OldEvents);

    for (int i = 0; i < 5; i++)
    GenerateRandomTick(cepRT);
    }

  32. Hi CoffeeOneSugar, Great Article, This one helped me get a good start into Esper world, Thanks!!!!

  33. Pingback: [에스퍼] 간단한 실시간 이벤트 처리 기술 | Not for Me

  34. Muhammad Asif Ihsan

    Very nice presentation and explanation. These five minutes tutorial is very useful. Nice work. What is your next post regarding EPL. Looking for it, If you have not posted yet, then waiting for it. You can direct me to the link if you have posted further somewhere.

  35. Pingback: Esper Engine | Kiril's

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s