For Tagging

For viewing the tag

>git tag

For pushing it for everyone

>git push --tags

For tagging an existing commit

> git tag -a v1.2 9fceb02 <start of the changeset code>

For fetching tags from repository

> git fetch --tags

SELECT @rn:=@rn+1 AS rank, itemID, ordercount
FROM (
  SELECT itemID, COUNT(*) AS ordercount
  FROM orders
  GROUP BY itemID
  ORDER BY ordercount DESC
) t1, (SELECT @rn:=0) t2;

for getting it in the right order

or simply

SET @rank=0;
SELECT @rank:=@rank+1 AS rank, itemID, COUNT(*) as ordercount
  FROM orders
  GROUP BY itemID
  ORDER BY ordercount DESC;

For limiting the number of rows however, there is the LIMIT keyword

[bash]

mysql> select * from CODE_PANEL_DETAIL limit 1,4;
+———————-+—————+——+—————————–+——————–+
| CODE_PANEL_DETAIL_ID | CODE_PANEL_ID | CODE | CODE_TYPE | CODE_SEARCH_STRING |
+———————-+—————+——+—————————–+——————–+
| 2 | 3 | 010 | Age appropriateness of Toys | +toy |
| 3 | 3 | 021 | Employee Helpfulness | age |
| 6 | 3 | 020 | Aisles | +toy |
| 7 | 3 | 030 | Large Variety | +toy |
+———————-+—————+——+—————————–+——————–+
4 rows in set (0.00 sec)

mysql> select * from CODE_PANEL_DETAIL limit 2;
+———————-+—————+——+—————————–+——————–+
| CODE_PANEL_DETAIL_ID | CODE_PANEL_ID | CODE | CODE_TYPE | CODE_SEARCH_STRING |
+———————-+—————+——+—————————–+——————–+
| 1 | 3 | 001 | Oranization | +toy |
| 2 | 3 | 010 | Age appropriateness of Toys | +toy |
+———————-+—————+——+—————————–+——————–+
2 rows in set (0.00 sec)
[/bash]

java.lang.VerifyError: Expecting a stackmap frame at branch target 6 bla bla bla ….

then a simple solution is to add the following default JVM argument -XX:-UseSplitVerifier

If you are doing it in eclipse, like i was, then goto Window -> Preferences -> Installed JREs -> and add the default parameter.

What does this setting do? Oracle just mentions that it a new type checker with StackMapTable attributes. Huh! Enjoy! It is Oracle now.

For the matter of benchmarks look at the graph below. As of now we are able to process over 2 million messages coming on the RabbitMQ server in a matter of less than 100 seconds. You might argue that this number does not tell you anything unless we know what we are processing. Fair point, but in this post we are not debating that. What we are trying to show is that once you have a lot of Akka actors running in parallel under a master worker kind of relationship, you can do wonders on the number of cores that you have.

The bisecting line that you see in the graph is the one that we got when we had our Akka actors listening to the queue with the help of AMQP client. See the earlier post. On the flick of a switch you can go parallel in the framework or hybrid as the case may be depending on your messages. When we switched to full parallel, we had actors consuming messages from the queue which would in-turn hand off the message to a Master Actor which would spawn Child Actors. This is based on the standard master-worker relationship. We are launching the number of workers either based on a configuration depending on the number of cores on which the framework is deployed or on the basis of the number of plugins which need to be launched in parallel.

Let us look at some code now, following is the process message of the Master Actor

class ParallelProcessor extends Actor {
…
...
 def receive = {
 case MessageFromQueue(message) => process(message)

 case ChildMessage(plugin, message) =>
 val transformedMessage = processMessage(message, plugin)
      self reply Result(message, plugin.target)

    case Result(message, target) =>
 parallelCounter -= 1
 messageList ++= List(new Result(message, target))

 if (parallelCounter == 0) {
        logger.info("**** REDUCER CALLED ---- ")
 val reducedMessage = reduceMessages(messageList)
 postPluginProcessing(reducedMessage)
 self ! PoisonPill
      }

  }
…
...

def process(message: VajraPMMessage) = {
 val numberOfWorkers = VajraPM.numberOfWorkers
 parallelCounter = currentStep.pluginsInStep.length
 val workers = Vector.fill(fetchWorkerCount(numberOfWorkers, parallelCounter))(actorOf[ParallelProcessor].start)
 val router = Routing.loadBalancerActor(CyclicIterator(workers)).start
    currentStep.pluginsInStep foreach { plugin =>
 router ! new ChildMessage(plugin, transformer.createVajraMessage(message.toXml().toString))
    }
 router ! Broadcast(PoisonPill)
 router ! PoisonPill
  }

...

As you would notice, the master actor (ParallelProcessor) receives a message. The case MessageFromQueue gets called and we call the process(message). As you would notice in the process(message), we spawn a number of child workers which is the same actor ParallelProcessor again.

Let us zoom into the process method, we create a router to handle the even delivery of ChildMessage to the workers. Right now, the router uses a CyclicIterator to distribute messages.

val workers = Vector.fill(fetchWorkerCount(numberOfWorkers))(actorOf[ParallelProcessor].start)
 val router = Routing.loadBalancerActor(CyclicIterator(workers)).start
    currentStep.pluginsInStep foreach { plugin =>
 router ! new ChildMessage(plugin, transformer.createVajraMessage(message.toXml().toString))

An interesting thing that we do here is that once the router has passed all the messages to the child workers, we issue a poisonpill to the child workers so that they can die after processing. We also pass a poison pill to the router.
Caution: If you are using a work stealing dispatcher, please ensure that your poison pills are not being eaten by actors which are yet to receive messages thus causing unintended behavior.

Let us see how the child receives the message, again we have the same receive method

 def receive = {
 case MessageFromQueue(message) => process(message)

 case ChildMessage(plugin, message) =>
 val transformedMessage = processMessage(message, plugin)
      self reply Result(message, plugin.target)

    case Result(message, target) =>
 parallelCounter -= 1
 messageList ++= List(new Result(message, target))

 if (parallelCounter == 0) {
        logger.info("**** REDUCER CALLED ---- ")
 val reducedMessage = reduceMessages(messageList)
 postPluginProcessing(reducedMessage)
 self ! PoisonPill
      }

  }

This time, we fall into the block of case ChildMessage(plugin, message) =>
Here, every child does some processing and passes back the results to the master with the self reply Result(message, plugin.target)

Finally, the results come back to the master, where it keeps a count of the number of results coming back. If you recall, the counter (parallelCounter) was set in the process method where we spawned child workers either on the basis of the number of parallel steps or depending on the number of cores that we have on the machine on which the framework is running.

 def receive = {
 case MessageFromQueue(message) => process(message)

 case ChildMessage(plugin, message) =>
 val transformedMessage = processMessage(message, plugin)
      self reply Result(message, plugin.target)

    case Result(message, target) =>
 parallelCounter -= 1
 messageList ++= List(new Result(message, target))

 if (parallelCounter == 0) {
        logger.info("**** REDUCER CALLED ---- ")
 val reducedMessage = reduceMessages(messageList)
 postPluginProcessing(reducedMessage)
 self ! PoisonPill
      }
  }

The case Result(message, target) => does a reduction logic on all the results coming back from the child workers.

Thus, with the master worker logic we were able to reduce the processing time considerably.

One of the things that we fell into was that we ended up creating a lot of new native threads on our OS as a result of which we were getting limited by the number of native threads which could be used. For this, we used a thread-pool instead of spawning a new thread. See Akka Dispatchers and our blog related on dispatchers. I would cover the threading issue in detail in another post.

So a producer would send a message to the exchange and it is the job of the message broker (RabbitMQ in our case) to ensure that the messages are delivered to the right queue.
But first the connections have to be built so that publishers can publish on the exchange and consumers can listen from it. Let us see how this is done in Akka in our project. Our framework gets initialized by the following method

def initializeFramework(xml: Elem) {
    connectionInfo = getAMQPChannelInformation(xml)
    val connection = getAMQPChannel(connectionInfo)
    val exchangeParameters = getExchangeParameters(connectionInfo)
    instantiateProducer(connection, exchangeParameters)

    logger.info("VajraPM is now initialized and ready to process messages...")
  }

I have taken out the business logic and left out the methods which would form the core of this post. Let us go method by method, first, we get all the connection information which would be relevant for us to make the connection

def getAMQPChannelInformation(xml: Elem): ConnectionInfo = {
    val momServer = (xml  "VAJRA"  "RABBITSERVER").text
    val momPort = (xml  "VAJRA"  "RABBITPORT").text
    val userName = (xml  "VAJRA"  "USERNAME").text
    val password = (xml  "VAJRA"  "PASSWORD").text
    val queueName = (xml  "VAJRA"  "QUEUENAME").text
    val exchangeName = (xml  "VAJRA"  "EXCHANGENAME").text
    val durable = (xml  "VAJRA"  "DURABLE").text
    val connectionInfo = new ConnectionInfo(momServer, momPort, userName, password, queueName, exchangeName, durable)
    connectionInfo
  }

Here, use the Scala XML magic and get done soon. Our sample XML through which

<VAJRA>
		<RABBITSERVER>localhost</RABBITSERVER>
		<RABBITPORT>5672</RABBITPORT>
		<USERNAME>guest</USERNAME>
		<PASSWORD>guest</PASSWORD>
		<QUEUENAME>myqueue</QUEUENAME>
		<EXCHANGENAME>myexchange</EXCHANGENAME>
	</VAJRA>

Now let us see how we get a connection

def getAMQPChannel(connectionInfo: ConnectionInfo): ActorRef = {
    val myAddresses = Array(new Address(connectionInfo.momServer, connectionInfo.momPort.toInt))
    val connectionParameters = ConnectionParameters(myAddresses, connectionInfo.userName, connectionInfo.password)
    val connection = AMQP.newConnection(connectionParameters)
    connection
  }

As you would notice, all the connection parameters are passed to the akka.amqp.AMQP.ConnectionParameters and a connection object is received.

Once we have the connection, let us instantiate the producer. The method instantiateProducer(connection, exchangeParameters) looks like this

 def instantiateProducer(connection: ActorRef, exchangeParameters: ExchangeParameters) {
    producer = AMQP.newProducer(connection, ProducerParameters(Some(exchangeParameters)))
  }

As you would notice, we pass exchangeParameters to the producer. You would see in the diagram above that AMQP producer requires a reference of the exchange to which it can send messages. We get the exchange information as follows

  def getExchangeParameters(connectionInfo: ConnectionInfo): ExchangeParameters = {
    var exchangeParameters: ExchangeParameters = ExchangeParameters(connectionInfo.exchangeName, Topic)
    connectionInfo.durable match {
      case "DURABLE" => exchangeParameters = ExchangeParameters(connectionInfo.exchangeName, Topic, ActiveDeclaration(durable = true, autoDelete = false))
      case _ => exchangeParameters
    }
    exchangeParameters
  }

If in our configuration we define the exchange to be durable, then we set the durable and autodelete properties of the exchange.

This would set up our producer actor and we would be able to send messages to the producer like this with the bang operator

producer ! Message(message)

As soon as the producer actor would receive the message, it would put it on the exchange that we have specified. Now, let us look at the consumer

 def createConsumer(exchangeParameters: ExchangeParameters, connection: ActorRef, queueName: String, actor: ActorRef, connectionInfo: ConnectionInfo): ActorRef = {
    connectionInfo.durable match {
      case "DURABLE" => {
        val queueDeclaration = ActiveDeclaration(durable = true, autoDelete = false)
        AMQP.newConsumer(connection, ConsumerParameters("@vajrafeed", actor, Some(queueName), Some(exchangeParameters), queueDeclaration))
      }
      case _ => AMQP.newConsumer(connection, ConsumerParameters("@vajrafeed", actor, Some(queueName), Some(exchangeParameters)))
    }
  }

Hence, now as shown in the figure above, we have a consumer listening on the queue called myqueue which is listening to the exchange myexchange. You would notice that we have a “@vajrafeed” This is the binding key which lets the messgage broker decide, which messages it should pass on onto the consumer. If there was another consumer attached to the queue with a binding key say @nyse then the messages for @vajrafeed would not be delivered to it. You can get more information about AMQP here.

• Extending an application’s functionality without compiling it again
• Adding functionality without requiring access to the original source code.
• Replacing or adding new functionality becomes easy
• Help in organizing large projects
• Help in extending the functionality of the system to unimagined areas. I know this one sounds extreme but believe me it is true. For our framework, plugins were written by third-party library providers whose domain ranged from health-care to natural language processing.

The major components of our plugin system are the following

1. Plugin trait – Which all the plugins would extend
2. PluginManager Object – Responsible for loading the plugin at the time of instantiation of the framework
3. MessageContext trait – All messages would extend this trait. This makes easy to pass a variety of messages to the plugin without restricting it to a few

Let us look at each of these one by one. Our Plugin trait looks like this

trait VajraPlugin {

  def name:String
  def performAction(context:MessageContext):Boolean
  def result:String

}

As you would notice, the implementation class would be defining a name, it would have an implementation for a performAction method which takes input as a MessageContext. We expect all our plugins to finally return the result with the result method.

A sample implementation of the plugin would look like this

class ZeroToHalfPlugin extends VajraPlugin {

  def name(): String = { "ZeroToHalf" }

  def performAction(context:MessageContext):Boolean = {
    val sleepTime = java.lang.Math.random() * 501
    val payload = context.getPayload()
    println("")
    println("I am busy for " + sleepTime/1000 + " seconds")
    Thread.sleep(sleepTime.toLong)
    println("")
    return true
  }

  def result(): String = { //return some result here }
}

Now, let us look at the Plugin manager

object PluginManager {

  var pluginMap = Map[String, String]()
  val logger = Logger(PluginManager.getClass())

  def init() {
    val classpath = List(".").map(new File(_))
    val finder = ClassFinder(classpath)
    val classes = finder.getClasses
    val classMap = ClassFinder.classInfoMap(classes)
    val plugins = ClassFinder.concreteSubclasses("com.vajra.plugin.VajraPlugin", classMap)

    plugins.foreach {
      pluginString =>
        val plugin = Class.forName(pluginString.name).newInstance().asInstanceOf[VajraPMPlugin]
        pluginMap += (plugin.name -> pluginString.name)
    }
  }

  def getPlugin(name: String): VajraPlugin = {
    if (pluginMap.isEmpty) init
    logger.debug("Fetching plugin from non-persistent pool.")
    Class.forName(pluginMap(name)).newInstance().asInstanceOf[VajraPMPlugin]

  }

Now, the consumer who needs to use a plugin,  would come to the PluginManager and ask for a plugin via the getPlugin() method. If you remember, each plugin defines its name.

When this method is called, the managers checks if the plugin map is empty. If it is then the init is called where we load the plugins into the framework. Loading the plugins is done on the basis of all classes which are extending the
com.vajra.plugin.VajraPlugin trait. For this, we use are using a utility (ClassFinder) provided by Clapper. More details can be found here.

The idea is simple. As soon as we are starting the framework, we should not have to specify the plugins that the framework would be using anywhere in any configuration. The framework should be able to pick up the plugins automatically. In our case, we pick up all the plugins which are implementing our plugin trait i.e. VajraPlugin

The last piece in our puzzle is the message which is passed to the plugin. Since we want to keep this also extensible, instead of passing the message directly, we use a trait MessageContext.

trait MessageContext {

  def toXml(): Elem
  def setPayload(payload:Payload): Unit
  def getPayload(): Payload
  def addOrAppendElement(elementName: String, data: String)
  def addOrReplaceElement(elementName: String, elementData: String)
  def getElementData(elementName: String): String

}

As you would notice we have a number of utility methods here which all the messages would be implementing. We have a large number of methods here because they are relevant to our system. You could have the relevant methods for your system. A typical message which extends the MessageContext would look like this

class MessageContextAdapter extends MessageContext {

  def toXml(): Elem = {
    null
  }
  def setPayload(payload: Payload): Unit = {}
  def getPayload(): Payload = { null }
  def addOrAppendElement(elementName: String, data: String) = {}
  def addOrReplaceElement(elementName: String, elementData: String) = {}
  def getElementData(elementName: String): String = { "" }
  def getPlugin(): Plugin = {
    return null
  }
  def removeStep {}
}

The main takeaways are that you should definitely have all the plugins extend a trait. The plugin manager is responsible for creation, destruction and maintenance of plugins. The object passed to the plugin should (could) be a trait so that the framework can be extensible.

JarClassLoader: Warning: org/hamcrest/BaseDescription.class in lib/mockito-all-1.8.5.jar is hidden by lib/junit-4.7.jar (with different bytecode)
JarClassLoader: Warning: org/hamcrest/BaseMatcher.class in lib/mockito-all-1.8.5.jar is hidden by lib/junit-4.7.jar (with different bytecode)

when we we executing code.

To remove this, you need to have the intransitive() suffixed to the dependency like this

libraryDependencies += “org.scalatest” % “scalatest_2.9.0″ % “1.6.1″ intransitive()

libraryDependencies += “org.specs2″ % “specs2_2.9.0-1″ % “1.5″

libraryDependencies += “org.clapper” % “classutil_2.9.0″ % “0.3.6″

libraryDependencies += “org.clapper” % “avsl_2.9.0″ % “0.3.2″

libraryDependencies += “org.jsoup” % “jsoup” % “1.6.1″ intransitive()

You would do it for all the dependencies where you have some jars which could be stepping on the toes of others because of transitive dependency.

AMQP.newConsumer(connection, ConsumerParameters("*", actor, Some(queueName), Some(exchangeParameters)))

As you would notice, each of the actors gets a message from the Q and invokes a plugin to do the processing and returns the results back to the result Q.
Now the fun happens when I execute the same code on my machine and my competitors (Meetu’s) machine.

Meetu has a Macbook pro – 2 cores – 2.3 GHz Intel core i5 – RAM 4GB – 64 bit
Mine is Dell Vostro – 4 cores – 2.4 GHz Intel core i5 – RAM 3GB – 32 bit

And here are the results

We start with 10 actors getting 100 messages and go all the way to 1,000 actors getting 10,000 messages. Everything in the code is the same except the hardware on which the tests are executed. I start of badly and see that on my machine 10 actors are processing messages in 5.298 s as compared to 3.039s on Meetu’s mac. This prompts me to skip to the other portion of the spectrum where I feel I can show him the power of my 4 cores. I am mistaken that the initial load of fewer messages and actors is actually having overhead for 4 cores and they would perform well on the other end.

Sadly with 1000 actors and 10000 messages the tiny mac with 2 cores outperforms the hulk dell with 4 cores by a margin of more than 10s.

What could be going wrong? There is no other heavy processing happening, on either machine, when we are running the tests. I doubt 64 bit vs 32 bit would cause such an upset. What else?

Some of the plugins are expensive to create. They would rather be created when the framework is started and pooled in memory. That brings us to the way we created our object pool(s). For our scenario, the we could actually end up having multiple object pools because each plugin would have a different # of pooled plugins. So we end up with something like this in our configuration file

…
…
  <pluginpool>
     <plugin>
         <name>foo</name>
         <size>10</size>
     </plugin>
  <pluginpool>
  <pluginpool>
     <plugin>
         <name>bar</name>
         <size>5</size>
     </plugin>
  <pluginpool>
…
…

As you would notice, we have a pool of 10 for foo and 5 for bar. Now, when our Akka actors need to pick up a plugin, they go to the PoolManager and ask for the plugin.

The PoolManager (is a Scala object) keeps track of the PluginPool(s) which are populated when the framework is initialized. It maintains a map of all the PluginPool(s) available. Hence, at the time of framework initialization the following function of PoolManager is called,

def createPluginPool(pluginName: String, pluginClassName: String, size: Int) {
    val pluginPool = new PluginPool(pluginClassName, size)
    for (i  pluginPool)
  }

The PluginPool class looks like this

import java.util.concurrent.ArrayBlockingQueue
import java.util.concurrent.atomic.AtomicInteger
import java.util.concurrent.TimeUnit

class PluginPool(pluginClassName: String, number: Int) extends Pool {

  private val size = new AtomicInteger(0)
  private val pool = new ArrayBlockingQueue[DPMPlugin](number)

  def fetchPlugin(): DPMPlugin = {
    pool.poll() match {
      case plugin: DPMPlugin => return plugin
      case null => createOrBlock
    }
  }

  def releasePlugin(plugin: DPMPlugin): Unit = {
    pool.offer(plugin)
  }

  def addPlugin(plugin: DPMPlugin):Unit ={
    pool.add(plugin)
  }

  private def createOrBlock: DPMPlugin = {
    size.get match {
      case e: Int if e == number => block
      case _ => create
    }
  }

  private def create: DPMPlugin = {
    size.incrementAndGet match {
      case e: Int if e > number => size.decrementAndGet; fetchPlugin()
      case e: Int => Class.forName(pluginClassName).newInstance().asInstanceOf[DPMPlugin]
    }
  }

  private def block: DPMPlugin = {
    val timeout = 5000
    pool.poll(timeout, TimeUnit.MILLISECONDS) match {
      case plugin: DPMPlugin => plugin
      case _ => throw new Exception("Couldn't acquire a connection in %d milliseconds.".format(timeout))
    }
  }
}

This has been inspired by the connection pool written for scala.

And now, let us look at how the Akka actor gets the plugin. It makes a call to getPlugin() with the name of the plugin

 def getPlugin(name: String): Option[DPMPlugin] = {
    val pluginPool = fetchPluginPool(name)
    pluginPool match {
      case null => None
      case _ => {
        logger.debug("Got the plugin_ _ _")
        Some(pluginPool.fetchPlugin())
      }
    }
  }

private def fetchPluginPool(name: String): PluginPool = {
    val pluginPool = pluginsPoolMap.get(name)
    pluginPool match {
      case Some(pool) => {
        logger.debug("Fetching plugin from persistent pool.")
        pool.asInstanceOf[PluginPool]
      }
      case None => {
        null
      }
    }
  }

The PoolManager picks up the right pool and calls fetchPlugin on the pool.

Once the actor is done with its processing, it releases the plugin back to the pool so that it is available for a different actor.

 def releasePlugin(name: String, plugin: DPMPlugin) {
    val pluginPool = fetchPluginPool(name)
    if (pluginPool != null) pluginPool.releasePlugin(plugin)

  }