activemq clients all hang

[ohad]

Hi All
I am developing a high load multi-process application which is sending a
fairly large amount of messages . I am using version 4.0.1 and those
features :
1.Connections are standard, non durable, non auto-reconnect.
2.sessions are non transactional , AutoAcknowledge
3.temporary queues are created one for each connection , and usually a
process has one
4.JMSConnection instance and multiple session objects.
5. I am using the ActiveMQConnection.destroyDestination(...) function
sometimes to destroy destinations that won't be used any more

The processes work fine for 3-4 hours , and then , i experience those
symptoms :
1. every MessageProducer.send() function hangs and doesn't return
2. Queues are not being emptied : i have a code that browses the queue ,
find the number of messages out of it , and the makes a receive() function
for every message. The queue is not empty even after the messages are
consumed.
3. memory consumption is normal for all i know.

I haven't compiled a juint test to recreate it , but it may be the only way
to handle this problem...
Any idea anyone ?

[Christopher Mihaly]

I am seeing exactly the same problem. Right now I am completely stuck, every
call to the activemq hangs. There is no diagnostics of note, other than the
subsequent calls all hang witha client already connected because the
original is hung. Any ideas on how to fixt this?

[ohad]

here is what i did to reduce the problems ( i have been running my app for
hours now and it seems fine ) :
1. use JMX with jconsole to hook onto activemq , and make sure that every
queue that you opened is closed . it is better to close a MessageProducer
before a MessageConsumer , because then the queue is purged entirely and it
is out of the system.
2. make sure that there are no consumers and producers that are hanging open
- i think that this is the main reason why it happened to me
3. I used ActiveMQConnection.destroyDestination(...) on each of my queues
once they are no longer needed - perhaps you can use it too.

Sorry for having only tips - i can't tell what truly caused that problem...

Ohad Serfaty
DIMES team

[ohad]

This problem is back again...
I thought i solved it but the minute i have put some more load on the system
it hanged all the processes. It seems that there is some deadlock that
locaks all the send() and createConsumer() calls - and i can't reconstruct
the problem.

Christpher , can you describe what you are doing exactly so we can somehow
see if we have some things in common ?

Anyone ?

[Christopher Mihaly]

On the test case, I created a topic publisher and create 100000 messages
using asyncsend. This works great. Then I make a durable subscription and
that client exits leaving his durable subsription. Then I send the
publisher and around 2000 messages it will hang. The only solution to
this is to run my client that subscribed to the subscription and read the
thousands of messages. Then I can send another couple thousand messages,
etc. It is completely repeatable. This happens if I am using journaling
or not, or out of the box, derby or MS SQLServer as the persistenace DB.
I believe this is what is happening on our production server. We have
clients that are run weekly, or at other unspecified times that will hold
durable subscriptions to the topics. I have not found any way around this
other than killing off subscriptions (sometimes not possible since you
cannot destroy a subsription that is off-line, not sure why they are
off-line, but I guess over time they become off-line). So the only
solutoins is to destroy the topic and recreate, but this destroys a lot of
data for us and is getting unusable :_(

Chris


Chris

[James Strachan]

Are you sending the messages with persistent message delivery mode?
Also what version are you using?

Post by Christopher Mihaly
On the test case, I created a topic publisher and create 100000 messages
using asyncsend. This works great. Then I make a durable subscription and
that client exits leaving his durable subsription. Then I send the
publisher and around 2000 messages it will hang. The only solution to
this is to run my client that subscribed to the subscription and read the
thousands of messages. Then I can send another couple thousand messages,
etc. It is completely repeatable. This happens if I am using journaling
or not, or out of the box, derby or MS SQLServer as the persistenace DB.
I believe this is what is happening on our production server. We have
clients that are run weekly, or at other unspecified times that will hold
durable subscriptions to the topics. I have not found any way around this
other than killing off subscriptions (sometimes not possible since you
cannot destroy a subsription that is off-line, not sure why they are
off-line, but I guess over time they become off-line). So the only
solutoins is to destroy the topic and recreate, but this destroys a lot of
data for us and is getting unusable :_(
Chris
Chris
--
View this message in context: http://www.nabble.com/activemq-clients-all-hang-tf1950608.html#a5461880
Sent from the ActiveMQ - User forum at Nabble.com.

[Christopher Mihaly]

Post by James Strachan
Are you sending the messages with persistent message delivery mode?
private int deliveryMode = javax.jms.DeliveryMode.PERSISTENT;
private int deliveryPriority = javax.jms.Message.DEFAULT_PRIORITY;
publisher.send(msg, deliveryMode, deliveryPriority,
Message.DEFAULT_TIME_TO_LIVE);
Also what version are you using?
Version Version 4.0.1

Post by Christopher Mihaly
On the test case, I created a topic publisher and create 100000 messages
using asyncsend. This works great. Then I make a durable subscription and
that client exits leaving his durable subsription. Then I send the
publisher and around 2000 messages it will hang. The only solution to
this is to run my client that subscribed to the subscription and read the
thousands of messages. Then I can send another couple thousand messages,
etc. It is completely repeatable. This happens if I am using journaling
or not, or out of the box, derby or MS SQLServer as the persistenace DB.
I believe this is what is happening on our production server. We have
clients that are run weekly, or at other unspecified times that will hold
durable subscriptions to the topics. I have not found any way around this
other than killing off subscriptions (sometimes not possible since you
cannot destroy a subsription that is off-line, not sure why they are
off-line, but I guess over time they become off-line). So the only
solutoins is to destroy the topic and recreate, but this destroys a lot of
data for us and is getting unusable :_(
Chris
Chris
--
http://www.nabble.com/activemq-clients-all-hang-tf1950608.html#a5461880
Sent from the ActiveMQ - User forum at Nabble.com.

--
James
-------
http://radio.weblogs.com/0112098/

[James Strachan]

Do you have any slow consumers (non-durable consumers) who are not keeping up?

Post by Christopher Mihaly

Post by James Strachan
Are you sending the messages with persistent message delivery mode?
private int deliveryMode = javax.jms.DeliveryMode.PERSISTENT;
private int deliveryPriority = javax.jms.Message.DEFAULT_PRIORITY;
publisher.send(msg, deliveryMode, deliveryPriority,
Message.DEFAULT_TIME_TO_LIVE);
Also what version are you using?
Version Version 4.0.1

Post by Christopher Mihaly
On the test case, I created a topic publisher and create 100000 messages
using asyncsend. This works great. Then I make a durable subscription and
that client exits leaving his durable subsription. Then I send the
publisher and around 2000 messages it will hang. The only solution to
this is to run my client that subscribed to the subscription and read the
thousands of messages. Then I can send another couple thousand messages,
etc. It is completely repeatable. This happens if I am using journaling
or not, or out of the box, derby or MS SQLServer as the persistenace DB.
I believe this is what is happening on our production server. We have
clients that are run weekly, or at other unspecified times that will hold
durable subscriptions to the topics. I have not found any way around this
other than killing off subscriptions (sometimes not possible since you
cannot destroy a subsription that is off-line, not sure why they are
off-line, but I guess over time they become off-line). So the only
solutoins is to destroy the topic and recreate, but this destroys a lot of
data for us and is getting unusable :_(
Chris
Chris
--
http://www.nabble.com/activemq-clients-all-hang-tf1950608.html#a5461880
Sent from the ActiveMQ - User forum at Nabble.com.

--
James
-------
http://radio.weblogs.com/0112098/

--
View this message in context: http://www.nabble.com/activemq-clients-all-hang-tf1950608.html#a5469193
Sent from the ActiveMQ - User forum at Nabble.com.

[Christopher Mihaly]

There are no non-durable consumers. It is a pretty simple test, I created
a durable subscriber that subscribed for messages on the topic. Then I shut
down that consumer and wrote a publisher that wrote persistent messages, up
to 100,000 of them. It will have after about 1800 messages every time. If
I then run my druable consumer, it will read the messages off the queue and
I can send another few messages before it hangs.

If I leave the durable subscriber inactive for a while, it will go OFF-LINE
and sometimes I am unable to get it to back on-lne ever. But that is a
different message thread.

Chris

Post by James Strachan
Do you have any slow consumers (non-durable consumers) who are not keeping up?

Post by Christopher Mihaly

Post by James Strachan
Are you sending the messages with persistent message delivery mode?
private int deliveryMode = javax.jms.DeliveryMode.PERSISTENT;
private int deliveryPriority = javax.jms.Message.DEFAULT_PRIORITY;
publisher.send(msg, deliveryMode, deliveryPriority,
Message.DEFAULT_TIME_TO_LIVE);
Also what version are you using?
Version Version 4.0.1

Post by Christopher Mihaly
On the test case, I created a topic publisher and create 100000 messages
using asyncsend. This works great. Then I make a durable

subscription

Post by James Strachan

Post by Christopher Mihaly
and
that client exits leaving his durable subsription. Then I send the
publisher and around 2000 messages it will hang. The only solution

to

Post by James Strachan

Post by Christopher Mihaly
this is to run my client that subscribed to the subscription and read

the

Post by James Strachan

Post by Christopher Mihaly
thousands of messages. Then I can send another couple thousand messages,
etc. It is completely repeatable. This happens if I am using journaling
or not, or out of the box, derby or MS SQLServer as the persistenace

DB.

Post by James Strachan

Post by Christopher Mihaly
I believe this is what is happening on our production server. We

have

Post by James Strachan

Post by Christopher Mihaly
clients that are run weekly, or at other unspecified times that will

hold

Post by James Strachan

Post by Christopher Mihaly
durable subscriptions to the topics. I have not found any way around this
other than killing off subscriptions (sometimes not possible since you
cannot destroy a subsription that is off-line, not sure why they are
off-line, but I guess over time they become off-line). So the only
solutoins is to destroy the topic and recreate, but this destroys a

lot

Post by James Strachan

Post by Christopher Mihaly
of
data for us and is getting unusable :_(
Chris
Chris
--

http://www.nabble.com/activemq-clients-all-hang-tf1950608.html#a5461880

Post by James Strachan

Post by Christopher Mihaly
Sent from the ActiveMQ - User forum at Nabble.com.

--
James
-------
http://radio.weblogs.com/0112098/

--
http://www.nabble.com/activemq-clients-all-hang-tf1950608.html#a5469193
Sent from the ActiveMQ - User forum at Nabble.com.

--
James
-------
http://radio.weblogs.com/0112098/

[James Strachan]

Could you raise a JIRA for this - its sounding like the cache eviction
for durable topics is not kicking in and things are blocking on the
amount of RAM in the <usageManager/>

Post by Christopher Mihaly
There are no non-durable consumers. It is a pretty simple test, I created
a durable subscriber that subscribed for messages on the topic. Then I shut
down that consumer and wrote a publisher that wrote persistent messages, up
to 100,000 of them. It will have after about 1800 messages every time. If
I then run my druable consumer, it will read the messages off the queue and
I can send another few messages before it hangs.
If I leave the durable subscriber inactive for a while, it will go OFF-LINE
and sometimes I am unable to get it to back on-lne ever. But that is a
different message thread.
Chris

Post by James Strachan
Do you have any slow consumers (non-durable consumers) who are not keeping up?

Post by Christopher Mihaly

Post by James Strachan
Are you sending the messages with persistent message delivery mode?
private int deliveryMode = javax.jms.DeliveryMode.PERSISTENT;
private int deliveryPriority = javax.jms.Message.DEFAULT_PRIORITY;
publisher.send(msg, deliveryMode, deliveryPriority,
Message.DEFAULT_TIME_TO_LIVE);
Also what version are you using?
Version Version 4.0.1

Post by Christopher Mihaly
On the test case, I created a topic publisher and create 100000 messages
using asyncsend. This works great. Then I make a durable

subscription

Post by James Strachan

Post by Christopher Mihaly
and
that client exits leaving his durable subsription. Then I send the
publisher and around 2000 messages it will hang. The only solution

to

Post by James Strachan

Post by Christopher Mihaly
this is to run my client that subscribed to the subscription and read

the

Post by James Strachan

Post by Christopher Mihaly
thousands of messages. Then I can send another couple thousand messages,
etc. It is completely repeatable. This happens if I am using journaling
or not, or out of the box, derby or MS SQLServer as the persistenace

DB.

Post by James Strachan

Post by Christopher Mihaly
I believe this is what is happening on our production server. We

have

Post by James Strachan

Post by Christopher Mihaly
clients that are run weekly, or at other unspecified times that will

hold

Post by James Strachan

Post by Christopher Mihaly
durable subscriptions to the topics. I have not found any way around this
other than killing off subscriptions (sometimes not possible since you
cannot destroy a subsription that is off-line, not sure why they are
off-line, but I guess over time they become off-line). So the only
solutoins is to destroy the topic and recreate, but this destroys a

lot

Post by James Strachan

Post by Christopher Mihaly
of
data for us and is getting unusable :_(
Chris
Chris
--

http://www.nabble.com/activemq-clients-all-hang-tf1950608.html#a5461880

Post by James Strachan

Post by Christopher Mihaly
Sent from the ActiveMQ - User forum at Nabble.com.

--
James
-------
http://radio.weblogs.com/0112098/

--
http://www.nabble.com/activemq-clients-all-hang-tf1950608.html#a5469193
Sent from the ActiveMQ - User forum at Nabble.com.

--
James
-------
http://radio.weblogs.com/0112098/

--
View this message in context: http://www.nabble.com/activemq-clients-all-hang-tf1950608.html#a5480407
Sent from the ActiveMQ - User forum at Nabble.com.

[Christopher Mihaly]

Ok, I submitted an issue, and I included a junit test case that in my case is
100% repeatatble. This one is a big problem for me, because this combined
with another problem I have with off-line subscriptions getting stuck, I
have to kill my topic after a few days which breaks a lot of stuff.

Thanks
Chris

[Christopher Mihaly]

Do we have any ETA for this. I know I just submitted this yesterday, but
this is become really critical for us. Right now, we are grinding to halt
every day, and we cannot work that way.

Chris

Post by Christopher Mihaly
Ok, I submitted an issue, and I included a junit test case that in my case
is 100% repeatatble. This one is a big problem for me, because this
combined with another problem I have with off-line subscriptions getting
stuck, I have to kill my topic after a few days which breaks a lot of
stuff.
Thanks
Chris

[James Strachan]

Post by ohad
This problem is back again...
I thought i solved it but the minute i have put some more load on the system
it hanged all the processes. It seems that there is some deadlock that
locaks all the send() and createConsumer() calls - and i can't reconstruct
the problem.

If its really a deadlock then you should be able to create a stack
trace and show us the objects which are deadlocking.

[ohad]

I have managed to find a peace of code that hangs my clients.
Here is the situation :
1. a process starts , opening connections and a consumer
2. some producers send messages to the queue
3. the process terminates abruptly without closing the connection or the
consumer
4. when i try and open a consumer on the same queue - it hangs.

I have some code here that gets activemq stuck. I hope someone will try it
out :
(Please note that you have to run it twice in order to have it stuck...)

package dimes.taskloader.testing.manual;

import java.io.IOException;

import javax.jms.JMSException;
import javax.jms.Message;
import javax.jms.MessageConsumer;
import javax.jms.MessageListener;
import javax.jms.MessageProducer;
import javax.jms.Queue;
import javax.jms.Session;

import org.apache.activemq.ActiveMQConnection;
import org.apache.activemq.ActiveMQConnectionFactory;

/**
* @author Ohad Serfaty
*
*/
public class HangTest {


public void testMe() throws IOException, InterruptedException,
JMSException{
ActiveMQConnectionFactory factory = new ActiveMQConnectionFactory();
ActiveMQConnection connection = (ActiveMQConnection)
factory.createConnection();
connection.setAlwaysSessionAsync(true);

final Session session = connection.createSession(false ,
Session.AUTO_ACKNOWLEDGE);

final Queue queue = session.createQueue("test.queue.1");


System.out.println("Creating consumer...");
MessageConsumer consumer = connection.createSession(true ,
Session.AUTO_ACKNOWLEDGE).createConsumer(queue);
System.out.println("Setting message listener...");
consumer.setMessageListener( new MessageListener()
{

public void onMessage(Message arg0) {
System.out.println("Hey!");
}

});


Thread[] pool = new Thread[5];
final MessageProducer[] producers = new MessageProducer[5];

for (int j=0; j<5; j++)
{
producers[j] = session.createProducer(queue);
final int k = j;
pool[j] = new Thread(){

public void run(){
System.out.println("running ?");
try
{


for (int i = 0; i < 1000; i++)
{
producers[k].send(session.createTextMessage("Hello!"));
System.out.println(i + ":Got here ?");
}
}
catch (Exception e)
{
e.printStackTrace();
}
System.out.println("Ended ?");
}
};
}

System.out.println("starting threads...");
for (int i=0; i<5; i++)
pool[i].start();
Thread.sleep(2000);
for (int i=0; i<5; i++)
producers[i].close();
//consumer.close();
session.close();
//connection.close();
}


// Please note that this method has to be run twice ( from two different
processes...)
public static void main(String[] args) throws IOException,
InterruptedException, JMSException
{
HangTest tester = new HangTest();
tester.testMe();
}
}