How do I compile and run Artemis tests?

Discussion:

Jiri Danek

2016-06-28 21:29:52 UTC

After somewhat lengthy struggle I still did not manage to run Artemis tests
in a VM. I'd like to ask if there is anybody who managed to run the tests
and I would like to ask for suggestions what could be causing the tests to
fail for me. Is there something obvious I need to do but I am not doing?

I am running this series of commands

mvn -Prelease clean package install
mvn -Ptests test

My issues to date were ... mostly hardware requirements.

First of all, I checked the source on Saturday when the build was broken,
so I had to checkout 1.3.0 tag.

Running the tests seems to require 4GB of RAM (tried 512MB, then 2GB, and
that did not work) and high open file descriptor limit, the default 1024 is
not enough, I set 40 000, because that seems to be what the Apache Jenkins
slaves use. It also requires a lot of time, because the tests run long.

The settings above got me probably furthest I ever got. Before I increased
the descriptor limit, tests got stuck at repeatedly printing

[main] 11:42:52,381 WARN [org.apache.activemq.artemis.core.client]
AMQ212054: Destination address=FailoverTestAddress is blocked. If the
system is configured to block make sure you consume messages on this
configuration.
[main] 11:43:02,381 WARN [org.apache.activemq.artemis.core.client]
AMQ212054: Destination address=FailoverTestAddress is blocked. If the
system is configured to block make sure you consume messages on this
configuration.
[main] 11:43:12,381 WARN [org.apache.activemq.artemis.core.client]
AMQ212054: Destination address=FailoverTestAddress is blocked. If the
system is configured to block make sure you consume messages on this
configuration.
[main] 11:43:22,381 WARN [org.apache.activemq.artemis.core.client]
AMQ212054: Destination address=FailoverTestAddress is blocked. If the
system is configured to block make sure you consume messages on this
configuration.

After increasing the limit, I got stuck at a test repeatedly printing

javax.jms.IllegalStateException: The Session is closed
at
org.apache.activemq.ActiveMQSession.checkClosed(ActiveMQSession.java:775)
at
org.apache.activemq.ActiveMQSession.configureMessage(ActiveMQSession.java:763)
at
org.apache.activemq.ActiveMQSession.createTextMessage(ActiveMQSession.java:440)
at
org.apache.activemq.artemis.tests.integration.openwire.amq.ProducerFlowControlSendFailTest$1.run(ProducerFlowControlSendFailTest.java:96)

But I do not know how to move from here. Thanks for any input,

--
JiÅÃ DanÄk

Clebert Suconic

2016-07-05 03:42:06 UTC

Permalink

Did you manage to do it. Or still have issues ?

Post by Jiri Danek
After somewhat lengthy struggle I still did not manage to run Artemis tests
in a VM. I'd like to ask if there is anybody who managed to run the tests
and I would like to ask for suggestions what could be causing the tests to
fail for me. Is there something obvious I need to do but I am not doing?
I am running this series of commands
mvn -Prelease clean package install
mvn -Ptests test
My issues to date were ... mostly hardware requirements.
First of all, I checked the source on Saturday when the build was broken,
so I had to checkout 1.3.0 tag.
Running the tests seems to require 4GB of RAM (tried 512MB, then 2GB, and
that did not work) and high open file descriptor limit, the default 1024 is
not enough, I set 40 000, because that seems to be what the Apache Jenkins
slaves use. It also requires a lot of time, because the tests run long.
The settings above got me probably furthest I ever got. Before I increased
the descriptor limit, tests got stuck at repeatedly printing
[main] 11:42:52,381 WARN [org.apache.activemq.artemis.core.client]
AMQ212054: Destination address=FailoverTestAddress is blocked. If the
system is configured to block make sure you consume messages on this
configuration.
[main] 11:43:02,381 WARN [org.apache.activemq.artemis.core.client]
AMQ212054: Destination address=FailoverTestAddress is blocked. If the
system is configured to block make sure you consume messages on this
configuration.
[main] 11:43:12,381 WARN [org.apache.activemq.artemis.core.client]
AMQ212054: Destination address=FailoverTestAddress is blocked. If the
system is configured to block make sure you consume messages on this
configuration.
[main] 11:43:22,381 WARN [org.apache.activemq.artemis.core.client]
AMQ212054: Destination address=FailoverTestAddress is blocked. If the
system is configured to block make sure you consume messages on this
configuration.
After increasing the limit, I got stuck at a test repeatedly printing
javax.jms.IllegalStateException: The Session is closed
at
org.apache.activemq.ActiveMQSession.checkClosed(ActiveMQSession.java:775)
at
org.apache.activemq.ActiveMQSession.configureMessage(ActiveMQSession.java:763)
at
org.apache.activemq.ActiveMQSession.createTextMessage(ActiveMQSession.java:440)
at
org.apache.activemq.artemis.tests.integration.openwire.amq.ProducerFlowControlSendFailTest$1.run(ProducerFlowControlSendFailTest.java:96)
But I do not know how to move from here. Thanks for any input,
--
JiÅÃ DanÄk

--
Clebert Suconic

Jiri Danek

2016-07-05 13:27:03 UTC

Permalink

After increasing the [memory] limit, I got stuck at a test repeatedly
printing
javax.jms.IllegalStateException: The Session is closed
at
org.apache.activemq.ActiveMQSession.checkClosed(ActiveMQSession.java:775)
at
org.apache.activemq.ActiveMQSession.configureMessage(ActiveMQSession.java:763)
at
org.apache.activemq.ActiveMQSession.createTextMessage(ActiveMQSession.java:440)
at
org.apache.activemq.artemis.tests.integration.openwire.amq.ProducerFlowControlSendFailTest$1.run(ProducerFlowControlSendFailTest.java:96)
Did you manage to do it. Or still have issues ?

No, I did not manage to get over the "ProducerFlowControlSendFailTest"
issue I mention above. I tried giving the VM even more memory, 8GB, but
that did not help. Any suggestions what I should try?

--
JiÅÃ DanÄk

Clebert Suconic

2016-07-05 15:01:19 UTC

Permalink

This test is currently failing, but I didn't see it hanging anywhere.

Perhaps we should add it to the ignore list until it's fixed.

Post by Jiri Danek

No, I did not manage to get over the "ProducerFlowControlSendFailTest"
issue I mention above. I tried giving the VM even more memory, 8GB, but
that did not help. Any suggestions what I should try?
--
Jiří Daněk

--
Clebert Suconic

Jiri Danek

2016-07-06 22:34:51 UTC

Permalink

Post by Clebert Suconic
This test is currently failing, but I didn't see it hanging anywhere.
Perhaps we should add it to the ignore list until it's fixed.

I am attaching my output of `mvn -Pfast-tests -Pextra-tests install`. It
looks much different than what Jenkins at Apache produces for the same
commit,
https://builds.apache.org/job/ActiveMQ-Artemis-PR-Build/1446/consoleFull

Could you please take look at it? Maybe you could guess why is my run so
much different. For me, three tests fail, whereas on Jenkins everything
succeeds. Also, in my run, there are scary messages like

*********************************************************************************
LEAKING THREADS
=============================================================================
Thread Thread[Thread-0
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$***@445b295b-1239807799
),5,ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$***@445b295b-1239807799]
is still alive with the following stackTrace:
sun.misc.Unsafe.unpark(Native Method)
java.util.concurrent.locks.LockSupport.unpark(LockSupport.java:141)
java.util.concurrent.locks.AbstractQueuedSynchronizer.unparkSuccessor(AbstractQueuedSynchronizer.java:662)
java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1264)
java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:457)
java.util.concurrent.ThreadPoolExecutor.tryTerminate(ThreadPoolExecutor.java:714)
java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1007)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
*********************************************************************************
[main] 20:06:48,806 INFO [org.apache.activemq.artemis.core.server] #test
forceGC
[main] 20:06:48,831 INFO [org.apache.activemq.artemis.core.server] #test
forceGC Done
******************** Threads cleared after retries ********************

Thanks for any help,

--
JiÅÃ DanÄk

Clebert Suconic

2016-07-07 18:15:38 UTC

Permalink

These Leaking threads are normal options...

The tests are using org.apache.activemq.artemis.tests.util.ThreadLeakCheckRule.

Since my early days in Messaging a common issue on running tests is a
leaking server bumping into the next test.

To solve this I had written checks to validate that the next test will
run clean. But sometimes an executor will take longer to have its
thread dying and there is a retry on the ThreadLeakCheckRule, at the
end you should see cleared after retries, and that should be fine. But
I keep these reported in case there's something else going on.

Post by Jiri Danek

Post by Clebert Suconic
This test is currently failing, but I didn't see it hanging anywhere.
Perhaps we should add it to the ignore list until it's fixed.

I am attaching my output of `mvn -Pfast-tests -Pextra-tests install`. It
looks much different than what Jenkins at Apache produces for the same
commit,
https://builds.apache.org/job/ActiveMQ-Artemis-PR-Build/1446/consoleFull
Could you please take look at it? Maybe you could guess why is my run so
much different. For me, three tests fail, whereas on Jenkins everything
succeeds. Also, in my run, there are scary messages like
*********************************************************************************
LEAKING THREADS
=============================================================================
Thread Thread[Thread-0
sun.misc.Unsafe.unpark(Native Method)
java.util.concurrent.locks.LockSupport.unpark(LockSupport.java:141)
java.util.concurrent.locks.AbstractQueuedSynchronizer.unparkSuccessor(AbstractQueuedSynchronizer.java:662)
java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1264)
java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:457)
java.util.concurrent.ThreadPoolExecutor.tryTerminate(ThreadPoolExecutor.java:714)
java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1007)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
*********************************************************************************
[main] 20:06:48,806 INFO [org.apache.activemq.artemis.core.server] #test
forceGC
[main] 20:06:48,831 INFO [org.apache.activemq.artemis.core.server] #test
forceGC Done
******************** Threads cleared after retries ********************
Thanks for any help,
--
Jiří Daněk

--
Clebert Suconic

Clebert Suconic

2016-07-07 18:18:18 UTC

Permalink

do you have any patches you are working on? what else should I look for?

On Thu, Jul 7, 2016 at 2:18 PM, Clebert Suconic

What else should I look for?

Post by Jiri Danek

Post by Clebert Suconic
This test is currently failing, but I didn't see it hanging anywhere.
Perhaps we should add it to the ignore list until it's fixed.

I am attaching my output of `mvn -Pfast-tests -Pextra-tests install`. It
looks much different than what Jenkins at Apache produces for the same
commit,
https://builds.apache.org/job/ActiveMQ-Artemis-PR-Build/1446/consoleFull
Could you please take look at it? Maybe you could guess why is my run so
much different. For me, three tests fail, whereas on Jenkins everything
succeeds. Also, in my run, there are scary messages like
*********************************************************************************
LEAKING THREADS
=============================================================================
Thread Thread[Thread-0
sun.misc.Unsafe.unpark(Native Method)
java.util.concurrent.locks.LockSupport.unpark(LockSupport.java:141)
java.util.concurrent.locks.AbstractQueuedSynchronizer.unparkSuccessor(AbstractQueuedSynchronizer.java:662)
java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1264)
java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:457)
java.util.concurrent.ThreadPoolExecutor.tryTerminate(ThreadPoolExecutor.java:714)
java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1007)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)
*********************************************************************************
[main] 20:06:48,806 INFO [org.apache.activemq.artemis.core.server] #test
forceGC
[main] 20:06:48,831 INFO [org.apache.activemq.artemis.core.server] #test
forceGC Done
******************** Threads cleared after retries ********************
Thanks for any help,
--
Jiří Daněk

--
Clebert Suconic