ARTEMIS: bad-performance behaviour after 7-10 days of usage

[Francesco PADOVANI]

Hello,
I'm using Apache Artemis as MQTT broker for our IOT projects.
It's a clean Artemis installation of version 1.5.1., on a server (CentOS 7) which has 2 vCPU, 8 GB of RAM (4GB of Heap Space dedicated to Artemis) and 50 GB of SSD data disk.
After the installation of artemis Broker we started to test it with about 10-15 clients constantly connected, 150 subscribed topics and an average of 2 messages per minute per client. I think these are not huge numbers, right?
For the few days after the installation, all was good and the broker worked perfectly: it was fast and reliable. But day by day performances have decreased and after about 10 days of usage it is became almost unusable: due to resources consumption. The following is my "top" situation on the server:


top - 08:59:50 up 2 days, 22:09, 1 user, load average: 2.56, 2.88, 3.00
Tasks: 92 total, 1 running, 91 sleeping, 0 stopped, 0 zombie
%Cpu(s): 99.5 us, 0.5 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 7747288 total, 290968 free, 4697976 used, 2758344 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 2734768 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4352 mqtt 20 0 7043380 4.348g 15192 S 200.0 58.9 6153:54 java
1 root 20 0 128096 5176 2420 S 0.0 0.1 0:04.01 systemd


Now I can see the broker starting to page on disk almost always. For sure it's a wrong configuration of ours. Currently, it seems that very old address/queue (and the related retained messages) are always kept, making memory and cpu consuption growing more and more. And even after a restart, the broker takes so much to get up. Before to become reachable, it starts to make many management operations like, for example, retrieve data from paging, etc.. But I also see the broker that starts to register old topics and queues we don't need. How to clean them? How to make a topic/queue expire?


Inside my broker.xml I did set up the parameter last-value-queue=true, thinking that this was the problem ....but it doesn't solve. Or better: probably it's me that I've not understood the correct meaning of the parameter. I did check also the clients' parameters when they connect to the broker, to be sure they don't set, for example, clean-session = false (saying the broker to keep all messages also when a client disconnects). But they make it in the right way. The only thing is that they don't specify a client-id. They connect by using a username/password and certificate (over tls). So, every time a client connects, Artemis automatically provide a random client-id for it (if I understood well).

Attached you can find my broker.xml configuration file: it's pretty much the same default created during the installation procedure, but for the acceptors (which I've customized for my MQTT purpose) and the addition of parameter last-value-queue = true inside the address-setting section.


Please: some of you could help me? How I have to configure my broker instance to understand and solve these performance issues?


Thanks in advance


Francesco



________________________________

Ce message, ainsi que tous les fichiers joints à ce message, peuvent contenir des informations sensibles et/ ou confidentielles ne devant pas être divulguées. Si vous n'êtes pas le destinataire de ce message (ou que vous recevez ce message par erreur), nous vous remercions de le notifier immédiatement à son expéditeur, et de détruire ce message. Toute copie, divulgation, modification, utilisation ou diffusion, non autorisée, directe ou indirecte, de tout ou partie de ce message, est strictement interdite.


This e-mail, and any document attached hereby, may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized, direct or indirect, copying, disclosure, distribution or other use of the material or parts thereof is strictly forbidden.

[Clebert Suconic]

An easy way to check what is going on is with


Artemis data print


(Preferably with the broker stopped)



You can then check what is not being asked and why it became paging.


You could maybe replicate a similar load pattern on a test? Like sending
many messages a second instead of 2 per minute?


I wonder what is being retained not acked leaving many messages in memory.
On Mon, Jan 23, 2017 at 5:55 AM Francesco PADOVANI <

Post by Francesco PADOVANI
Hello,
I'm using Apache Artemis as MQTT broker for our IOT projects.
It's a clean Artemis installation of version 1.5.1., on a server (CentOS
7) which has 2 vCPU, 8 GB of RAM (4GB of Heap Space dedicated to Artemis)
and 50 GB of SSD data disk.
After the installation of artemis Broker we started to test it with about
10-15 clients constantly connected, 150 subscribed topics and an average of
2 messages per minute per client. I think these are not huge numbers, right?
For the few days after the installation, all was good and the broker
worked perfectly: it was fast and reliable. But day by day performances
have decreased and after about 10 days of usage it is became almost
unusable: due to resources consumption. The following
*top - 08:59:50 up 2 days, 22:09, 1 user, load average: 2.56, 2.88,
3.00Tasks: 92 total, 1 running, 91 sleeping, 0 stopped, 0
zombie%Cpu(s): 99.5 us, 0.5 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0
si, 0.0 stKiB Mem : 7747288 total, 290968 free, 4697976 used, 2758344
buff/cacheKiB Swap: 0 total, 0 free, 0 used. 2734768
avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM
TIME+ COMMAND 4352 mqtt 20 0 7043380 4.348g 15192 S 200.0 58.9
6153:54
java
1 root 20 0 128096 5176 2420 S 0.0 0.1 0:04.01
systemd*
Now I can see the broker starting to page on disk almost always. For sure
it's a wrong configuration of ours. Currently, it seems that very old
address/queue (and the related retained messages) are always kept, making
memory and cpu consuption growing
more and more. And even after a restart, the broker takes so much to get
up.
Before to become reachable, it starts to make many management operations
like, for example, retrieve data from paging, etc.. But I also see the
broker that starts to register old topics and queues we don't need. How to
clean them? How to make a topic/queue
expire?
Inside my broker.xml I did set up the parameter last-value-queue=true,
probably it's me that I've not understood the correct meaning of the
parameter.
I did check also the clients' parameters when they connect to the broker,
to be sure they don't set, for example, clean-session = false (saying the
broker to keep all messages also when a client disconnects). But they make
it in the right way. The only
thing is that they don't specify a client-id. They connect by using a
username/password and certificate (over tls). So, every time a client
connects, Artemis automatically provide a random client-id for it (if I
understood well).
Attached you can find my broker.xml configuration file: it's pretty much
the same default created during the installation procedure, but for the
acceptors (which I've customized for my MQTT purpose) and the addition of
parameter
last-value-queue = true inside the address-setting
section.
Please: some of you could help me? How I have to configure my broker
instance to understand and solve these performance issues?
Thanks in advance
Francesco
------------------------------
Ce message, ainsi que tous les fichiers joints à ce message, peuvent
contenir des informations sensibles et/ ou confidentielles ne devant pas
être divulguées. Si vous n'êtes pas le destinataire de ce message (ou que
vous recevez ce message par erreur), nous
vous remercions de le notifier immédiatement à son expéditeur, et de
détruire ce message. Toute copie, divulgation, modification, utilisation ou
diffusion, non autorisée, directe ou indirecte, de tout ou partie de ce
message, est strictement interdite.
This e-mail, and any document attached hereby, may contain confidential
and/or privileged information. If you are not the intended recipient (or
have received this e-mail in error) please notify the sender immediately
and destroy this e-mail. Any unauthorized,
direct or indirect, copying, disclosure, distribution or other use of the
material or parts thereof is strictly forbidden.

[Clebert Suconic]

A question. Are u consuming from the subscriptions or u intend to leave
them hinging?


While paging. Do u need transactions? If u start to page not using
transactions would make paging to act like partitions on Kafka.



I will be waiting for more data from you before we can help some more.



On Mon, Jan 23, 2017 at 5:55 AM Francesco PADOVANI <

Post by Francesco PADOVANI
Hello,
I'm using Apache Artemis as MQTT broker for our IOT projects.
It's a clean Artemis installation of version 1.5.1., on a server (CentOS
7) which has 2 vCPU, 8 GB of RAM (4GB of Heap Space dedicated to Artemis)
and 50 GB of SSD data disk.
After the installation of artemis Broker we started to test it with about
10-15 clients constantly connected, 150 subscribed topics and an average of
2 messages per minute per client. I think these are not huge numbers, right?
For the few days after the installation, all was good and the broker
worked perfectly: it was fast and reliable. But day by day performances
have decreased and after about 10 days of usage it is became almost
unusable: due to resources consumption. The following
*top - 08:59:50 up 2 days, 22:09, 1 user, load average: 2.56, 2.88,
3.00Tasks: 92 total, 1 running, 91 sleeping, 0 stopped, 0
zombie%Cpu(s): 99.5 us, 0.5 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0
si, 0.0 stKiB Mem : 7747288 total, 290968 free, 4697976 used, 2758344
buff/cacheKiB Swap: 0 total, 0 free, 0 used. 2734768
avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM
TIME+ COMMAND 4352 mqtt 20 0 7043380 4.348g 15192 S 200.0 58.9
6153:54
java
1 root 20 0 128096 5176 2420 S 0.0 0.1 0:04.01
systemd*
Now I can see the broker starting to page on disk almost always. For sure
it's a wrong configuration of ours. Currently, it seems that very old
address/queue (and the related retained messages) are always kept, making
memory and cpu consuption growing
more and more. And even after a restart, the broker takes so much to get
up.
Before to become reachable, it starts to make many management operations
like, for example, retrieve data from paging, etc.. But I also see the
broker that starts to register old topics and queues we don't need. How to
clean them? How to make a topic/queue
expire?
Inside my broker.xml I did set up the parameter last-value-queue=true,
probably it's me that I've not understood the correct meaning of the
parameter.
I did check also the clients' parameters when they connect to the broker,
to be sure they don't set, for example, clean-session = false (saying the
broker to keep all messages also when a client disconnects). But they make
it in the right way. The only
thing is that they don't specify a client-id. They connect by using a
username/password and certificate (over tls). So, every time a client
connects, Artemis automatically provide a random client-id for it (if I
understood well).
Attached you can find my broker.xml configuration file: it's pretty much
the same default created during the installation procedure, but for the
acceptors (which I've customized for my MQTT purpose) and the addition of
parameter
last-value-queue = true inside the address-setting
section.
Please: some of you could help me? How I have to configure my broker
instance to understand and solve these performance issues?
Thanks in advance
Francesco
------------------------------
Ce message, ainsi que tous les fichiers joints à ce message, peuvent
contenir des informations sensibles et/ ou confidentielles ne devant pas
être divulguées. Si vous n'êtes pas le destinataire de ce message (ou que
vous recevez ce message par erreur), nous
vous remercions de le notifier immédiatement à son expéditeur, et de
détruire ce message. Toute copie, divulgation, modification, utilisation ou
diffusion, non autorisée, directe ou indirecte, de tout ou partie de ce
message, est strictement interdite.
This e-mail, and any document attached hereby, may contain confidential
and/or privileged information. If you are not the intended recipient (or
have received this e-mail in error) please notify the sender immediately
and destroy this e-mail. Any unauthorized,
direct or indirect, copying, disclosure, distribution or other use of the
material or parts thereof is strictly forbidden.

[francesco81]

Hello,
just an update about this topic.

By disabling persistence, the broker starts again to work fast and reliable.
But just for about 2 more weeks: then the memory come bake to be full and
the broker starts to page constantly, and cpu usage goes to 100%.

I tried to run the command "artemis data print" as you suggested, but I have
to admit my ignorance to read the results.
I did redirect the command result to a file and it produced about 758 MB of
data.
I noticed no strange things among the initial lines but this:

/*17:39:47,214 FINE [io.netty.util.internal.PlatformDependent] Javassist:
unavailable
17:39:47,215 FINE [io.netty.util.internal.PlatformDependent] You don't have
Javassist in your class path or you don't have enough permission to load
dynamically generated classes. Please check the configuration for better
performance.*/

...I don't know how much this will weigh on performances.

Then I have a "B I N D I N G S J O U R N A L" section with thousands of
lines like the followings:

/****@AddRecordTX;txID=109,recordID=110;userRecordType=21;isUpdate=false;compactCount=0;PersistentQueueBindingEncoding
[id=110,
name=b7478faf-f0fd-481a-93ef-b6109dc3a689..cro.plantid.1073403.gwid.25324.device_profile.04.set,
address=.cro.plantid.1073403.gwid.25324.device_profile.04.set,
filterString=NOT ((AMQAddress = 'jms.queue.activemq.management') OR
(AMQAddress = 'activemq.notifications')), user=gateway, autoCreated=false]
***@Commit;txID=109,numberOfRecords=1*/

...where "cro.plantid.1073403.gwid.25324.device_profile.04.set" is an
example of a topic for a specific plant (there are about 48 topic for
plant).
What does "isUpdate=false;compactCount=0;PersistentQueueBindingEncoding"
mean?

Then I have a "### Surviving Records Summary ###" section, again with
thousands of lines like this:

/*recordID=5205;userRecordType=21;isUpdate=false;compactCount=0;PersistentQueueBindingEncoding
[id=5205, name=1447f6a659bc9bc0df2e55..cro.plantid.1071617.gwid.30889.
device_profile.02.err,
address=.cro.plantid.1071617.gwid.30889.device_profile.02.err,
filterString=NOT ((AMQAddress = 'jms.queue.activemq.management') OR
(AMQAddre
ss = 'activemq.notifications')), user=gateway, autoCreated=false]*/

Then I have the section "### Prepared TX ###", which is empty, and the ###
Message Counts ### as following:
message count=0
message reference count
prepared message count=0*/

Then I have the "M E S S A G E S J O U R N A L" section with thousands of
lines like the followings:

***@AddRecord;recordID=10088;userRecordType=31;isUpdate=false;compactCount=0;Message(messageID=10086;msg=ServerMessage[messageID=10086,durable=true,userID=nu
ll,priority=0, bodySize=512, timestamp=0,expiration=0, durable=true,
address=$sys.mqtt.retain..cro.plantid.1073620.gwid.32646.device_state.err,properties=TypedProp
erties[mqtt.message.retain=true,mqtt.qos.level=1]]@1144068272
***@Update;recordID=10088;userRecordType=32;isUpdate=true;compactCount=0;AddRef;QueueEncoding
[queueID=65]
***@Update;recordID=10088;userRecordType=32;isUpdate=true;compactCount=0;AddRef;QueueEncoding
[queueID=9747]
***@Update;recordID=10088;userRecordType=32;isUpdate=true;compactCount=0;AddRef;QueueEncoding
[queueID=7774]

... and no deleteRecord operation for that id (10088): I don't know if it is
right.

Then I have again the "### Surviving Records Summary ###" with thousands of
line like the followings;

recordID=196;userRecordType=31;isUpdate=false;compactCount=0;Message(messageID=196;msg=ServerMessage[messageID=196,durable=true,userID=null,priority=0,
bodySize=2048, timestamp=0,expiration=0, durable=true,
address=.cro.plantid.1073403.gwid.25324.device_profile.04,properties=TypedProperties[mqtt.message.retain=true,mqtt.qos.level=2]]@207430813
recordID=196;userRecordType=32;isUpdate=true;compactCount=0;AddRef;QueueEncoding
[queueID=65]

...again my ignorance: why "durable=true"? because retained? because qos2?

for the MESSAGES JOURNAL I have the ### Message Counts ### with thousands of
lines like these:
message count=55958
message reference count
queue id 2,count=11
queue id 196608,count=2
queue id 4,count=1
queue id 196619,count=4
queue id 65547,count=1
queue id 196625,count=2
queue id 196630,count=2
queue id 65559,count=1
queue id 1081351,count=1
queue id 163869,count=17
queue id 163875,count=2307
... what does this mean?

Finally I have the "P A G I N G" section with thousands of lines like the
following:

/*Exploring store .cro.plantid.1074918.gwid.33300.device_profile.02 folder =
/app/mqtt/activeMQ/brokers/eltbroker01/./data/paging/5ac5a57d-dcc1-11e6-a52b-069133a4f5c
f
******* Page 1
17:40:26,550 DEBUG [org.apache.activemq.artemis.core.paging.impl.Page]
reading page 1 on address =
.cro.plantid.1074918.gwid.33300.device_profile.02
pg=1, msg=0,pgTX=-1,userMessageID=,
msg=ServerMessage[messageID=1034283,durable=true,userID=null,priority=0,
bodySize=2048, timestamp=0,expiration=0, durable=true,
address=.cro.plantid.1074918.gwid.33300.device_profile.02,properties=TypedProperties[mqtt.message.retain=true,mqtt.qos.level=2]]@1029639001,Queues
= 1034178
pg=1, msg=1,pgTX=-1,userMessageID=,
msg=ServerMessage[messageID=1036477,durable=true,userID=null,priority=0,
bodySize=2048, timestamp=0,expiration=0, durable=true,
address=.cro.plantid.1074918.gwid.33300.device_profile.02,properties=TypedProperties[mqtt.message.retain=true,mqtt.qos.level=2]]@745530122,Queues
= 1036418,103417
8,1035409
pg=1, msg=2,pgTX=-1,userMessageID=,
msg=ServerMessage[messageID=1038280,durable=true,userID=null,priority=0,
bodySize=2048, timestamp=0,expiration=0, durable=true,
address=.cro.plantid.1074918.gwid.33300.device_profile.02,properties=TypedProperties[mqtt.message.retain=true,mqtt.qos.level=2]]@752575846,Queues
= 1036418,103797
9,1034178,1035409,1036769,1037082
...
*/

...but, in this case for example, since on topic/address
"cro.plantid.1074918.gwid.33300.device_profile.02" travel only retained
messages, I thought that at anytime there could be at most only 1 message
per time (because the next retained will substitute the previous one)...
It's not so?
And also here: why durable true? Because QoS2? Does "durable=true" mean that
these messages will persist in memory forever?

sorry for these patches of information ...but the result of "data print"
command was too big for an upload.
Anyway, hope can help you to help me.

Francesco




--
View this message in context: http://activemq.2283324.n4.nabble.com/ARTEMIS-bad-performance-behaviour-after-7-10-days-of-usage-tp4721272p4721395.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

[Clebert Suconic]

apparently there's a bug MQTT retained queues. when you disabled
persistence on the broker we removed the capacity of validating or
debugging it.


On Mon, Jan 30, 2017 at 9:58 AM, francesco81

Post by Francesco PADOVANI
Hello,
just an update about this topic.
By disabling persistence, the broker starts again to work fast and reliable.
But just for about 2 more weeks: then the memory come bake to be full and
the broker starts to page constantly, and cpu usage goes to 100%.
I tried to run the command "artemis data print" as you suggested, but I have
to admit my ignorance to read the results.
I did redirect the command result to a file and it produced about 758 MB of
data.
unavailable
17:39:47,215 FINE [io.netty.util.internal.PlatformDependent] You don't have
Javassist in your class path or you don't have enough permission to load
dynamically generated classes. Please check the configuration for better
performance.*/
...I don't know how much this will weigh on performances.
Then I have a "B I N D I N G S J O U R N A L" section with thousands of
[id=110,
name=b7478faf-f0fd-481a-93ef-b6109dc3a689..cro.plantid.1073403.gwid.25324.device_profile.04.set,
address=.cro.plantid.1073403.gwid.25324.device_profile.04.set,
filterString=NOT ((AMQAddress = 'jms.queue.activemq.management') OR
(AMQAddress = 'activemq.notifications')), user=gateway, autoCreated=false]
...where "cro.plantid.1073403.gwid.25324.device_profile.04.set" is an
example of a topic for a specific plant (there are about 48 topic for
plant).
What does "isUpdate=false;compactCount=0;PersistentQueueBindingEncoding"
mean?
Then I have a "### Surviving Records Summary ###" section, again with
/*recordID=5205;userRecordType=21;isUpdate=false;compactCount=0;PersistentQueueBindingEncoding
[id=5205, name=1447f6a659bc9bc0df2e55..cro.plantid.1071617.gwid.30889.
device_profile.02.err,
address=.cro.plantid.1071617.gwid.30889.device_profile.02.err,
filterString=NOT ((AMQAddress = 'jms.queue.activemq.management') OR
(AMQAddre
ss = 'activemq.notifications')), user=gateway, autoCreated=false]*/
Then I have the section "### Prepared TX ###", which is empty, and the ###
message count=0
message reference count
prepared message count=0*/
Then I have the "M E S S A G E S J O U R N A L" section with thousands of
ll,priority=0, bodySize=512, timestamp=0,expiration=0, durable=true,
address=$sys.mqtt.retain..cro.plantid.1073620.gwid.32646.device_state.err,properties=TypedProp
[queueID=65]
[queueID=9747]
[queueID=7774]
... and no deleteRecord operation for that id (10088): I don't know if it is
right.
Then I have again the "### Surviving Records Summary ###" with thousands of
line like the followings;
recordID=196;userRecordType=31;isUpdate=false;compactCount=0;Message(messageID=196;msg=ServerMessage[messageID=196,durable=true,userID=null,priority=0,
bodySize=2048, timestamp=0,expiration=0, durable=true,
recordID=196;userRecordType=32;isUpdate=true;compactCount=0;AddRef;QueueEncoding
[queueID=65]
...again my ignorance: why "durable=true"? because retained? because qos2?
for the MESSAGES JOURNAL I have the ### Message Counts ### with thousands of
message count=55958
message reference count
queue id 2,count=11
queue id 196608,count=2
queue id 4,count=1
queue id 196619,count=4
queue id 65547,count=1
queue id 196625,count=2
queue id 196630,count=2
queue id 65559,count=1
queue id 1081351,count=1
queue id 163869,count=17
queue id 163875,count=2307
... what does this mean?
Finally I have the "P A G I N G" section with thousands of lines like the
/*Exploring store .cro.plantid.1074918.gwid.33300.device_profile.02 folder =
/app/mqtt/activeMQ/brokers/eltbroker01/./data/paging/5ac5a57d-dcc1-11e6-a52b-069133a4f5c
f
******* Page 1
17:40:26,550 DEBUG [org.apache.activemq.artemis.core.paging.impl.Page]
reading page 1 on address =
.cro.plantid.1074918.gwid.33300.device_profile.02
pg=1, msg=0,pgTX=-1,userMessageID=,
msg=ServerMessage[messageID=1034283,durable=true,userID=null,priority=0,
bodySize=2048, timestamp=0,expiration=0, durable=true,
= 1034178
pg=1, msg=1,pgTX=-1,userMessageID=,
msg=ServerMessage[messageID=1036477,durable=true,userID=null,priority=0,
bodySize=2048, timestamp=0,expiration=0, durable=true,
= 1036418,103417
8,1035409
pg=1, msg=2,pgTX=-1,userMessageID=,
msg=ServerMessage[messageID=1038280,durable=true,userID=null,priority=0,
bodySize=2048, timestamp=0,expiration=0, durable=true,
= 1036418,103797
9,1034178,1035409,1036769,1037082
...
*/
...but, in this case for example, since on topic/address
"cro.plantid.1074918.gwid.33300.device_profile.02" travel only retained
messages, I thought that at anytime there could be at most only 1 message
per time (because the next retained will substitute the previous one)...
It's not so?
And also here: why durable true? Because QoS2? Does "durable=true" mean that
these messages will persist in memory forever?
sorry for these patches of information ...but the result of "data print"
command was too big for an upload.
Anyway, hope can help you to help me.
Francesco
--
View this message in context: http://activemq.2283324.n4.nabble.com/ARTEMIS-bad-performance-behaviour-after-7-10-days-of-usage-tp4721272p4721395.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

[francesco81]

Hi Clebert,
good to know it.
But, just to be clear, I have the same problem also with persistence enabled
(indeed, it's even worse).
Regardless of the persistence, the behavior is the same: after a certain
window of usage, ram fills and the broker starts to work constantly in
paging mode (cpu --> 100%), becoming almost unusable.
- If persistence is disabled, I can restart the broker and it comes back to
work fast (ram is free) ..but the problem, of course, is only postponed for
another window of usage (currently about 10 days, for my use case).
- If persistence is enabled, I can restart the broker, but it starts to
retrieve all stored messages from disk, ram fills quickly again, it returns
to work in paging mode and cpu goes to 100%. Therefore I'm pratically
obliged to disable persistence.

Could it be something related to a misconfiguration of my broker instance? I
notice also that all messages witha QoS > 0 (regardless whether retained or
not) have the property durable=true: does this mean that all those messages
will persist in memory forever? There is no one else who is facing the same
issue by using Artemis as mqtt broker?

Following is my "address-settings" configuration section within broker.xml
file. It's the only one that I changed in the standard configuration (apart
the acceptor for mqtt of course). Hope can help:

* <address-settings>

<address-setting match="#">
<dead-letter-address>jms.queue.DLQ</dead-letter-address>
<expiry-address>jms.queue.ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>

<max-size-bytes>-1</max-size-bytes>

<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<last-value-queue>true</last-value-queue>
</address-setting>
</address-settings>*

Francesco



--
View this message in context: http://activemq.2283324.n4.nabble.com/ARTEMIS-bad-performance-behaviour-after-7-10-days-of-usage-tp4721272p4721417.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

[Clebert Suconic]

The bug wouldn't be on the persistence layer itself. The MQTT protocol
manager is managing retainers by removing (acking) older messages. And
something is broken. I'm not sure if it's a general case or something
specifically to your use case. I was wondering if you could replicate
the issue on a simple testsuite...


Martyn has done a lot of work around the MQTT protocol manager. I know
him personally and he's not around today, so I'm wondering if he will
be able to pick up this issue tomorrow.

but if you had a test replicating the issue it would help anyone to
jump on the issue quicker.


On Tue, Jan 31, 2017 at 6:27 AM, francesco81

Post by francesco81
Hi Clebert,
good to know it.
But, just to be clear, I have the same problem also with persistence enabled
(indeed, it's even worse).
Regardless of the persistence, the behavior is the same: after a certain
window of usage, ram fills and the broker starts to work constantly in
paging mode (cpu --> 100%), becoming almost unusable.
- If persistence is disabled, I can restart the broker and it comes back to
work fast (ram is free) ..but the problem, of course, is only postponed for
another window of usage (currently about 10 days, for my use case).
- If persistence is enabled, I can restart the broker, but it starts to
retrieve all stored messages from disk, ram fills quickly again, it returns
to work in paging mode and cpu goes to 100%. Therefore I'm pratically
obliged to disable persistence.
Could it be something related to a misconfiguration of my broker instance? I
notice also that all messages witha QoS > 0 (regardless whether retained or
not) have the property durable=true: does this mean that all those messages
will persist in memory forever? There is no one else who is facing the same
issue by using Artemis as mqtt broker?
Following is my "address-settings" configuration section within broker.xml
file. It's the only one that I changed in the standard configuration (apart
* <address-settings>
<address-setting match="#">
<dead-letter-address>jms.queue.DLQ</dead-letter-address>
<expiry-address>jms.queue.ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<max-size-bytes>-1</max-size-bytes>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<address-full-policy>PAGE</address-full-policy>
<last-value-queue>true</last-value-queue>
</address-setting>
</address-settings>*
Francesco
--
View this message in context: http://activemq.2283324.n4.nabble.com/ARTEMIS-bad-performance-behaviour-after-7-10-days-of-usage-tp4721272p4721417.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

[francesco81]

Ok, perfect.
I'd be very happy if Martyn could take a look on this.
Meanwile, I'll try to put in place a test environment on AWS, accessible to
everyone of the community ...this way I hope you'll be able to help me to
debug and solve the problem.

Thanks

Francesco



--
View this message in context: http://activemq.2283324.n4.nabble.com/ARTEMIS-bad-performance-behaviour-after-7-10-days-of-usage-tp4721272p4721470.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

[Clebert Suconic]

I actually thought about a self contained test, that we can use to
debug at our dev environment.

Putting something complex to be executed over the AWS wouldn't help
much. We would still need to figure out your whole app framework.

Why don't you use some of the examples infra-structure to create such
thing? if you want to do an UnitTest on the integration testsuite..
it's even better.

On Wed, Feb 1, 2017 at 8:02 AM, francesco81

Post by francesco81
Ok, perfect.
I'd be very happy if Martyn could take a look on this.
Meanwile, I'll try to put in place a test environment on AWS, accessible to
everyone of the community ...this way I hope you'll be able to help me to
debug and solve the problem.
Thanks
Francesco
--
View this message in context: http://activemq.2283324.n4.nabble.com/ARTEMIS-bad-performance-behaviour-after-7-10-days-of-usage-tp4721272p4721470.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

[francesco81]

Hi Clebert,
ok, we will build some junit tests to replicate the problem, follwing the
exemple section.
Meanwhile, I can confirm to you the bug(s): it seems that the reference to
the message is never removed from the queue. If the message is not retained
it persists 1 reference on the queue and every time my client reconnect to
the broker it will receive always the same message. If the message is
retained is even worse, because it goes to increment the references on the
queue and every time the client connects to the broker it will receive n + 1
times the same retained message where n is the re-connection number n.
Following a simple test I made by using mosquitto client:

CLIENT 1 (publisher):
mosquitto_pub -h <host> -q 1 -i <client_id_pub> -t
/cro/plantid/0000000/gwid/00001/device_state -m "test non retained"

CLIENT 2 (subscriber):
1 connection:
mosquitto_sub -h <host> -q 2 -i <client_id_sub> -t
/cro/plantid/0000000/gwid/00001/device_state
--> received:
*test non retained*
2 connection:
mosquitto_sub -h <host> -q 2 -i <client_id_sub> -t
/cro/plantid/0000000/gwid/00001/device_state
--> received again:
*test non retained*
3 connection:
mosquitto_sub -h <host> -q 2 -i <client_id_sub> -t
/cro/plantid/0000000/gwid/00001/device_state
--> received again:
*test non retained*
...
and so on

CLIENT 1 (publisher):
mosquitto_pub -h <host> -q 1 -i <client_id_pub> -r -t
/cro/plantid/0000000/gwid/00001/device_state -m "test retained"

CLIENT 2 (subscriber):
4 connection:
mosquitto_sub -h <host> -q 2 -i <client_id_sub> -t
/cro/plantid/0000000/gwid/00001/device_state
--> received:
*test retained
test non retained
test retained*
5 connection:
mosquitto_sub -h <host> -q 2 -i <client_id_sub> -t
/cro/plantid/0000000/gwid/00001/device_state
--> received:
*test retained
test non retained
test retained
test retained*
6 connection:
mosquitto_sub -h <host> -q 2 -i <client_id_sub> -t
/cro/plantid/0000000/gwid/00001/device_state
--> received:
*test non retained
test retained
test retained
test retained
test retained*
...
and so on.

If you agree, I would open a bug for this.

Francesco



--
View this message in context: http://activemq.2283324.n4.nabble.com/ARTEMIS-bad-performance-behaviour-after-7-10-days-of-usage-tp4721272p4721562.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

[Martyn Taylor]

Francesco,

Thanks for providing this information. I think there's a couple of things
going on and I'd like to work with you to create some reproducers, we could
have a chat IRC #apache-***@freenode.net and I'll see if I can
recreate your env / case in our test suite.

With the case above, the mosquitto client is sending a subscribe frame on
every connection. Artemis is then treating this as a new subscription
(even though the subscription queue already exists), and is copying over
the retain messages. It's a reasonably trivial change, I should have a fix
for you shortly.

Thanks
Martyn

Post by francesco81
Hi Clebert,
ok, we will build some junit tests to replicate the problem, follwing the
exemple section.
Meanwhile, I can confirm to you the bug(s): it seems that the reference to
the message is never removed from the queue. If the message is not retained
it persists 1 reference on the queue and every time my client reconnect to
the broker it will receive always the same message. If the message is
retained is even worse, because it goes to increment the references on the
queue and every time the client connects to the broker it will receive n + 1
times the same retained message where n is the re-connection number n.
mosquitto_pub -h <host> -q 1 -i <client_id_pub> -t
/cro/plantid/0000000/gwid/00001/device_state -m "test non retained"
mosquitto_sub -h <host> -q 2 -i <client_id_sub> -t
/cro/plantid/0000000/gwid/00001/device_state
*test non retained*
mosquitto_sub -h <host> -q 2 -i <client_id_sub> -t
/cro/plantid/0000000/gwid/00001/device_state
*test non retained*
mosquitto_sub -h <host> -q 2 -i <client_id_sub> -t
/cro/plantid/0000000/gwid/00001/device_state
*test non retained*
...
and so on
mosquitto_pub -h <host> -q 1 -i <client_id_pub> -r -t
/cro/plantid/0000000/gwid/00001/device_state -m "test retained"
mosquitto_sub -h <host> -q 2 -i <client_id_sub> -t
/cro/plantid/0000000/gwid/00001/device_state
*test retained
test non retained
test retained*
mosquitto_sub -h <host> -q 2 -i <client_id_sub> -t
/cro/plantid/0000000/gwid/00001/device_state
*test retained
test non retained
test retained
test retained*
mosquitto_sub -h <host> -q 2 -i <client_id_sub> -t
/cro/plantid/0000000/gwid/00001/device_state
*test non retained
test retained
test retained
test retained
test retained*
...
and so on.
If you agree, I would open a bug for this.
Francesco
--
View this message in context: http://activemq.2283324.n4.
nabble.com/ARTEMIS-bad-performance-behaviour-after-7-10-days-of-usage-
tp4721272p4721562.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

[francesco81]

Hi Martyn,
I'll be happy to enjoy the IRC chat as soon as I can.
Effectively, your words about the "treating as new subscription" would
explain the issue with retained messages.
However there's still something that I don't understand: for example why
also the non-retained messages are resent on resubscription. Regardless if
retained or not, it seems that all messages persist on queue (or, in case of
retained, they persist on address and one more reference is added on queue
at each resubscription... I don't know, I'm trying to guess) and are never
discarded.
How are they managed by artemis? In a scenario where clients never connect
with "cleanSession=false" (that is our usecase), I thought:
- non-retained messages are removed from address (and references from
queues) once they are consumed by clients (or immediatly if no client is
connected).
- retained messages are removed from address when a new one arrives (and the
reference to the new one substitutes the previous on the queue).
Is not so? I'm wrong somewhere?

Again, thanks for your patience.

Francesco



--
View this message in context: http://activemq.2283324.n4.nabble.com/ARTEMIS-bad-performance-behaviour-after-7-10-days-of-usage-tp4721272p4721678.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

[Martyn Taylor]

Francesco,

I think I've identified the cause of this problem. There were two issues
which are now fixed as part of:
https://github.com/apache/activemq-artemis/pull/1002

I'll get these fixes cherry-picked onto Artemis 1.x stream.

I plan on doing a 1.5.3 (with these changes included) within the next
couple of days.

Cheers
Martyn

Post by francesco81
Hi Martyn,
I'll be happy to enjoy the IRC chat as soon as I can.
Effectively, your words about the "treating as new subscription" would
explain the issue with retained messages.
However there's still something that I don't understand: for example why
also the non-retained messages are resent on resubscription. Regardless if
retained or not, it seems that all messages persist on queue (or, in case of
retained, they persist on address and one more reference is added on queue
at each resubscription... I don't know, I'm trying to guess) and are never
discarded.
How are they managed by artemis? In a scenario where clients never connect
- non-retained messages are removed from address (and references from
queues) once they are consumed by clients (or immediatly if no client is
connected).
- retained messages are removed from address when a new one arrives (and the
reference to the new one substitutes the previous on the queue).
Is not so? I'm wrong somewhere?
Again, thanks for your patience.
Francesco
--
View this message in context: http://activemq.2283324.n4.
nabble.com/ARTEMIS-bad-performance-behaviour-after-7-10-days-of-usage-
tp4721272p4721678.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

[Francesco PADOVANI]

Hi Martyn,

Thanks a lot for this!

It could save us just in time...


As soon as I can, I'll try to build artemis from your last commit and make some tests.

I'll give you a feedback.


Francesco

________________________________
From: Martyn Taylor <***@redhat.com>
Sent: Thursday, February 9, 2017 1:22:52 PM
To: ***@activemq.apache.org
Subject: Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage

Francesco,

I think I've identified the cause of this problem. There were two issues
which are now fixed as part of:
https://github.com/apache/activemq-artemis/pull/1002

I'll get these fixes cherry-picked onto Artemis 1.x stream.

I plan on doing a 1.5.3 (with these changes included) within the next
couple of days.

Cheers
Martyn

Post by francesco81
Hi Martyn,
I'll be happy to enjoy the IRC chat as soon as I can.
Effectively, your words about the "treating as new subscription" would
explain the issue with retained messages.
However there's still something that I don't understand: for example why
also the non-retained messages are resent on resubscription. Regardless if
retained or not, it seems that all messages persist on queue (or, in case
of
retained, they persist on address and one more reference is added on queue
at each resubscription... I don't know, I'm trying to guess) and are never
discarded.
How are they managed by artemis? In a scenario where clients never connect
- non-retained messages are removed from address (and references from
queues) once they are consumed by clients (or immediatly if no client is
connected).
- retained messages are removed from address when a new one arrives (and
the
reference to the new one substitutes the previous on the queue).
Is not so? I'm wrong somewhere?
Again, thanks for your patience.
Francesco
--
View this message in context: http://activemq.2283324.n4.
nabble.com/ARTEMIS-bad-performance-behaviour-after-7-10-days-of-usage-
tp4721272p4721678.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

________________________________

Ce message, ainsi que tous les fichiers joints ? ce message, peuvent contenir des informations sensibles et/ ou confidentielles ne devant pas ?tre divulgu?es. Si vous n'?tes pas le destinataire de ce message (ou que vous recevez ce message par erreur), nous vous remercions de le notifier imm?diatement ? son exp?diteur, et de d?truire ce message. Toute copie, divulgation, modification, utilisation ou diffusion, non autoris?e, directe ou indirecte, de tout ou partie de ce message, est strictement interdite.


This e-mail, and any document attached hereby, may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized, direct or indirect, copying, disclosure, distribution or other use of the material or parts thereof is strictly forbidden.

[Martyn Taylor]

On Thu, Feb 9, 2017 at 3:54 PM, Francesco PADOVANI <

Post by francesco81
Hi Martyn,
Thanks a lot for this!
It could save us just in time...

I hope so.

Post by francesco81
As soon as I can, I'll try to build artemis from your last commit and make some tests.

I've created a snapshot build you can test with:

https://repository.apache.org/content/groups/snapshots/org/apache/activemq/apache-artemis/1.6.0-SNAPSHOT/apache-artemis-1.6.0-20170209.161917-15-bin.zip

Please let me know how you get on.

Thanks

Post by francesco81
Francesco
________________________________
Sent: Thursday, February 9, 2017 1:22:52 PM
Subject: Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage
Francesco,
I think I've identified the cause of this problem. There were two issues
https://github.com/apache/activemq-artemis/pull/1002
I'll get these fixes cherry-picked onto Artemis 1.x stream.
I plan on doing a 1.5.3 (with these changes included) within the next
couple of days.
Cheers
Martyn
On Tue, Feb 7, 2017 at 10:33 AM, francesco81 <

Post by francesco81
Hi Martyn,
I'll be happy to enjoy the IRC chat as soon as I can.
Effectively, your words about the "treating as new subscription" would
explain the issue with retained messages.
However there's still something that I don't understand: for example why
also the non-retained messages are resent on resubscription. Regardless

if

Post by francesco81
retained or not, it seems that all messages persist on queue (or, in case of
retained, they persist on address and one more reference is added on

queue

Post by francesco81
at each resubscription... I don't know, I'm trying to guess) and are

never

Post by francesco81
discarded.
How are they managed by artemis? In a scenario where clients never

connect

Post by francesco81
- non-retained messages are removed from address (and references from
queues) once they are consumed by clients (or immediatly if no client is
connected).
- retained messages are removed from address when a new one arrives (and the
reference to the new one substitutes the previous on the queue).
Is not so? I'm wrong somewhere?
Again, thanks for your patience.
Francesco
--
View this message in context: http://activemq.2283324.n4.
nabble.com/ARTEMIS-bad-performance-behaviour-after-7-10-days-of-usage-
tp4721272p4721678.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

________________________________
Ce message, ainsi que tous les fichiers joints ? ce message, peuvent
contenir des informations sensibles et/ ou confidentielles ne devant pas
?tre divulgu?es. Si vous n'?tes pas le destinataire de ce message (ou que
vous recevez ce message par erreur), nous vous remercions de le notifier
imm?diatement ? son exp?diteur, et de d?truire ce message. Toute copie,
divulgation, modification, utilisation ou diffusion, non autoris?e, directe
ou indirecte, de tout ou partie de ce message, est strictement interdite.
This e-mail, and any document attached hereby, may contain confidential
and/or privileged information. If you are not the intended recipient (or
have received this e-mail in error) please notify the sender immediately
and destroy this e-mail. Any unauthorized, direct or indirect, copying,
disclosure, distribution or other use of the material or parts thereof is
strictly forbidden.

[Francesco PADOVANI]

Hi Martyn,

we're testing your 1.6.0 snapshot.

The issues related to retained messages ACK and durable queues seem ok now. Great!

I noticed that this snapshot doesn't include the fix for weird chars when messages are recovered from journal: do you think it will be included in 1.5.3 release?


And last question: with artemis 1.5.1 we were able to connect also by using clientID longer than 23 chars. Now, with 1.6.0 (and maybe also with 1.5.2 ...I admit I've not yet tested it) we can't.

It is changed something about clientid length support?


Thanks in advance.


Francesco

________________________________
From: Martyn Taylor <***@redhat.com>
Sent: Thursday, February 9, 2017 5:36:19 PM
To: ***@activemq.apache.org
Subject: Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage

On Thu, Feb 9, 2017 at 3:54 PM, Francesco PADOVANI <

Post by francesco81
Hi Martyn,
Thanks a lot for this!
It could save us just in time...

I hope so.

Post by francesco81
As soon as I can, I'll try to build artemis from your last commit and make
some tests.

I've created a snapshot build you can test with:

https://repository.apache.org/content/groups/snapshots/org/apache/activemq/apache-artemis/1.6.0-SNAPSHOT/apache-artemis-1.6.0-20170209.161917-15-bin.zip

Please let me know how you get on.

Thanks

Post by francesco81
Francesco
________________________________
Sent: Thursday, February 9, 2017 1:22:52 PM
Subject: Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage
Francesco,
I think I've identified the cause of this problem. There were two issues
https://github.com/apache/activemq-artemis/pull/1002
I'll get these fixes cherry-picked onto Artemis 1.x stream.
I plan on doing a 1.5.3 (with these changes included) within the next
couple of days.
Cheers
Martyn
On Tue, Feb 7, 2017 at 10:33 AM, francesco81 <

Post by francesco81
Hi Martyn,
I'll be happy to enjoy the IRC chat as soon as I can.
Effectively, your words about the "treating as new subscription" would
explain the issue with retained messages.
However there's still something that I don't understand: for example why
also the non-retained messages are resent on resubscription. Regardless

if

Post by francesco81
retained or not, it seems that all messages persist on queue (or, in case of
retained, they persist on address and one more reference is added on

queue

Post by francesco81
at each resubscription... I don't know, I'm trying to guess) and are

never

Post by francesco81
discarded.
How are they managed by artemis? In a scenario where clients never

connect

Post by francesco81
- non-retained messages are removed from address (and references from
queues) once they are consumed by clients (or immediatly if no client is
connected).
- retained messages are removed from address when a new one arrives (and the
reference to the new one substitutes the previous on the queue).
Is not so? I'm wrong somewhere?
Again, thanks for your patience.
Francesco
--
View this message in context: http://activemq.2283324.n4.
nabble.com/ARTEMIS-bad-performance-behaviour-after-7-10-days-of-usage-
tp4721272p4721678.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

________________________________
Ce message, ainsi que tous les fichiers joints ? ce message, peuvent
contenir des informations sensibles et/ ou confidentielles ne devant pas
?tre divulgu?es. Si vous n'?tes pas le destinataire de ce message (ou que
vous recevez ce message par erreur), nous vous remercions de le notifier
imm?diatement ? son exp?diteur, et de d?truire ce message. Toute copie,
divulgation, modification, utilisation ou diffusion, non autoris?e, directe
ou indirecte, de tout ou partie de ce message, est strictement interdite.
This e-mail, and any document attached hereby, may contain confidential
and/or privileged information. If you are not the intended recipient (or
have received this e-mail in error) please notify the sender immediately
and destroy this e-mail. Any unauthorized, direct or indirect, copying,
disclosure, distribution or other use of the material or parts thereof is
strictly forbidden.

________________________________

Ce message, ainsi que tous les fichiers joints ? ce message, peuvent contenir des informations sensibles et/ ou confidentielles ne devant pas ?tre divulgu?es. Si vous n'?tes pas le destinataire de ce message (ou que vous recevez ce message par erreur), nous vous remercions de le notifier imm?diatement ? son exp?diteur, et de d?truire ce message. Toute copie, divulgation, modification, utilisation ou diffusion, non autoris?e, directe ou indirecte, de tout ou partie de ce message, est strictement interdite.


This e-mail, and any document attached hereby, may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized, direct or indirect, copying, disclosure, distribution or other use of the material or parts thereof is strictly forbidden.

[Francesco PADOVANI]

Sorry Martyn,

about the clientID lenght: it was a problem of my mqtt Client which had the compatibility with mqtt 3.1.1 disabled (stupid me).


Instead, I confirm to you the lack of fixing for weird chars on messages recovered from journal.


Francesco

________________________________
From: Francesco PADOVANI
Sent: Friday, February 10, 2017 2:58:19 PM
To: ***@activemq.apache.org
Subject: Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage


Hi Martyn,

we're testing your 1.6.0 snapshot.

The issues related to retained messages ACK and durable queues seem ok now. Great!

I noticed that this snapshot doesn't include the fix for weird chars when messages are recovered from journal: do you think it will be included in 1.5.3 release?


And last question: with artemis 1.5.1 we were able to connect also by using clientID longer than 23 chars. Now, with 1.6.0 (and maybe also with 1.5.2 ...I admit I've not yet tested it) we can't.

It is changed something about clientid length support?


Thanks in advance.


Francesco

________________________________
From: Martyn Taylor <***@redhat.com>
Sent: Thursday, February 9, 2017 5:36:19 PM
To: ***@activemq.apache.org
Subject: Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage

On Thu, Feb 9, 2017 at 3:54 PM, Francesco PADOVANI <

Post by francesco81
Hi Martyn,
Thanks a lot for this!
It could save us just in time...

I hope so.

Post by francesco81
As soon as I can, I'll try to build artemis from your last commit and make
some tests.

I've created a snapshot build you can test with:

https://repository.apache.org/content/groups/snapshots/org/apache/activemq/apache-artemis/1.6.0-SNAPSHOT/apache-artemis-1.6.0-20170209.161917-15-bin.zip

Please let me know how you get on.

Thanks

Post by francesco81
Francesco
________________________________
Sent: Thursday, February 9, 2017 1:22:52 PM
Subject: Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage
Francesco,
I think I've identified the cause of this problem. There were two issues
https://github.com/apache/activemq-artemis/pull/1002
I'll get these fixes cherry-picked onto Artemis 1.x stream.
I plan on doing a 1.5.3 (with these changes included) within the next
couple of days.
Cheers
Martyn
On Tue, Feb 7, 2017 at 10:33 AM, francesco81 <

Post by francesco81
Hi Martyn,
I'll be happy to enjoy the IRC chat as soon as I can.
Effectively, your words about the "treating as new subscription" would
explain the issue with retained messages.
However there's still something that I don't understand: for example why
also the non-retained messages are resent on resubscription. Regardless

if

Post by francesco81
retained or not, it seems that all messages persist on queue (or, in case of
retained, they persist on address and one more reference is added on

queue

Post by francesco81
at each resubscription... I don't know, I'm trying to guess) and are

never

Post by francesco81
discarded.
How are they managed by artemis? In a scenario where clients never

connect

Post by francesco81
- non-retained messages are removed from address (and references from
queues) once they are consumed by clients (or immediatly if no client is
connected).
- retained messages are removed from address when a new one arrives (and the
reference to the new one substitutes the previous on the queue).
Is not so? I'm wrong somewhere?
Again, thanks for your patience.
Francesco
--
View this message in context: http://activemq.2283324.n4.
nabble.com/ARTEMIS-bad-performance-behaviour-after-7-10-days-of-usage-
tp4721272p4721678.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

________________________________
Ce message, ainsi que tous les fichiers joints ? ce message, peuvent
contenir des informations sensibles et/ ou confidentielles ne devant pas
?tre divulgu?es. Si vous n'?tes pas le destinataire de ce message (ou que
vous recevez ce message par erreur), nous vous remercions de le notifier
imm?diatement ? son exp?diteur, et de d?truire ce message. Toute copie,
divulgation, modification, utilisation ou diffusion, non autoris?e, directe
ou indirecte, de tout ou partie de ce message, est strictement interdite.
This e-mail, and any document attached hereby, may contain confidential
and/or privileged information. If you are not the intended recipient (or
have received this e-mail in error) please notify the sender immediately
and destroy this e-mail. Any unauthorized, direct or indirect, copying,
disclosure, distribution or other use of the material or parts thereof is
strictly forbidden.

________________________________

Ce message, ainsi que tous les fichiers joints ? ce message, peuvent contenir des informations sensibles et/ ou confidentielles ne devant pas ?tre divulgu?es. Si vous n'?tes pas le destinataire de ce message (ou que vous recevez ce message par erreur), nous vous remercions de le notifier imm?diatement ? son exp?diteur, et de d?truire ce message. Toute copie, divulgation, modification, utilisation ou diffusion, non autoris?e, directe ou indirecte, de tout ou partie de ce message, est strictement interdite.


This e-mail, and any document attached hereby, may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized, direct or indirect, copying, disclosure, distribution or other use of the material or parts thereof is strictly forbidden.

[Martyn Taylor]

Looks like I made a bit of a git foobar when cherry-picking the commits
from master on the 1.x branch. I will rectify this. Thanks for pointing
this out.

On Fri, Feb 10, 2017 at 2:06 PM, Francesco PADOVANI <

Post by Francesco PADOVANI
Sorry Martyn,
about the clientID lenght: it was a problem of my mqtt Client which had
the compatibility with mqtt 3.1.1 disabled (stupid me).
Instead, I confirm to you the lack of fixing for weird chars on messages
recovered from journal.
Francesco
________________________________
From: Francesco PADOVANI
Sent: Friday, February 10, 2017 2:58:19 PM
Subject: Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage
Hi Martyn,
we're testing your 1.6.0 snapshot.
The issues related to retained messages ACK and durable queues seem ok now. Great!
I noticed that this snapshot doesn't include the fix for weird chars when
messages are recovered from journal: do you think it will be included in
1.5.3 release?
And last question: with artemis 1.5.1 we were able to connect also by
using clientID longer than 23 chars. Now, with 1.6.0 (and maybe also with
1.5.2 ...I admit I've not yet tested it) we can't.
It is changed something about clientid length support?
Thanks in advance.
Francesco
________________________________
Sent: Thursday, February 9, 2017 5:36:19 PM
Subject: Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage
On Thu, Feb 9, 2017 at 3:54 PM, Francesco PADOVANI <

Post by francesco81
Hi Martyn,
Thanks a lot for this!
It could save us just in time...

I hope so.

Post by francesco81
As soon as I can, I'll try to build artemis from your last commit and

make

Post by francesco81
some tests.

https://repository.apache.org/content/groups/snapshots/org/
apache/activemq/apache-artemis/1.6.0-SNAPSHOT/apache-
artemis-1.6.0-20170209.161917-15-bin.zip
Please let me know how you get on.
Thanks

Post by francesco81
Francesco
________________________________
Sent: Thursday, February 9, 2017 1:22:52 PM
Subject: Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage
Francesco,
I think I've identified the cause of this problem. There were two issues
https://github.com/apache/activemq-artemis/pull/1002
I'll get these fixes cherry-picked onto Artemis 1.x stream.
I plan on doing a 1.5.3 (with these changes included) within the next
couple of days.
Cheers
Martyn
On Tue, Feb 7, 2017 at 10:33 AM, francesco81 <

Post by francesco81
Hi Martyn,
I'll be happy to enjoy the IRC chat as soon as I can.
Effectively, your words about the "treating as new subscription" would
explain the issue with retained messages.
However there's still something that I don't understand: for example

why

Post by francesco81

Post by francesco81
also the non-retained messages are resent on resubscription. Regardless

if

Post by francesco81
retained or not, it seems that all messages persist on queue (or, in

case

Post by francesco81

Post by francesco81
of
retained, they persist on address and one more reference is added on

queue

Post by francesco81
at each resubscription... I don't know, I'm trying to guess) and are

never

Post by francesco81
discarded.
How are they managed by artemis? In a scenario where clients never

connect

Post by francesco81
- non-retained messages are removed from address (and references from
queues) once they are consumed by clients (or immediatly if no client

is

Post by francesco81

Post by francesco81
connected).
- retained messages are removed from address when a new one arrives

(and

Post by francesco81

Post by francesco81
the
reference to the new one substitutes the previous on the queue).
Is not so? I'm wrong somewhere?
Again, thanks for your patience.
Francesco
--
View this message in context: http://activemq.2283324.n4.
nabble.com/ARTEMIS-bad-performance-behaviour-after-7-10-days-of-usage-
tp4721272p4721678.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

________________________________
Ce message, ainsi que tous les fichiers joints ? ce message, peuvent
contenir des informations sensibles et/ ou confidentielles ne devant pas
?tre divulgu?es. Si vous n'?tes pas le destinataire de ce message (ou que
vous recevez ce message par erreur), nous vous remercions de le notifier
imm?diatement ? son exp?diteur, et de d?truire ce message. Toute copie,
divulgation, modification, utilisation ou diffusion, non autoris?e,

directe

Post by francesco81
ou indirecte, de tout ou partie de ce message, est strictement interdite.
This e-mail, and any document attached hereby, may contain confidential
and/or privileged information. If you are not the intended recipient (or
have received this e-mail in error) please notify the sender immediately
and destroy this e-mail. Any unauthorized, direct or indirect, copying,
disclosure, distribution or other use of the material or parts thereof is
strictly forbidden.

________________________________
Ce message, ainsi que tous les fichiers joints ? ce message, peuvent
contenir des informations sensibles et/ ou confidentielles ne devant pas
?tre divulgu?es. Si vous n'?tes pas le destinataire de ce message (ou que
vous recevez ce message par erreur), nous vous remercions de le notifier
imm?diatement ? son exp?diteur, et de d?truire ce message. Toute copie,
divulgation, modification, utilisation ou diffusion, non autoris?e, directe
ou indirecte, de tout ou partie de ce message, est strictement interdite.
This e-mail, and any document attached hereby, may contain confidential
and/or privileged information. If you are not the intended recipient (or
have received this e-mail in error) please notify the sender immediately
and destroy this e-mail. Any unauthorized, direct or indirect, copying,
disclosure, distribution or other use of the material or parts thereof is
strictly forbidden.

[francesco81]

Hi Martyn,
good Monday.

Unfortunately, I've an update about our tests on artemis snapshot 1.6 you g=
ave me last week.
It seems that after few time of use (sometimes an hour ...other times less)=
it stops to accept connections. Resources on server are totally ok: we hav=
e about 40/50 clients, each one who sends 2/3 messages per minute., and cpu=
and mem consumption is really low.
But I don't know why... at a certain point, clients can't connect any more =
to the Artemis broker. And I don't see any error on log (set to DEBUG level=
). Simply, whoever try to connect, stays in "CONNECT" status forever (perfo=
rming retry attempts).
I've done a thread dump of JVM in that situation: you can find it attached.
You can see there are few threads that are BLOCKED, waiting to =
lock at "org.apache.activemq.artemis.core.protocol.mqtt.MQTTConnectionManag=
er.connect". I don't know if this can help to find the problem. Unfortunate=
ly I'm not yet able to identify the exact steps to reproduce it. I'm curren=
tly committed exactly on this. I'll let you know as s threaddumps.tdump
<http://activemq.2283324.n4.nabble.com/file/n4721908/threaddumps.tdump> oon
as I can.

Meanwhile: have you any idea?

Francesco



--
View this message in context: http://activemq.2283324.n4.nabble.com/ARTEMIS-bad-performance-behaviour-after-7-10-days-of-usage-tp4721272p4721908.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

[Martyn Taylor]

Hi Francesco,

Thanks for the Thread dump. I think I know what the issue is, I'll get it
fixed today. This fix won't make the 1.5.3 release but I can provide you
with another snapshot with all the required fixes.

We can try to do another revision release soon after including this fix.

Cheers
Martyn

Post by francesco81
Hi Martyn,
good Monday.
Unfortunately, I've an update about our tests on artemis snapshot 1.6 you g=
ave me last week.
It seems that after few time of use (sometimes an hour ...other times less)=
it stops to accept connections. Resources on server are totally ok: we hav=
e about 40/50 clients, each one who sends 2/3 messages per minute., and cpu=
and mem consumption is really low.
But I don't know why... at a certain point, clients can't connect any more =
to the Artemis broker. And I don't see any error on log (set to DEBUG level=
). Simply, whoever try to connect, stays in "CONNECT" status forever (perfo=
rming retry attempts).
I've done a thread dump of JVM in that situation: you can find it attached.
You can see there are few threads that are BLOCKED, waiting to =
lock at "org.apache.activemq.artemis.core.protocol.mqtt.
MQTTConnectionManag=
er.connect". I don't know if this can help to find the problem. Unfortunate=
ly I'm not yet able to identify the exact steps to reproduce it. I'm curren=
tly committed exactly on this. I'll let you know as s threaddumps.tdump
<http://activemq.2283324.n4.nabble.com/file/n4721908/threaddumps.tdump>
oon
as I can.
Meanwhile: have you any idea?
Francesco
--
View this message in context: http://activemq.2283324.n4.
nabble.com/ARTEMIS-bad-performance-behaviour-after-7-10-days-of-usage-
tp4721272p4721908.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

[francesco81]

Super,
you'd be so kind (as usual)!
I'll wait for a feedback on this new snapshot to test.

Again: thanks for your "huge" patience.

Francesco!




--
View this message in context: http://activemq.2283324.n4.nabble.com/ARTEMIS-bad-performance-behaviour-after-7-10-days-of-usage-tp4721272p4721911.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

[Martyn Taylor]

Thank you for your patience. This is really great feedback. I'll have
something for you by the end of today.

Regards
Martyn

Post by francesco81
Super,
you'd be so kind (as usual)!
I'll wait for a feedback on this new snapshot to test.
Again: thanks for your "huge" patience.
Francesco!
--
View this message in context: http://activemq.2283324.n4.
nabble.com/ARTEMIS-bad-performance-behaviour-after-7-10-days-of-usage-
tp4721272p4721911.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

[Martyn Taylor]

Francesco,

Do you have any logs I could look at? Do you hang out on IRC we could have
a quick chat in #apache-activemq what is your IRC nick?

Cheers

Post by Martyn Taylor
Thank you for your patience. This is really great feedback. I'll have
something for you by the end of today.
Regards
Martyn
On Mon, Feb 13, 2017 at 11:53 AM, francesco81 <

Post by francesco81
Super,
you'd be so kind (as usual)!
I'll wait for a feedback on this new snapshot to test.
Again: thanks for your "huge" patience.
Francesco!
--
View this message in context: http://activemq.2283324.n4.nab
ble.com/ARTEMIS-bad-performance-behaviour-after-7-10-days-
of-usage-tp4721272p4721911.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.