Error: zabbix gateway unable to support wildfly http-remoting service
Here is my workaround to support http-remoting in zabbix,
I have confiured zabbix server with zabbix agent to monitor production server, since we switch jboss-4.2 to wildfly-8.2 it got messed-up, zabbix unable to communicate with wildfly. the following error found the Zabbix-gateway server. The error seems like wildlfy is communicating with native RMI service.
2017-07-20 01:18:10.247 [pool-1-thread-3] WARN com.zabbix.gateway.SocketProcessor - error processing request com.zabbix.gateway.ZabbixException: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 192.168.101.197; nested exception is: java.net.ConnectException: Connection refused (Connection refused)] at com.zabbix.gateway.JMXItemChecker.getValues(JMXItemChecker.java:97) ~[zabbix-java-gateway-3.2.6.jar:na] at com.zabbix.gateway.SocketProcessor.run(SocketProcessor.java:62) ~[zabbix-java-gateway-3.2.6.jar:na] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131] Caused by: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 192.168.101.197; nested exception is: java.net.ConnectException: Connection refused (Connection refused)] at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369) ~[na:1.8.0_131] at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) ~[na:1.8.0_131] at com.zabbix.gateway.ZabbixJMXConnectorFactory$1.run(ZabbixJMXConnectorFactory.java:76) ~[zabbix-java-gateway-3.2.6.jar:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_131] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_131] ... 3 common frames omitted
I have download srcRPM https://repo.zabbix.com/zabbix/3.2/rhel/7/SRPMS/zabbix-3.2.6-1.el7.src.rpm , to check (JMXItemChecker.java:97) since it was mentioned in the error message. the same like my thought the protocol was hotcoded in the source. I have did some patch work and recompile the RPM and install Zabbix-gatway it worked as expected.
Please follow the steps which worked for me, you can download the version of zabbix server installed and reinstall the zabbix-gateway rpm.
1, Check the zabbix version installed. those installation was through yum
rpm -qa | grep zabbix zabbix-java-gateway-3.2.6-1.el7.x86_64 zabbix-server-mysql-3.2.6-1.el7.x86_64 zabbix-agent-3.2.6-1.el7.x86_64 zabbix-release-3.2-1.el7.noarch zabbix-web-3.2.6-1.el7.noarch zabbix-get-3.2.6-1.el7.x86_64 zabbix-web-mysql-3.2.6-1.el7.noarch zabbix-sender-3.2.4-1.el7.x86_64
2, Remove installed zabbix-gateway only
rpm -e zabbix-java-gateway-3.2.6-1.el7.x86_64 warning: /etc/zabbix/zabbix_java_gateway.conf saved as /etc/zabbix/zabbix_java_gateway.conf.rpmsave
3, Download and extract Source RPM
wget https://repo.zabbix.com/zabbix/3.2/rhel/7/SRPMS/zabbix-3.2.6-1.el7.src.rpm rpm -ivh zabbix-3.2.6-1.el7.src.rpm
4, Modify the source to support http-remoting
cd /root/rpmbuild/SOURCES/ tar -zxvf zabbix-3.2.6.tar.gz vi zabbix-3.2.6/src/zabbix_java/src/com/zabbix/gateway/JMXItemChecker.java
Modify with below entries, it checks the port 9990 with go to the “if” block others will go to the “else” block, so your wildlfy management port should be default port (9990) or you can modify the code as what you would like to use for wildlfy custom management port.
String conn = request.getString(JSON_TAG_CONN); int port = request.getInt(JSON_TAG_PORT); Integer remoting = new Integer("9990"); int retval = remoting.compareTo(port); if (retval == 0) { url = new JMXServiceURL("service:jmx:http-remoting-jmx://" + conn + ":" + port); } else { url = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://" + conn + ":" + port + "/jmxrmi"); } //url = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://[" + conn + "]:" + port + "/jmxrmi"); jmxc = null; mbsc = null;
Save and exit the file
5, Pack the source file which extracted to modify the .java
tar -zcvf zabbix-3.2.6.tar.gz zabbix-3.2.6
6, Install dependency to rebuild rpm
yum install iksemel-devel java-devel libssh2-devel OpenIPMI-devel unixODBC-devel gnutls-devel net-snmp-devel postgresql-devel -y
7, Rebuild the rpm
cd /root/rpmbuild/ rpmbuild -v -bb --clean SPECS/zabbix.spec
8, Install new zabbix-gateway
cd /root/rpmbuild//RPMS/ rpm -ivh zabbix-java-gateway-3.2.6-1.el7.centos.x86_64.rpm rm -rf /etc/zabbix/zabbix_java_gateway.conf mv /etc/zabbix/zabbix_java_gateway.conf.rpmsave /etc/zabbix/zabbix_java_gateway.conf
9, Now restart zabbix-gateway
service zabbix-java-gateway restart
Now you can see the zabbix can connect to wildfly and read metrics, I believe this could be very helpful those who switching to wildfly.
Newly installed zabbix-java-gateway package
rpm -qa | grep zabbix zabbix-java-gateway-3.2.6-1.el7.centos.x86_64 zabbix-server-mysql-3.2.6-1.el7.x86_64 zabbix-agent-3.2.6-1.el7.x86_64 zabbix-release-3.2-1.el7.noarch zabbix-web-3.2.6-1.el7.noarch zabbix-get-3.2.6-1.el7.x86_64 zabbix-web-mysql-3.2.6-1.el7.noarch zabbix-sender-3.2.4-1.el7.x86_64