Error: zabbix gateway unable to support wildfly http-remoting service

Posted on Updated on

Here is my workaround to support http-remoting in zabbix,

I have confiured zabbix server with zabbix agent to monitor production server, since we switch jboss-4.2 to wildfly-8.2 it got messed-up, zabbix unable to communicate with  wildfly. the following error found the Zabbix-gateway server. The error seems like wildlfy is communicating with native RMI service.

2017-07-20 01:18:10.247 [pool-1-thread-3] WARN com.zabbix.gateway.SocketProcessor - error processing request
com.zabbix.gateway.ZabbixException: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 192.168.101.197; nested exception is:
 java.net.ConnectException: Connection refused (Connection refused)]
 at com.zabbix.gateway.JMXItemChecker.getValues(JMXItemChecker.java:97) ~[zabbix-java-gateway-3.2.6.jar:na]
 at com.zabbix.gateway.SocketProcessor.run(SocketProcessor.java:62) ~[zabbix-java-gateway-3.2.6.jar:na]
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
 at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
Caused by: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 192.168.101.197; nested exception is:
 java.net.ConnectException: Connection refused (Connection refused)]
 at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369) ~[na:1.8.0_131]
 at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) ~[na:1.8.0_131]
 at com.zabbix.gateway.ZabbixJMXConnectorFactory$1.run(ZabbixJMXConnectorFactory.java:76) ~[zabbix-java-gateway-3.2.6.jar:na]
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_131]
 at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_131]
 ... 3 common frames omitted

I have download srcRPM https://repo.zabbix.com/zabbix/3.2/rhel/7/SRPMS/zabbix-3.2.6-1.el7.src.rpm , to check (JMXItemChecker.java:97) since it was mentioned in the error message. the same like my thought the protocol was hotcoded in the source. I have did some patch work and recompile the RPM and install Zabbix-gatway it worked as expected.

 

 

Please follow the steps which worked for me, you can download the version of zabbix server installed and reinstall the zabbix-gateway rpm.

1, Check the zabbix version installed. those installation was through yum

rpm -qa | grep zabbix
zabbix-java-gateway-3.2.6-1.el7.x86_64
zabbix-server-mysql-3.2.6-1.el7.x86_64
zabbix-agent-3.2.6-1.el7.x86_64
zabbix-release-3.2-1.el7.noarch
zabbix-web-3.2.6-1.el7.noarch
zabbix-get-3.2.6-1.el7.x86_64
zabbix-web-mysql-3.2.6-1.el7.noarch
zabbix-sender-3.2.4-1.el7.x86_64

2, Remove installed zabbix-gateway only

rpm -e zabbix-java-gateway-3.2.6-1.el7.x86_64
warning: /etc/zabbix/zabbix_java_gateway.conf saved as /etc/zabbix/zabbix_java_gateway.conf.rpmsave

3, Download and extract Source RPM

wget https://repo.zabbix.com/zabbix/3.2/rhel/7/SRPMS/zabbix-3.2.6-1.el7.src.rpm
rpm -ivh zabbix-3.2.6-1.el7.src.rpm

4, Modify the source to support http-remoting

cd /root/rpmbuild/SOURCES/
tar -zxvf zabbix-3.2.6.tar.gz
vi zabbix-3.2.6/src/zabbix_java/src/com/zabbix/gateway/JMXItemChecker.java

Modify with below entries, it checks the port 9990 with go to the “if” block others will go to the “else” block, so your wildlfy management port should be default port (9990) or you can modify the code as what you would like to use for wildlfy custom management port.

String conn = request.getString(JSON_TAG_CONN);
 int port = request.getInt(JSON_TAG_PORT);

 Integer remoting = new Integer("9990");
 int retval = remoting.compareTo(port);

if (retval == 0)
 {
 url = new JMXServiceURL("service:jmx:http-remoting-jmx://" + conn + ":" + port);
 }
 else
 {
 url = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://" + conn + ":" + port + "/jmxrmi");
 }

//url = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://[" + conn + "]:" + port + "/jmxrmi");
 jmxc = null;
 mbsc = null;

Save and exit the file

5, Pack the source file which extracted to modify the .java

tar -zcvf zabbix-3.2.6.tar.gz zabbix-3.2.6

6, Install dependency to rebuild rpm

yum install iksemel-devel java-devel libssh2-devel OpenIPMI-devel unixODBC-devel gnutls-devel net-snmp-devel postgresql-devel -y

7, Rebuild the rpm

cd /root/rpmbuild/
rpmbuild -v -bb --clean SPECS/zabbix.spec

8, Install new zabbix-gateway

cd /root/rpmbuild//RPMS/

rpm -ivh zabbix-java-gateway-3.2.6-1.el7.centos.x86_64.rpm
rm -rf /etc/zabbix/zabbix_java_gateway.conf
mv /etc/zabbix/zabbix_java_gateway.conf.rpmsave /etc/zabbix/zabbix_java_gateway.conf

9, Now restart zabbix-gateway

service zabbix-java-gateway restart

Now you can see the zabbix can connect to wildfly and read metrics, I believe this could be very helpful those who switching to wildfly.

Newly installed zabbix-java-gateway package

 

 


rpm -qa | grep zabbix
zabbix-java-gateway-3.2.6-1.el7.centos.x86_64
zabbix-server-mysql-3.2.6-1.el7.x86_64
zabbix-agent-3.2.6-1.el7.x86_64
zabbix-release-3.2-1.el7.noarch
zabbix-web-3.2.6-1.el7.noarch
zabbix-get-3.2.6-1.el7.x86_64
zabbix-web-mysql-3.2.6-1.el7.noarch
zabbix-sender-3.2.4-1.el7.x86_64

Howto: Disable 2FA in getlab for a user

Posted on

Gitlab supported 2FA and U2F device support to improve security in the public network. We need to configure GoogleAuthenticator in smartphone and register to the existing Gitlab user to allow logins, in some cases if we lost phone or  unable to access GoogleAuthenticator  we can’t to login GitLab,

GitLab share recovery codes to initiate recovery process, but if we also lost recovery codes, the following method help to disable 2FA and allow us to login,

Login to the SSH on Gitlab Server and connect to PostgreSQL.

[root@control3 master]# sudo -u gitlab-psql -i bash

Login to the postgres

bash-4.1$ /opt/gitlab/embedded/bin/psql --port 5432 -h /var/opt/gitlab/postgresql -d gitlabhq_production

Disable 2FA for user, here I used username root,

gitlabhq_production=# UPDATE public.users SET otp_required_for_login = false WHERE username = 'root';

Now you can login through browser, it ask to enable  2FA. hope this help you.

 

 

 

HoTo: Create HA Cluster on Centos6.7

Posted on

Worked on versions

OS : Centos 6.7
Building the cluster
To build this simple cluster, we need a few basic components:
Resource manager that can start and stop resources (like Pacemaker)
Messaging component which is responsible for communication and membership (like Corosync or Heartbeat)
Optionally: Cluster manager to easily manange the cluster settings on all nodes (like PCS)

Preparation
Start with configuring both cluster nodes with a static IP, a hostname, make sure that they are in the same subnet and can reach each other by nodename.

1, Local name binding using hosts

cat /etc/hosts
10.0.0.11 dir01 dir01.cluster.domain.com
10.0.0.12 dir02 dir02.cluster.domain.com
10.0.0.13 dir03 dir03.cluster.domain.com
10.0.0.10 ldap-ha ldap-ha.cluster.domain.com

2, Disable SELINUX

vi /etc/selinux/config
SELINUX=disabled

3, Clean update the server

yum install clean all
yum update

Basic Firewall setting for All the nodes in the cluster:
When testing the cluster, we could temporarily disable the firewall to be sure that blocked ports aren’t causing unexpected problems.

1, Open UDP-ports 5404 and 5405 for Corosync:

iptables -I INPUT -m state --state NEW -p udp -m multiport --dports 5404,5405 -j ACCEPT

2, Open TCP-port 2224 for PCS

iptables -I INPUT -p tcp -m state --state NEW -m tcp --dport 2224 -j ACCEPT

3, Allow IGMP-traffic

iptables -I INPUT -p igmp -j ACCEPT

4, Allow multicast-traffic

iptables -I INPUT -m addrtype --dst-type MULTICAST -j ACCEPT

5, Save the changes made in iptables and restart

service iptables save
service iptables start

Installation
1, After setting up the basics, we need to install the packages for the components on all the server,

yum install corosync pcs pacemaker cman

2, To manage the cluster nodes, we will use PCS. This allows us to have a single interface to manage all cluster nodes. By installing the necessary packages, Yum also created a user, hacluster, which can be used together with PCS to do the configuration of the cluster nodes. Before we can use PCS, we need to configure public key authentication or give the user a password on all the nodes:

echo "hapasswd" | passwd hacluster --stdin

3, Startng pcsd pacemaker manager in all the nodes

service pacemaker start
service pcsd start
chkconfig pacemaker on
chkocnfig pcsd on

4, Create new corosync multicast configuration with the below given,

vi /etc/corosync/corosync.conf

compatibility: whitetank
totem {
 version: 2
 # Time (in ms) to wait for a token (1)
 token: 10000
 # How many token retransmits before forming a new
 # configuration
 token_retransmits_before_loss_const: 10
 # How long to wait for join messages in the membership protocol (ms)
 join: 1000
 # How long to wait for consensus to be achieved before starting a new
 # round of membership configuration (ms)
 consensus: 7500
 # Number of messages that may be sent by one processor on receipt of the token
 max_messages: 20
 # Stagger sending the node join messages by 1..send_join ms
 send_join: 45
 # Limit generated nodeids to 31-bits (positive signed integers)
 clear_node_high_bit: yes
 # Turn off the virtual synchrony filter
 vsftype: none
 # Enable encryption (2)
 secauth: on
 # How many threads to use for encryption/decryption
 threads: 0
 # This specifies the redundant ring protocol, which may be
 # none, active, or passive. (3)
 rrp_mode: active

# The following is a two-ring multicast configuration. (4)
 interface {
 ringnumber: 0
 bindnetaddr: 10.0.0.11
 mcastaddr: 239.255.1.1
 mcastport: 5405
 }
}

amf {
 mode: disabled
}

service {
 # Load the Pacemaker Cluster Resource Manager (5)
 ver: 1
 name: pacemaker
}

aisexec {
 user: root
 group: root
}

logging {
 fileline: off
 to_stderr: yes
 to_logfile: yes
 logfile: /var/log/cluster/corosync.log
 to_syslog: yes
 syslog_facility: daemon
 debug: off
 timestamp: on
 logger_subsys {
 subsys: AMF
 debug: off
 tags: enter|leave|trace1|trace2|trace3|trace4|trace6
 }}

5, Since we will configure all nodes from one point, we need to authenticate on all nodes before we are allowed to change the configuration. Use the previously configured hacluster user and password.

pcs cluster auth dir01 dir02 -u hacluster

From here, we can control the cluster by using PCS from dir01. It’s no longer required to repeat all commands on all the nodes.
Authorisation tokens are stored in the file /var/lib/pcsd/tokens.

Create the cluster and add nodes
1, start adding all nodes to a cluster named LDAP-HA-Cluster

pcs cluster setup --name LDAP-HA-Cluster dir01 dir02

2, After creating the cluster and adding nodes, start cluster packeages from the single poing , it will start pacemaker and corosync in all the nodes.

pcs cluster start --all

3, Optionally, depending on requirements, we can enable cluster services to start on boot,

pcs cluster enable --all

To check the status of the cluster after starting it:

pcs status
service pacemaker status
service corosync status

To check the status of the nodes in the cluster

pcs status nodes
corosync-objctl runtime.totem.pg.mrp.srp.members
corosync-cfgtool -s
pcs status corosync

Cluster configuration
1, Check the configuration for errors, and there still are some

crm_verify -L -V

The above command tells that erros still in the cluster, First time we can see error regarding STONITH (Shoot The Other Node In The Head), which is a mechanism to ensure that you don’t end up with two nodes that both think they are active and claim to be the service and virtual IP owner, also called a split brain situation. Since we have simple cluster, we’ll just disable the stonith option

pcs property set stonith-enabled=false

2, Ignore a low quorum

pcs property set no-quorum-policy=ignore

The below settings needed only if we have 3 servers in the cluster

pcs property set expected-quorum-votes=”3”

3, Set the basic cluster properties

pcs property set pe-warn-series-max=1000 \
 pe-input-series-max=1000 \
 pe-error-series-max=1000 \
 cluster-recheck-interval=5min

4, I believe we already configured HA-Proxy in the server, if not let start with basic install and start haproxy. Because we need to configure haproxy lsb in the cluster.

Yum install haproxy
service haproxy start

5, Adding Floating IP with hearbeat to monitor servers, This IP is used to connect HA-proxy and won’t assign to any serve where haproxy failed to start

pcs resource create LDAPfrontendIP0 ocf:heartbeat:IPaddr2 ip=10.0.0.10 cidr_netmask=32 op monitor interval=30s

To check the status;

pcs status resources

Now we can get the responce from the floating IP,

ping -c1 10.0.0.10

To see who is the current owner of the resource/virtual IP:

pcs status|grep virtual_ip

Adding HA-Proxy to Pacemaker configuration
1, Because there is no OCF agent for HA-Proxy we define a LSB resource haproxy (Note: This must be the same name as the startup script in /etc/init.d and comply to the LSB standard. The expected behavior of the startup scripts can be found at Linux-HA documentation. Fortunately the haproxy script can be used, so a recource LDAP-HA-Proxy will be created:

pcs resource create LDAP-HA-Proxy lsb:haproxy op monitor interval=5s

The resource will start on the node with the LDAPfrontendIP0 resource but complain about the other hosts in the HA-Cluster:

pcs status

2, Obviously the haproxy service fails to start if the IP adress of the loadbalancer does not exist. The default behavior of Pacemaker spreads the resources across all cluster nodes. Because the LDAPfrontendIP0 and LDAP-HA-Proxy resources are related to each other LDAP-HA-Proxy can only run on the node with the LDAPfrontendIP0 resource. To archive this a “colocation constraint” is needed. The weight score of INFINITY makes it mandatory to start the LDAP-HA-Proxy resource on the node with the LDAPfrontendIP0 resource:

pcs constraint colocation add LDAP-HA-Proxy LDAPfrontendIP0 INFINITY

3, The order of the resource is important otherwise the LDAPfrontendIP0 resource will be started on the node with the LDAP-HA-Proxy resource (which can not start because the LDAPfrontendIP0 resource provides the interface configuration for LDAP-HA-Proxy). Futhermore the LDAPfrontendIP0 resource should always start before LDAP-HA-Proxy resource so we have to enforce the resource start/stop ordering:

pcs constraint order LDAPfrontendIP0 then LDAP-HA-Proxy

After configuring the cluster with the correct constraints, restart it and check the status:

pcs cluster stop --all && pcs cluster start –all 
pcs status

Hence we completed cluster setup with HA-proxy, the following setup required to know how we can switch/Adding/removing resources

Howto: Backup & Restore Database in PostgreSQL (pg_dump,pg_restore)

Posted on

H ow to backup and restore dabatase in PostgreSQL

1)Backup data with pg_dump

pg_dump -i -h localhost -p 5432 -U postgres -F c -b -v -f  "/home/anand/ltchiedb.backup" ltchiedb

To list all of the available options of pg_dump , please issue following command.

pg_dump -?
-p, –port=PORT database server port number
-i, –ignore-version proceed even when server version mismatches
-h, –host=HOSTNAME database server host or socket directory
-U, –username=NAME connect as specified database user
-W, –password force password prompt (should happen automatically)
-d, –dbname=NAME connect to database name
-v, –verbose verbose mode
-F, –format=c|t|p output file format (custom, tar, plain text)
-c, –clean clean (drop) schema prior to create
-b, –blobs include large objects in dump
-v, –verbose verbose mode
-f, –file=FILENAME output file name

2) Restore data with pg_restore

pg_restore -i -h localhost -p 5432 -U postgres -d ltchiedb -v "/home/anand/ltchiedb.backup"

To list all of the available options of pg_restore , please issue following command.

pg_restore -?
-p, –port=PORT database server port number
-i, –ignore-version proceed even when server version mismatches
-h, –host=HOSTNAME database server host or socket directory
-U, –username=NAME connect as specified database user
-W, –password force password prompt (should happen automatically)
-d, –dbname=NAME connect to database name
-v, –verbose verbose mode

Error: “ldap_bind: Can’t contact LDAP server (-1)” on nagios check

Posted on Updated on

Nagios check_ldaps plugin working with SSL or TLS
Error:

[root@nagios libexec]# ./check_ldaps  -H 10.0.0.51  -w 10 -c 15 -b dc=tolven,dc=com -p 636 -v
ldap_bind: Can't contact LDAP server (-1)
 additional info: TLS error -8172:Peer's certificate issuer has been marked as not trusted by the user.
 Could not bind to the LDAP server

 

To fix this issue, simple understand the client is not issuing certificate, The client environment is not fully configured. so I configure the bellow setting. It works charm in both ways, byt ignoring the SSL check or adding client certificate,

Create new configuration file if not exist, /etc/openldap/ldap.conf

Ignoring SSL certificate, Add the bellow settings

TLS_REQCERT never
TLS_CACERT /etc/openldap/certs/ldap-client-ca.crt

Output:

root@nagios libexec]# ./check_ldaps -H 10.0.0.51 -w 10 -c 15 -b dc=tolven,dc=com -p 636 -v
LDAP OK - 0.062 seconds response time|time=0.061526s;10.000000;15.000000;0.000000

ERROR: Got error on installing nagios-plugin on centos7 “In file included from localcharset.c:26:0:”

Posted on

Build failure on installing nagios plugin on centos7,

Error:

In file included from localcharset.c:26:0:
./stdio.h:456:1: error: ‘gets’ undeclared here (not in a function)
_GL_WARN_ON_USE (gets, "gets is a security hole - use fgets instead");
^
make[4]: *** [localcharset.o] Error 1
make[4]: Leaving directory `/opt/installer/nagios-plugins-1.4.16/gl'
make[3]: *** [all-recursive] Error 1
make[3]: Leaving directory `/opt/installer/nagios-plugins-1.4.16/gl'
make[2]: *** [all] Error 2
make[2]: Leaving directory `/opt/installer/nagios-plugins-1.4.16/gl'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/opt/installer/nagios-plugins-1.4.16'
make: *** [all] Error 2

It seems that nagios has created a custom version of the standard stdio.h file so even if the bug has been fixed in the c libraries it may still exists in nagios.

maybe you can do a quick fix by running this command in the same folder as stdio.in.h:

sed -i -e '/gets is a security/d' ./gl/stdio.in.h

Thats just a long shot though, I haven’t tried it myself.
PS. Make a backup copy of stdio.in.h first so you can revert the source back to its original state if it doesn’t work.

HowTo: Change facebook theme

Posted on Updated on

Facebook is not officially support for changing its appearance, but some people craze about changing their FB theme and it wants to be look so different,

Here is my workout for the theme changer, Firefox release addons for adding custom CSS for a particular websites, we can do the same for FaceBook.

1, Install Styles addon for firefox
https://addons.mozilla.org/en-us/firefox/addon/stylish/

Once installed the addon reboot the firefox is reqired to edit the settings, then you will see the S button on the Tool bar.

stylish

 

2, Add custom CSS for Facebook,
https://userstyles.org/styles/browse/facebook

      Go to the above url and select once of the style, Here I used FaceBlack (Black Facebook by a3cAnton)
click on installstylish

3, Turn the Styles for all websites,

Now goto “Styles” button on toolbar and select “Turn all styles on”

Now facebook is ready to start with new CSS, Checkout other similar CSS for other Social network websites. https://userstyles.org/

Here is My FB looks like.

Screenshot from 2015-11-04 15:21:59