Scripting

HowTo: Tomcat Logging – log customized with {X-Forwarded-For}

Posted on Updated on

Tomcat is allowing us to track back logs with enamours of information by customizing the log pattern. There is preset patten is available, we can also implement is in single line

I enabled few more information like execution time , request size , cookies etc..

Default tag should be like this

<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"  
               prefix="localhost_access_log." suffix=".txt" pattern="common" resolveHosts="false"/>

Common : %{X-Forwarded-For}i %l %u %t “%r” %s %b
Combined : %{X-Forwarded-For}i %l %u %t %r %s %b %{User-Agent}i %{Referer}i %{Cookie}i

You can change either Common or Combined

I have implemented my own pattern like below, so it should more detailed

pattern="%h %{X-Forwarded-For}i %l %u %t  &quot;%r&quot; %s %b  &quot;%{User-Agent}i&quot; &quot;%{Referer}i&quot; &quot;%{Cookie}i&quot; %T"

Access Log pattern new look

-----------------------------
192.168.1.185 - - - [18/Mar/2014:10:52:06 +0530]  "GET /ajax/norm/list/status?ids=23%2C11%2C9%2C7%2C6%2C5%2C2%2C1%2C HTTP/1.1" 200 42  "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0" "http://192.168.1.188/norm/list" "JSESSIONID=4FD1DBEB911CD2E19AA4798F9A26DCA8" 0.007
-----------------------------
Log Details : 192.168.1.185 : Remote host name (or IP address if resolveHosts is false)
– : X-Forwarded-For – : Remote logical username
– : Remote user that was authenticated
[18/Mar/2014:10:52:06 +0530]  : Date and time, in Common Log Format
GET /ajax/norm/list/…… : First line of the request (method and request URI)
HTTP/1.1 : Request protocol
200 : HTTP status code of the response
42 : Bytes sent, excluding HTTP headers (Content size)
Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Firefox/24.0: User Agent
http://192.168.1.188/norm/list : Referer
JSESSIONID=4FD1DBEB911CD2E19AA4798F9A26DCA8 : Cookie header
0.007 : Time taken to process the request, in seconds

Once every thing has been done you can restart the tomcat to make it effect, more options are given below

%a – Remote IP address
%A – Local IP address
%b – Bytes sent, excluding HTTP headers, or ‘-‘ if zero
%B – Bytes sent, excluding HTTP headers
%h – Remote host name (or IP address if resolveHosts is false)
%H – Request protocol
%l – Remote logical username from identd (always returns ‘-‘)
%m – Request method (GET, POST, etc.)
%p – Local port on which this request was received
%q – Query string (prepended with a ‘?’ if it exists)
%r – First line of the request (method and request URI)
%s – HTTP status code of the response
%S – User session ID
%t – Date and time, in Common Log Format
%u – Remote user that was authenticated (if any), else ‘-‘
%U – Requested URL path
%v – Local server name
%D – Time taken to process the request, in millis
%T – Time taken to process the request, in seconds
%I – current request thread name (can compare later with stacktraces)
%f – X-Forwarded-For IP address
%F – X-Forwarded-For address

Bash: History appending for multiple sessions

Posted on

I got a requirement for listing all the command history if  multiple terminal sessions using in different region for the single user. I followed the below steps.

step1 : Create a new file with the below entries

root@appserver:# cat /etc/profile.d/bash_history.sh
function share_history {
 history -a
 history -c
 history -r
}
HISTSIZE=99999
HISTCONTROL=ignoredups
HISTTIMEFORMAT=`echo -e "33[1;34m%d/%h/%Y 33[1;31m%H:%M:%S 33[0m"`
PROMPT_COMMAND='share_history'
shopt -u histappend

Step2: activate it in run-time

root@appserver:# source /etc/profile.d/bash_history.sh

Now you can see the list of aged histories

Sample Output :

1005 26/Dec/2013 14:23:08 vi /etc/profile.d/bash_history.sh
1006 26/Dec/2013 14:23:27 source /etc/profile.d/bash_history.sh
1007 26/Dec/2013 14:23:31 history

Info: Configure Redmine on cpanel hosting account with sending and receiving emails.

Posted on Updated on

Wiki : http://en.wikipedia.org/wiki/Redmine

Redmine is a free and open source, web-based project management and bug-tracking tool. It includes a calendar and Gantt charts to aid visual representation of projects and their deadlines. It handles multiple projects. Redmine provides integrated project management features, issue tracking, and support for various version control systems.
The design of Redmine is significantly influenced by Trac, a software package with some similar features.
Redmine is written using the Ruby on Rails framework. It is cross-platform and cross-database. It is part of the Bitnami app library that provides an installer and virtual machine for ease of deployment.

Before starting installation you have to make sure that Ruby on rails is working fine in your environment, If not you can follow the installation document for more help.

Installaing Ruby on Rails with Cpanel : https://enlook.wordpress.com/2013/11/19/howto-install-ruby-on-rails-with-cpanel/

Once you have done, then start the redmine installation steps.

Login to the terminal using primary account logins.

#ssh myaccount@mydomain.com

  1. Create rails_app folder and redmine folder within it then go inside that folder
    # mkdir -p ~/rails_apps/redmine/
    # cd ~/rails_apps/redmine/
  2.  Download redmine redmine-2.3.3 or latest stable version, extract it and move the content out of it, then delete the files not being used.
    1. # wget http://files.rubyforge.vm.bytemark.co.uk/redmine/redmine-2.3.3.tar.gz
      # tar -zxvf redmine-2.3.3.tar.gz
      # mv redmine-2.3.3/* ./
      # rm -rf redmine-2.3.3/
  3. Move example files where they can be used
    # cd config
    # mv database.yml.example database.yml
    # mv configuration.yml.example configuration.yml
  4. Creating the MySQL Database/User/Password
    Login to Cpanel account, Create a database , user and grant full privilege to the new user for the particular database.
    cPanelXdatabase
  5. Modifying your database.yml file.
    # vi database.yml
    production:
    adapter: mysql
    database: redmine
    host: localhost
    username: myaccount_databaseuser
    password: newpassowd
    encoding: utf8
  6. Updating the ~/rails_apps/redmine/public/.htaccess file
    # cd ../public/
    # pwd
    1. You should see something similar to this.

    /home/myaccountuser/rails_apps/redmine/public

        Add these lines
    Options -MultiViews
    PassengerResolveSymlinksInDocumentRoot on
    #Set this to whatever environment you'll be running in
    RailsEnv production
    RackBaseURI /
    SetEnv GEM_HOME /home/myaccountuser/rails_apps/redmine/public
    
    # set to resolve avoid rails control to the folder for image resolution   
    RewriteEngine On
    RewriteCond %{REQUEST_URI} ^/images.*
    RewriteRule .* - [L]
  7. Create a subdomain eg: projects.mydomain.com
    Follow cpanel procedure to create subdomain. Subdomains
  8. Remove projects folder inside public_html and create symbolic link.
    # rm -rf ~/public_html/projects
        Creating the symlink
    # ln -s ~/rails_app/redmine/public ~/public_html/projects
  9. Updating Environment variables in ~/.bashrc file
        Add these lines to the bottom of your ~/.bashrc file
               export HPATH=$HOME
               export GEM_HOME=$HPATH/ruby/gems
               export GEM_PATH=$GEM_HOME:/lib64/ruby/gems/1.9.3
               export GEM_CACHE=$GEM_HOME/cache
               export PATH=$PATH:$HPATH/ruby/gems/bin
               export PATH=$PATH:$HPATH/ruby/gems
        after which source your .bashrc file
            # source ~/.bashrc
        You will then need to check your rails version
            rails -v && rake --version && gem -v
          You should get this message

    ruby

    Rails 4.0.1
    rake, version 0.9.2.2
    1.8.23
  10. Running bundle install
    # cd ~/rails_apps/redmine/
    # bundle install
    # rake generate_session_store
  11. Running generate_session_store or generate_secret_token
    1. # rake generate_session_store
        If you get an error saying that command is deprecated, run this command instead;
     # rake generate_secret_token
  12. Start the site session
    # rake db:migrate RAILS_ENV=production
  13. Configuring outgoing emailsUpdate the setting in configuration.yml
    default:
     email_delivery:
     delivery_method: :smtp
     smtp_settings:
     address: localhost
     port: 25
     domain: mydomain.com
     authentication: :none
    enable_starttls_auto: false

    Now the redmine have capable to send emails using exim install in the cpanel server.

  14. Configuring Incomming emails for IMAPCreate a cron job for the script to get continuous email feaching
    cPanelX

    For the first this script must execute from the terminal, so it will display error if any.

    /usr/bin/rake -f /home1/innovat4/rails_apps/redmine/Rakefile --silent redmine:email:receive_imap RAILS_ENV="production" port=143 host=mydomain.com username=projects@mydomain.com password=myemailpassword

    For more help follow the official link http://www.redmine.org/projects/redmine/wiki/RedmineReceivingEmails#Enabling-unknown-users-to-create-issues-by-email

Note : Each configuration required rails environment reboot for that you can follow the simple way.

# touch ~/rails_app/redmine/tmp/reboot.txt

script: Bash script to backup MySQL databases.

Posted on Updated on

 

#!/bin/bash 
# Simple script to backup MySQL databases 
# 
# You have to enter the credintials, the scritp will make backup of all the databases 
# including information schema and perfomance schema as well, and store it as a gunzip format 
# in the backup directory. Each databases are dump as seperate files. 
# 
# This will maintain 30 days backup. If you need to extend, edit the WEIGHT as your own. 
# Website : https://enlook.wordpress.com , http://planetcure.info , http://xtermpro.com 
# Created by : Anandbabu 
# 
#################################################################################################
# Parent backup directory
backup_parent_dir="/backup/"
#Enter multiple email ID using space
Email="email@domain.com email@domain.com"
Email_Content="/tmp/Mail_db"
WEIGHT=30 # 30 days
# MySQL settings
mysql_user="my_database_user"
mysql_password='database_password'
mysql_databases="Default_database"
#Creating file for email
[ ! -f ${Email_Content} ] && touch ${Email_Content} || :> ${Email_Content}
E_mail(){
 for email in ${Email}
 do
 cat ${Email_Content} | mail -s "Notification: Mysql Database Backup $@ from MyServer " ${email} -aFrom:Backup\<backup@domain.com\>
 done
 }
# Read MySQL password from stdin if empty
if [ -z "${mysql_password}" ]; then
 echo -n "Enter MySQL ${mysql_user} password: " >> ${Email_Content}
 read -s mysql_password
 echo
fi
# Check MySQL password
echo exit | mysql --user=${mysql_user} --password=${mysql_password} -B 2>/dev/null
if [ "$?" -gt 0 ]; then
 echo "MySQL ${mysql_user} password incorrect" >> ${Email_Content}
 E_mail Failed
 exit 1
else
 echo "MySQL ${mysql_user} password correct." >> ${Email_Content}
fi
# Create backup directory and set permissions
backup_date=`date +%Y_%m_%d_%H_%M`
backup_dir="${backup_parent_dir}/${backup_date}"
echo "Backup directory: ${backup_dir}" >> ${Email_Content}
mkdir -p "${backup_dir}"
chmod 700 "${backup_dir}"
# Get MySQL databases
mysql_databases=`echo 'show databases' | mysql --user=${mysql_user} --password=${mysql_password} -B | sed /^Database$/d`
# Backup and compress each database
for database in $mysql_databases
do
if [[ "$database" =~ "information_schema" || "$database" =~ "performance_schema" ]] ; then
 additional_mysqldump_params="--skip-lock-tables"
else
 additional_mysqldump_params=""
fi
 echo "Creating backup of \"${database}\" database" >> ${Email_Content}
 mysqldump ${additional_mysqldump_params} --user=${mysql_user} --password=${mysql_password} ${database} | gzip > "${backup_dir}/${database}.sql.gz"
 chmod 600 "${backup_dir}/${database}.sql.gz"
done

##Removing folder older than 30 days
ECOUT=""
echo "" >> ${Email_Content}
ECOUT=`find ${backup_parent_dir} -type d -ctime +$WEIGHT`
if [ -z $ECOUT ]; then
 echo "No more older backups to remove" >> ${Email_Content}
 E_mail Success
 exit
else
 echo "Following older backups are removed" >> ${Email_Content}
 for i in $ECOUT
 do
 rm -rvf $i 1>>${Email_Content} 2>>${Email_Content}
 done
 E_mail Success
 exit
fi

Error: 500 OOPS: vsftpd: refusing to run with writable root inside chroot()

Posted on

Each time while am installing VSFTPD on ubuntu and enable chroot for the users it will refuse to login to the home directory because of write permission in its parent dir, to fix this I used the command

chmod a-w  /path/to/the/ftp/home

but is was most annoying and frustrating problem. I supposed to update the vsftpd package with security fix. the steps are below.

wget http://ftp.us.debian.org/debian/pool/main/v/vsftpd/vsftpd_3.0.2-3_amd64.deb
dpkg -i vsftpd_3.0.2-3_amd64.deb
echo "allow_writeable_chroot=YES" >> /etc/vsftpd.conf
echo "seccomp_sandbox=NO" >> /etc/vsftpd.conf
service vsftpd reload

now the FTP service will work calm in my server.

Howto : Install yum after installing OS on centos

Posted on

Yum  is a awesome tool written in python used to install packages in redhat flavours. It have ability to choose dependency packages as it own from the repository needed by the installing packages. If you missed to find yum in your server, follow the below steps to install

you can get the packages from here ftp://rpmfind.net/linux/centos/6.4/os/x86_64/Packages/ 

Use wget to download these pacakages.

Package list:

gpgme-1.1.8-3.el6.x86_64.rpm
pygpgme-0.1-18.20090824bzr68.el6.x86_64.rpm
python-iniparse-0.3.1-2.1.el6.noarch.rpm
python-pycurl-7.19.0-8.el6.x86_64.rpm
python-urlgrabber-3.9.1-8.el6.noarch.rpm
rpm-4.8.0-32.el6.x86_64.rpm
rpm-python-4.8.0-32.el6.x86_64.rpm
yum-3.2.29-40.el6.centos.noarch.rpm
yum-metadata-parser-1.1.2-16.el6.x86_64.rpm
yum-plugin-fastestmirror-1.1.30-14.el6.noarch.rpm

upgragde rpm to latest version to avoid package confilct error follow the below switchs

 [root@anand ~]# cd /root/installer/
[root@anand installer]# rpm -ivh --replacefiles rpm-4.8.0-32.el6.x86_64.rpm
 warning: rpm-4.8.0-32.el6.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEY
 Preparing... ########################################### [100%]
        1:rpm ########################################### [100%]
[root@anand installer]# mv rpm-4.8.0-32.el6.x86_64.rpm ../.
[root@anand installer]# rpm -ivh *
 warning: gpgme-1.1.8-3.el6.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID c105b9de: NOKEY
 Preparing...                         ########################################### [100%]
 1:yum-metadata-parser                ########################################### [ 11%]
 2:rpm-python                         ########################################### [ 22%]
 3:python-pycurl                      ########################################### [ 33%]
 4:python-urlgrabber                  ########################################### [ 44%]
 5:python-iniparse                    ########################################### [ 56%]
 6:gpgme                              ########################################### [ 67%]
 7:pygpgme                            ########################################### [ 78%]
 8:yum-plugin-fastestmirro            ########################################### [ 89%]
 9:yum                                ########################################### [100%]

Now you can create a package repository as your own. floks 🙂

Error: Native SSL support failed to pass the startup test.

Posted on

If you are getting  above error then  it mean CPanel has been  updated to  the lasted version  and it also updates  the “Net::SSLeay”  version to  1.36 and that is why  you get  the error :

Native SSL support failed to pass the startup test.  stunnel was started instead.
The exact error was: [linktest=BinCheck ok
] [binary=/usr/local/cpanel/cpsrvd-ssl] [cpsrvd=Net::SSLeay object version 1.36 does not match $Net::SSLeay::VERSION 1.32.
]

In order to fix this follow the below steps :
1) Download the Net_SSLeay 1.30 or 1.32

root@server[#] cd /usr/local/src
root@server[#] wget http://mirror.yourconnect.com/CPAN/authors/id/F/FL/FLORA/Net_SSLeay.pm-1.30.tar.gz


or


root@server[#] wget http://mirror.yourconnect.com/CPAN/authors/id/F/FL/FLORA/Net-SSLeay-1.32.tar.gz

2) Then extract the tar.gz file

root@server[#] tar -zxf Net_SSLeay.pm-1.30.tar.gz

or

root@server[#] tar -zxf Net-SSLeay-1.32.tar.gz

Then go to dir Net_SSLeay.pm-1.30 or Net-SSLeay-1.32

root@server[#] cd Net_SSLeay.pm-1.30/

or

root@server[#] cd Net-SSLeay-1.32/

and then execute Make file and then perform make install

root@server[/usr/local/src]# ./Makefile.PL -t
root@server[/usr/local/src]# make install

Now, finally you can check the installed module from WHM >> Main >> Software >> Install a Perl Module

That’s all you are done.