sandip's blog

Check bots success POSTs in apache access log

Here is a one liner to check for IPs of bots that are misusing your site.

$ awk '$6 ~ /POST/ && $9 ~ /200/ {print $1 " " $7}' /var/log/httpd/access_log | sort | uniq -c | sort -n | tail

This will give you the top 10 IPs and URIs with a hit count.

Consider blocking those rogue IPs with a high hit count via iptables.

Unicode keyword translation

Do you see garbled text, unicode keywords in your referral web stats reports like below?

เรียน ที่ไหน

Use the below url for the conversion:

Unicode keyword translation

Here is the simple php source code that uses google translate for translation to English:

<?php header("Content-Type: text/html; charset=iso-8859-1"); ?>

  <title>Unicode keyword translation</title>
  <br />
  Copy/paste unicode keyword:
  <br />
  <br />
  <form action="convert_keywords.php">
    <input size="100" name="input" value="<?= $_GET['input']; ?>" type="text">
    <input value="Go" type="submit">
  <br />
  <br />

=explode("=", $_SERVER['QUERY_STRING']);
$translate_url="|en|" . $query;
$query)) {

  <iframe src="<?= $translate_url; ?>" width="100%" height="100%">
  Iframe unsupported...

<?php } ?>


Check and tune network speed

I've had to do this on a couple of servers to check on the network speeds and sysctl tuning.

On the receiving end ( bring up netcat listening to a port:

while true; do nc -l 8001 >/dev/null ; done

Use the below one liner to send over some data via netcat from which prints out the transfer speed.

( dd if=/dev/zero bs=64K count=1000 | nc 8001 ) 2>&1 | awk '/MB/{print $8*8 " " tolower($9)}'

Increase the default maximum TCP buffer size and rerun the above test for proper tuning. Make sure to restart netstat listening socket upon sysctl changes.

The following are recommended:

# max TCP buffer size: 16MB (16 * 1024 * 1024).
# Could increase to 32MB for GigE.
# Increasing the TCP send buffers will increase the performance
# if you have large files to send.
net.core.wmem_max = 16777216

# If you have a lot of large file uploads,
# increasing the receive buffers will help.
net.core.rmem_max = 16777216

# increase Linux autotuning TCP buffer limits:
# min, default, and max number of bytes to use
# (only change the 3rd value, and make it 16 MB or more)
net.ipv4.tcp_rmem = 4096        87380   16777216
net.ipv4.tcp_wmem = 4096        65535   16777216

# Support for the above large TCP send and receive windows.
# Needs to be set to 1 if the Max TCP Window is over 65535 (64K).
net.ipv4.tcp_window_scaling = 1

# Increase backlog to avoid dropped packets and increase throughput.
# Check with `netstat -st | grep packets` and check for
# dropped and packet errors.
net.core.netdev_max_backlog = 5000

ssh keygen RSA versus DSA

While generating ssh keys, I usually use RSA type since it can be used to generate 2048 bits key, while DSA is restricted to exactly 1024 bits.

ssh-keygen -t rsa -b 2048

smartctl with 3ware RAID controller

The below information is extracted from the smartctl man page.

To look at ATA disks behind 3ware SCSI RAID controllers:

smartctl -a -d 3ware,N /dev/sda
smartctl -a -d 3ware,N /dev/twe0
smartctl -a -d 3ware,N /dev/twa0

Where N is the disk number.

Devices /dev/sda and /dev/twe0 is used with 3ware series 6000, 7000, and 8000 controllers that use the 3x-xxxx driver.

/dev/sda form is deprecated starting with the Linux 2.6 kernel.

/dev/twa0 is used with 3ware 9000 series controllers, which use the 3w-9xxx driver.

You can also check on the series with the below command:

lspci |grep 3ware

Find files used for htauth

Below will list all of the files that are used for apache authentication in /var/www/html file path:

find /var/www/html -name .htaccess | xargs awk '{sub(/^[ \t]+/,"")};/File/{print $2}' | sort | uniq

Here is the breakdown:

find /var/www/html -name .htaccess

Find all files named ".htaccess" at path "/var/www/html"

xargs awk '{sub(/^[ \t]+/,"")};/File/{print $2}'

The search output gets piped via xargs to awk, deleting leading whitespace (spaces and tabs) from front of each line and output is of only the second field of lines containing the text "File".

sort | uniq

Awk output is further piped through sort and uniq which results in the files being used for apache authentication.

Converting Shared to Exclusive IP for Plesk Domain

First go into "Clients", select the corresponding client IP Addresses and assign the additional IP address.

Then go into "Domains", select the domain that should be converted to exclusive IP and modify the "Web Host Settings" and assign it the new IP.

VIM with Ruby support on CentOS-5.5

While trying to get command-t VIM plugin installed I noticed that the default vim-7.0.x installed from CentOS Base repository does not come compiled with ruby support and needed to be upgraded to version 7.2.x:

$ vim --version|grep ruby
+python +quickfix +reltime +rightleft -ruby +scrollbind +signs +smartindent

Notice "-ruby" which states that it is without ruby support.

Once the rpmbuild environment is setup, install the source rpm:

# su - rpmbuild
$ cd ~/rpm/SRPMS/
$ wget
$ rpm -Uvh --nomd5 ~/rpm/SRPMS/vim-7.2.411-1.6.el6.src.rpm

Edit the vim.spec file, line 496 and remove perl-devel from the requirement list.

BuildRequires: python-devel ncurses-devel gettext perl-devel

Then build rpm binary:

$ rpmbuild -bb ~/rpm/SPEC/vim.spec

Install the rpms via:

# rpm -Uvh ~/rpm/RPMS/x86_64/vim-{m,c,e}*

jps and jstat for tomcat on jdk-1.6.0_24

Recently upgrading to java version "1.6.0_24", jps and jstat seemed to be broken to get monitoring information from running tomcat process.

By default is /tmp, however tomcat usually uses it's own temp directory. That is where jps/jstat looks for hsperfdata_* dirs. If is not set, it would look in /tmp. If jps can't find the hsperfdata directory, it won't report anything.

jps/jstat however allows to specify in case you're using a JVM that places those directories in a different location.

So to get it to work:

$JAVA_HOME/bin/jps -l
$JAVA_HOME/bin/jstat -gc $PID

Django HTTPS Redirects

This works for both HTTP and HTTPS where any front end web server such as nginx which handles the actual request sets a header when request comes via HTTPS. In Apache configuration you then use mod_setenvif to set the HTTPS variable, which Django then picks up to use for redirection.

With front end nginx server which handles SSL, set header "X-Forwarded-Proto=https" via:

  proxy_set_header X-Forwarded-Proto https;

On Apache, add directive:

  SetEnvIf X-Forwarded-Proto https HTTPS=1

The HTTPS variable is picked up as being special by mod_wsgi and it will fix the wsgi.url_scheme in WSGI environment which Django then uses for redirection.

This way you don't need to customize Django stack.

Syndicate content