Saturday, January 17, 2015

Zabbix. JMX monitoring. Working around the "javax.naming.namenotfoundexception: jmxrmi" issue

Aim: to monitor Active MQ queues bundled within Jboss Fuse with Zabbix

But there's a problem:
https://support.zabbix.com/browse/ZBXNEXT-1274

So what do we do?
Skip the java gateway altogether.

http://snippets.syabru.ch/nagios-jmx-plugin/commands.html



java -jar check_jmx.jar --h
Usage: check_jmx -U  -O  -A 
    [-K ] [-w ] [-c ]
    [-o ] [--username ] [--password ]
    [-u ] [-v] [-h]


We incorporate this jmx client command line utility into the zabbix toolset by creating a user parameter that looks like this:


UserParameter=jmx_query[*],/usr/bin/java -jar /usr/local/bin/nagios_jmx/check_jmx.jar -U  "$1" -O "$2" -A $3 --username $4 --password $5 | grep -oP "\d*" | head -1


Restart the zabbix agent and perform some tests with zabbix_get


For instance:

zabbix_get  -s 10.88.XXX.XXX -I 10.88.XXX.XXX -k 'jmx_query["service:jmx:rmi://IP:44444/jndi/rmi://IP:1099/karaf-FOO","org.apache.activemq:type=Broker,brokerName=FOO,destinationType=Queue,destinationName=FOO.BAR.QUEUE",DequeueCount,myuser,mypassword]'


Tip: if your object definition includes commas, you'll need to set the UnsafeUserParameters = 1 and double quote the parameters that include them.

A very hackish way of doing things, but that's the beauty of openness, you can always hack your way around.
Feel free to post doubts in the comments.

Saturday, September 13, 2014

MediaWiki a wonderful CMS: Get the TurnKey appliance to use PostgreSQL

To configure the TurnKey appliance

http://www.turnkeylinux.org/mediawiki

to use PostgreSQL as the database backend, take this script as a reference:

apt-get install postgresql php5-pgsql
dpkg-reconfigure locales
createuser -S -D -R -P -E wikiuser
createdb -O wikiuser --locale=es_MX.UTF-8 --template=template0 wikidb
psql -d wikidb -c "grant select on pg_ts_config to wikiuser;"
psql -d wikidb -c "grant select on pg_ts_config_map to wikiuser;"
psql -d wikidb -c "grant select on pg_ts_dict to wikiuser;"
psql -d wikidb -c "grant select on pg_ts_parser to wikiuser;"
mv /var/www/config/LocalSettings.php /var/www/config/LocalSettings.php.original
php /usr/share/mediawiki/maintenance/install.php --dbname=wikidb --dbtype=postgres --dbuser=wikiuser --dbpass=yourdbpass --dbport=5432 --server=http://192.168.122.92 --scriptpath='' --pass=wikipass --lang=es yourwikiname admin
service apache2 restart

Friday, August 30, 2013

PostgreSQL. Handling timestamps with timezone

Regarding timestamps and their proper handling, the world of databases is packed with bittersweet stories. With PostgreSQL this issue is rather straightforward if you are aware of the following principles:

- PostgreSQL doesn't store timezone metadata, it converts every timestamp with timezone you throw at it to UTC, and stores it that way.
- If you use timestamp without timezone fields, they're stored using the local timezone valid for the client session, in this way, we know nothing about relative to what that timestamp was stored in the database.
- To present timestamps with timezone to users it converts "on the fly" timestamps to the client's configured timezone.

show timezone;
    TimeZone    
----------------
 Mexico/General

select now();
             now              
------------------------------
 2013-08-30 00:39:55.82538-05

CREATE TABLE test_timestamptz(
id serial primary key,
mytimestamptz timestamp with time zone,
mytimestamp timestamp without time zone
);

INSERT INTO test_timestamptz SELECT 1,now(),now();

select * from test_timestamptz;
 id |        mytimestamptz         |        mytimestamp        
----+------------------------------+---------------------------
  1 | 2013-08-30 00:48:19.55422-05 | 2013-08-30 00:48:19.55422

set timezone = 'UTC';

select * from test_timestamptz;
 id |        mytimestamptz         |        mytimestamp        
----+------------------------------+---------------------------
  1 | 2013-08-30 05:48:19.55422+00 | 2013-08-30 00:48:19.55422

There are some provisions for us not to shoot ourselves in the foot when dealing with timezones, but I'll write about that later.
Cheers!

Friday, August 16, 2013

PostgreSQL. pg_memcache

In high performance applications caching alleviates a great deal of database workload, let's propose a little example using pg_memcache.

Prerequisites:

yum install memcached libmemcached libmemcached-devel
wget 

Download and install pg_memcache

wget http://pgfoundry.org/frs/download.php/3018/pgmemcache_2.0.6.tar.bz2
tar -xzf pgmemcache_2.0.6.tar.bz2
cd pgmemcache
make
sudo make install 

Tuesday, August 6, 2013

Pentaho Data Integration (Kettle) on Fedora 19 x86_64 Core dumped. Problematic frame: C [libsoup-2.4.so.1]

Symptom:
You issue the script spoon.sh on Fedora 19 x86_64, it breaks with the following error:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00000031e926d9c1, pid=10487, tid=140492000786176
#
# JRE version: 7.0_25-b15
# Java VM: Java HotSpot(TM) 64-Bit Server VM (23.25-b01 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libsoup-2.4.so.1+0x6d9c1]  soup_session_feature_detach+0x11
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/darwin/data-integration/hs_err_pid10487.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
spoon.sh: line 166: 10487 Aborted                 (core dumped) "$_PENTAHO_JAVA"

Cause:
It turns out that it is an eclipse issue as depicted here:

After updating to GNOME 3.8 with libsoup 2.42.0, Eclipse reliably crashes after a few seconds of using:

https://bugs.eclipse.org/bugs/show_bug.cgi?id=405786

Solution:
the workaround is modifying your spoon.sh file as follows

OPT="$OPT $PENTAHO_DI_JAVA_OPTIONS -Djava.library.path=$LIBPATH -DKETTLE_HOME=$KETTLE_HOME -DKETTLE_REPOSITORY=$KETTLE_REPOSITORY -DKETTLE_USER=$KETTLE_USER -DKETTLE_PASSWORD=$KETTLE_PASSWORD -DKETTLE_PLUGIN_PACKAGES=$KETTLE_PLUGIN_PACKAGES -DKETTLE_LOG_SIZE_LIMIT=$KETTLE_LOG_SIZE_LIMIT -Dorg.eclipse.swt.browser.DefaultType=mozilla"

Note that the thing you need to add to the JAVA options is:
 -Dorg.eclipse.swt.browser.DefaultType=mozilla

Should problems still arise, try adding this option too:
 -Dorg.eclipse.swt.browser.XULRunnerPath=/usr/lib64/xulrunner/
or the equivalent path in your system to xulrunner

and that's it, I hope it helps.


Saturday, June 22, 2013

python. plotting the normal distribution in 3,2,1, ... go!

Just for reference in case I needed later or should someone find it useful, this is the shortest path I've found to graphing the normal distribution using python


#!/bin/python
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import norm
items = []
with open('data','r') as f:
    for line in f:
        items.append(int(line))
plt.plot(items, norm.pdf(items,np.mean(items),np.std(items)))
plt.show()

Tuesday, June 18, 2013

python. geocoding whith geopy

What if you have a bunch of mail addresses and need to get approximate coordinate positions? That's what geocoding is all about, and with geopy is as easy as it gets.


from geopy import geocoders
g = geocoders.GoogleV3()
place, (lat,lng) = g.geocode("AV. DIVISION DEL NORTE NO. 911-A COLONIA DEL VALLE BENITO JUAREZ")
print "%s: %.5f, %.5f" % (place, lat, lng)

And then, as if it were some sort of magic:


Avenida División del Norte 911, General Anaya, Benito Juarez, Mexico City, Federal District, Mexico: 19.35731, -99.15289


Geocoding

geopy includes geocoder classes for MediaWiki (with the GIS extension), Semantic MediaWiki, the Google v2 geocoder, the Yahoo! geocoder,geocoder.usBing Maps API, and GeoNames. The geocoder classes are located in geopy.geocoders.


https://code.google.com/p/geopy/wiki/GettingStarted

Wednesday, May 29, 2013

splitting words into syllables with python

There is one very easy way to achieve it:

pip install pyhyphen

from hyphen import Hyphenator
from hyphen.dictools import *

for lang in ['es_MX', 'es_SP']:
        if not is_installed(lang): install(lang)
h_mx = Hyphenator('es_MX')
print h_mx.syllables(u"modernismo")

For more info:

https://pypi.python.org/pypi/PyHyphen/0.9.3

Just in case you wondered, I need this as I'm working on mastering a mnemonic system based on the phonetic sound of words.




Monday, May 27, 2013

A virtual IP? What for? #linux

Layman's definition.
 It's an additional IP you assign to an interface by creating a virtual interface upon a real interface.

What for?
 For high availability scenarios, in which a service connects to a virtual IP pointing to a service (i.e. a database) .
1. The service breaks and a standby takes over bringing up the same virtual IP address automatically.
2. The application server reconnects to the standby as if nothing happened.

What kind of IP?
 Any available IP on the same network as the primary interface.

Example:

- An application server is associated to a database on 192.168.0.222/24
(address pointing to the database's virtual interface)
- The database server breaks down
- Bring down its virtual interface (if possible)
- On the standby server bring up the virtual interface with a virtual IP address.
ifconfig p2p1:0 inet 192.168.0.222 netmask 255.255.255.0

- Refresh the arp cache on a neighbouring server (that in which the application making use of the service is hosted).
arping -U -w 1  -I p2p1 192.168.0.222

(no response is expected)

- From the same neighbour:

ping 192.168.0.222
64 bytes from 192.168.0.222: icmp_seq=1 ttl=64 time=0.538 ms
64 bytes from 192.168.0.222: icmp_seq=2 ttl=64 time=0.318 ms
64 bytes from 192.168.0.222: icmp_seq=3 ttl=64 time=0.320 ms
64 bytes from 192.168.0.222: icmp_seq=4 ttl=64 time=0.623 ms
64 bytes from 192.168.0.222: icmp_seq=5 ttl=64 time=0.278 ms
64 bytes from 192.168.0.222: icmp_seq=6 ttl=64 time=0.287 ms
64 bytes from 192.168.0.222: icmp_seq=7 ttl=64 time=0.197 ms
64 bytes from 192.168.0.222: icmp_seq=8 ttl=64 time=0.280 ms

- Provided a standby database takes over as a part of a failover procedure, it must take over the same virtual IP.

- Care must be taken to make sure that the old database server's virtual interface gets brought down so it doesn't cause havoc (lookup the term STONITH)
- If using Dynamic Host Configuration Protocol make sure the virtual IP doesn't get assigned to any host automatically by restricting automatic leases to a lower range.

References:
http://itsecureadmin.blogspot.mx/2007/02/creating-virtual-ip-addresses-on-linux.html
http://en.wikipedia.org/wiki/Arping

Tuesday, May 7, 2013

python-fabric. Easing multiple servers administration

I'm deploying a test scenario involving 16 pristine virtual machines and as a lazy administrator I'd hate to do the same configuration 16 times, that's what python-fabric is useful for.

From the official website:
http://docs.fabfile.org/en/1.6/tutorial.html

What is Fabric?

As the README says:
Fabric is a Python (2.5 or higher) library and command-line tool for streamlining the use of SSH for application deployment or systems administration tasks.
More specifically, Fabric is:
  • A tool that lets you execute arbitrary Python functions via the command line;
  • A library of subroutines (built on top of a lower-level library) to make executing shell commands over SSH easy and Pythonic.
Naturally, most users combine these two things, using Fabric to write and execute Python functions, or tasks, to automate interactions with remote servers. Let’s take a look.


This is my first script which copies my public key into the test machines:


#!/bin/python
from fabric.api import local,env

hosts = []
for i in range(1,17):
   hosts.append( "box%d" % i)

env.hosts= hosts
def test_ssh():
      try:
         local("""ssh -o PasswordAuthentication=no root@%s "ifconfig | grep  -E  'inet.*192.168.0' | sed 's/\s\s*/ /g' | cut -d' ' -f3" """%env.host)
      except SystemExit as e:
         local("ssh-copy-id root@%s"%env.host)

If you have to administer multiple machines on a daily basis, I advice you give python-fabric a try.

Sunday, April 28, 2013

Sunday project. My desperate dictionary "attack"

I was stuck with one word of this game:
https://play.google.com/store/apps/details?id=de.lotum.whatsinthefoto.es
you look at 4 pictures and guess the word that correlates them. I couldn't go further so, I decided to use brute force.

Objective: Get all the possible permutations of the given letters (12 ) of the given size (5) and match them against a dictionary.


darwin@evolution:~/spwords> wget http://www.insidepro.com/dictionaries/Spanish.rar
darwin@evolution:~/spwords> unrar Spanish.rar


darwin@evolution:~/spwords> file Spanish.dic
Spanish.dic: ISO-8859 text, with CRLF line terminators

The database has unicode encoding, so lets do some encoding conversion.

idarwin@evolution:~/spwords> iconv -f ISO-8859-1 -t UTF-8 Spanish.dic > Spanish.dic.unicode
darwin@evolution:~/spwords> dos2unix Spanish.dic.unicode


I'm using my favorite RDBMS (PostgreSQL) for the word matching.

postgres=# CREATE TABLE words(id serial primary key, word varchar, word_unnacented varchar);
NOTICE:  CREATE TABLE creará una secuencia implícita «words_id_seq» para la columna serial «words.id»
NOTICE:  CREATE TABLE / PRIMARY KEY creará el índice implícito «words_pkey» para la tabla «words»
CREATE TABLE
postgres=# \copy words(word) FROM 'Spanish.dic.unicode'
postgres=# select count(*) from words;
 count
--------
 413527


postgres=# CREATE EXTENSION unaccent;
CREATE EXTENSION
postgres=# UPDATE words set word_unnacented = unaccent(word);
UPDATE 413527
postgres=# CREATE INDEX words_word_idx ON words(word_unnacented varchar_pattern_ops);
CREATE INDEX

postgres=# create extension plpython2u;

 CREATE OR REPLACE FUNCTION match_word(letters varchar,len int) RETURNS TABLE (match varchar)
AS $$
import itertools
result = []
for i in itertools.permutations(letters,len):
        rs = plpy.execute("SELECT word_unnacented FROM words WHERE word_unnacented = '%s' " % ''.join(i).lower())
        result += [(r['word_unnacented']) for r in rs]
return result
$$ LANGUAGE plpython2u;

postgres=# select distinct match_word('srtkjncnmonu',5) order by 1;
match_word 
------------
 comun
 cornu
 crujo
 cruos
 cujon
 cunos
 curso
 curto
 cutos
 junco
 junos
 junto
 jurco
 juros
 justo
 moscu
 munon
 muros
 murto
...

etc.

It took 10 seconds to yield the results (Intel Pentium Dual Core) .

It turns out that the correct word, which I found through this "attack", was "curso", that had in my opinion nothing to do with the pictures. Now I can continue playing =-) .





Thursday, May 10, 2012

Python.-Closures.Functions that build other functions

I'm studying Dive Into Python 3 and I'm pretty much in love with it.
http://getpython3.com/diveintopython3/

From Chapter 6. Closures and Generators.
Guess what? Functions are just other kind of objects in Python!

The technique of using the values of outside parameters within a dynamic function is called closures
import re

def build_match_and_apply_functions(pattern,search,replace):
    '''
    builds dynamically a tuple containing two functions (matches_rule,apply_rule) for each tuple (pattern,search,replace) received
    the constants patter,search and replaced get substituted by the actual parameters passed to the function accordingly
    '''
    def matches_rule(word):
        return re.search(pattern,word)
    def apply_rule(word):
        return re.sub(search,replace,word)
    return (matches_rule,apply_rule)

patterns = \
'''
for each of these patterns two functions will be built a match_rule  and an apply_rule functions
'''
(
    ('[sxz]$','$','es'),
    ('[^aeioudgkprt]h$','$','es'),
    ('(qu|[^aeiou])y$','y$','ies'),
    ('$','$','s')
)


'''
this is called a list comprehension in Python terms
in this case we have as a result, a list containing tuples of match_rule and apply_rule functions
'''
rules = [build_match_and_apply_functions(pattern,search,replace)
         for (pattern, search,replace) in patterns]

print(rules)

def plural(noun):
    for matches_rule, apply_rule in rules:
        if matches_rule(noun):
            return apply_rule(noun)
        
nouns = ["sufix","trash","lady","cat","math","boy","day",'pita','vacancy']
for noun in nouns:
    print(plural(noun))

I'm still in the process of assimilation, and this is pretty cool stuff.

Wednesday, April 25, 2012

Python - An insignificant script to grade tests

I'm just backing up this script to track my progress with python, it ain't a great thing, it just grades a bunch of test of some students.

f = open('/home/rene/Dropbox/respuestas.txt','r')
o = open('/home/rene/Dropbox/hoja_resultados.txt','w')
r = open('/home/rene/Dropbox/cals_uaem.txt','r')
respuestas = f.readline().split()
secciones = f.readline().split()    
temas = f.readline().split(',')
f.close()
reactivos = [""]*29
counter = 0
for i in r:
    reactivos[counter] = i.split()
    counter = counter + 1
r.close()
temas_estudiar = []
preguntas_por_seccion = {'BI':9,'ETL':6,'SQL':4,'BI-Documentos':3}
aciertos = {'BI':0,'ETL':0,'SQL':0,'BI-Documentos':0}
errores = {'BI':0,'ETL':0,'SQL':0,'BI-Documentos':0}
ptc_aciertos = {'BI':0.0,'ETL':0.0,'SQL':0.0,'BI-Documentos':0.0}
for i in range(0,29):
    for j in range(0,22):
        if (j!=5):
            if (respuestas[j] == reactivos[i][j+1]):
                aciertos[secciones[j]] = aciertos[secciones[j]] + 1
            else:
                errores[secciones[j]] = errores[secciones[j]] + 1
                temas_estudiar.append(temas[j])
        else:
            aciertos[secciones[j]] = aciertos[secciones[j]] + 1       
    num_aciertos = aciertos['BI']+aciertos['SQL']+aciertos['ETL']+aciertos['BI-Documentos']
    
    o.write("\nNum.examen: ")
    o.write(reactivos[i][0]+'\n') 
    o.write("score: "+str(num_aciertos)+" de 22\n")  
    o.write("Aciertos\n")
    o.write("BI ETL SQL BI-Documentos\n") 
    o.write(str(aciertos['BI'])+" "+str(aciertos['ETL'])+" "+str(aciertos['SQL'])+" "+str(aciertos['BI-Documentos'])+'\n')
    o.write("Errores\n")
    o.write("BI ETL SQL BI-Documentos\n") 
    o.write(str(errores['BI'])+" "+str(errores['ETL'])+" "+str(errores['SQL'])+" "+str(errores['BI-Documentos'])+'\n')
    o.write("Promedios\n")
    o.write("BI ETL SQL BI-Documentos\n") 
    o.write("%.1f"%(float(aciertos['BI'])/float(preguntas_por_seccion['BI']))+" "+"%.1f"%(float(aciertos['ETL'])/float(preguntas_por_seccion['ETL']))+" "+"%.2f"%(float(aciertos['SQL'])/float(preguntas_por_seccion['SQL']))+' '+"%.2f"%(float(aciertos['BI-Documentos'])/float(preguntas_por_seccion['BI-Documentos']))+'\n')
    o.write("Promedio final: "+"%.2f"%(float(num_aciertos)/22.0)+"\n")
    aciertos['BI'] = 0
    aciertos['ETL'] = 0
    aciertos['SQL'] = 0
    aciertos['BI-Documentos'] = 0
    errores['BI'] = 0
    errores['ETL'] = 0
    errores['SQL'] = 0
    errores['BI-Documentos'] = 0
    o.write("Temas a estudiar: \n")
    for i in temas_estudiar:
        o.write(i+",")
    o.write('\n')
    del temas_estudiar[:]
o.close()

Things I learned:
The write function doesn't put a new line character at the end.
http://docs.python.org/release/1.5.1p1/tut/range.html
The range function is not inclusive of its second parameter.
To get a better understanding of formatting operations.
http://docs.python.org/release/2.4.4/lib/typesseq-strings.html
To use the "del" statement to clear a list, i.e. del myList[:]
To use the split function.
To make lists of lists such as:
mlist3 = [[0]*3]*3
http://www.daniweb.com/software-development/python/threads/58916/making-a-list-of-lists
To use dictionaries

Saturday, April 21, 2012

Weave Maker. Rapid Web Application Development

I'm giving Weave Maker a try, I love coding, but coding algorithms ... the logic behind applications, I don't fancy myself as a GUI developer nor I think I would be good at it... anyway:

http://www.wavemaker.com/product/

Here's a screenshot of my hello weavey world web application that only took 2 minutes to build:


Did I mention that it has Apache license?