יום רביעי, 27 בנובמבר 2013

Tail, grep, and stdio buffering

I was engaged in some troubleshooting once and needed to follow a log and specifically capture some string as soon as it appears in the log.
I have started my way, believing that using tail logfile | grep string will suffice, but after a few minutes of struggling to understand a certain delay between the actual server responses and the output on my screen I have realised there is some buffer at work here.
some google query afterwards I have came to acknowledge a known buffer that exists specifically in linux distributions (wasn’t able to replicate in solaris) which in order to override needs the flag --line-buffered added to grep.
so now that I run
 tail logfile | grep --line-buffered string
I expect to immediately (as possible) see the requested string shown on the stdio as soon as it is being written on the logfile.

יום רביעי, 8 במאי 2013

JMX Connection to Sonic MQ 8.5

For monitoring needs I was requested to set up a JMX connection from jConsole to Progress Sonic MQ 8.5. As far as my eyes can tell, this method will be relevant for previous versions of Sonic MQ, I will perhaps get to test it on 7.5 one day in my qa prod.
According to their overview of this matter;
In order to access Sonic MBeans, the container to be analyzed needs no special setup or
libraries. Some Sonic libraries must be on the JMX console’s classpath:
● mgmt_client.jar
● sonic_Client.jar
If the container is hosting a SonicMQ broker, the following library must also be on the
JMX console’s classpath:
● sonic_mgmt_client.jar
that means that if you want to initiate the connection from a remote computer you have to take 2 prerequisites in mind:
  1. Network: Make sure tcp port used for the containers on both sides is open
  2. Java Classes: you will have to have all three jars from Sonic MQ installation and copy them to the remote computer running the JMX client.
If you plan on initiating this connection locally, just make sure you have your paths straight as they will come in handy when running the JMX client.
The example brought from Progress documentation is referring to a Windows version of jConsole:
1. As JConsole is packaged with Sun 1.5 and newer Java Development Kits (JDKs),
install a JDK on the system where the JConsole will run. In this example, the JDK is
installed at C:\jdk1.6.0_11.
2. Copy or install the required libraries onto the local system. In this example, the Sonic
Domain Manager (which includes the required libraries) is installed at
C:\Program Files\Progress\Sonic on the same system where JConsole will run.
3. Start the Sonic management container; in this example, Domain1.DomainManager.Progress SonicMQ Configuration and Management Guide 8.5 643
Using JConsole to Examine a Sonic Domain Manager
4. Enter (as one line):
C:\jdk1.6.0_11\bin\java.exe \
-Djmx.remote.protocol.provider.pkgs=com.sonicsw.jmx.remote.protocol \
-cp C:\jdk1.6.0_11\lib\tools.jar;
C:\Program Files\Progress\Sonic\MQ8.0\lib\sonic_mgmt_client.jar;
C:\Program Files\Progress\Sonic\MQ8.0\lib\mgmt_client.jar;
C:\Program Files\Progress\Sonic\MQ8.0\lib\sonic_Client.jar \
5. In the Java Monitoring & Management Console window, the New Connection dialog
box lets you define the connection:
On the local system, you could choose the process identifier (PID) to make the
connection. For remote systems, use the syntax:
● host – hostname (or localhost)
● port – the broker TCP port accepting connections
● domain – the container’s Sonic domain name
● container – the container’s Sonic container name
For this example, the Remote Process entry would be:
In order to adapt this command to linux (OpenSuse 11.0) I merely changed the java syntax and that’s what I got:
/usr/java/default/bin/java -Djmx.remote.protocol.provider.pkgs=com.sonicsw.jmx.remote.protocol -cp /usr/java/default/lib/tools.jar:/usr/java/default/lib/jconsole.jar:/home/user/installations/progress/sonic/MQ8.5/lib/sonic_mgmt_client.jar:/home/user/installations/progress/sonic/MQ8.5/lib/mgmt_client.jar:/home/user/installations/progress/sonic/MQ8.5/lib/sonic_Client.jar:: sun.tools.jconsole.JConsole
NOTICE: You can not connect to Sonic MQ JMX through the VM PID, so you will have to use the complete url format as suggested before: service:jmx:sonic+tcp://localhost:2506/DomainContainer
hopefully this will save you the time I had to put into this matter. Would love to hear if you any other takes on this matter as well.

jConsole connection to Progress Sonic MQ 8.5

יום חמישי, 25 באפריל 2013

openssl usage: remove passphrase from rsa key

This issue maybe outdated and probably has a 1000 better resolutions which consider security precautions, as this method does not.
It is risky to use it and I would never advise using this practice on production servers; however:
creating passphrase-less SSL keys has its twisted benefits for when you want to have ssl on apache/nginx reloading without needing any addon to provide the passphrase, you can use the following method to re-pack the keys after creating them:
cp www.domain.com.key passphrase.www.domain.com.key
openssl rsa -in passphrase.www.domain.com.key -out www.domain.com.key
again, this compromises your key and security. avoid from using it on production systems.

יום שישי, 29 במרץ 2013

How to package MobaXterm plugins with Cygwin

2015 update:The following post is somewhat redundant, MobaTek's dedicated team of developers already adpated a working port of apt-cyg including all of the steps below (sans the plugin packaging), preventing the need to pack your own plugins. however plugins can still be created to supply different versioning and configuration deployments where needed.

2013 Mar 29th
If you don’t know MobaXterm, here are a couple of words from their Developer’s website
MobaXterm is an enhanced terminal for Windows with an X11 server, a SSH client and several other network tools for remote computing (VNC, RDP, telnet, rlogin). MobaXterm brings all the essential Unix commands to Windows desktop, in a single portable exe file which works out of the box.
MobaXterm provides useful features for developers, sysadmins, webmasters and advanced computer users:
multitab terminal with embedded Unix commands (ls, cd, cat, sed, grep, awk, rsync, wget, …)
embedded X11 server for easily exporting your Unix/Linux display
a session manager with several network utilities: SSH, RDP, VNC, Telnet, Rlogin, FTP, SFTP and XDMCP
passwords management for SSH, RDP, VNC, SFTP (on demand password saving)
easy graphical file transfer using drag and drop during SSH sessions
advanced SSH tunnels creation tool (graphical port forwarding builder)
tasks automation using scripts or macros
I stumbled upon them when I was looking for an X-server solution for a friend which hadn’t the time nor the energy to setup his own cygwin/xterm/virtualbox. as I have dug into the spec and features, my passion grew stronger about this product as it seemed to be tailored for my own desires and needs.
in order to make myself feel at home using it, I had to know I am able to maintain my environment and to have the packages and libraries I need for my tasks, and the ability to control them, as it appears, their plugin system fits like a glove.
The concept is of a sandbox, whenever you start the program you get a clean-slate, vanilla instance of cygwin with some basic needed components, and if you have any plugins in your plugin directory, they are being loaded freshly every time. This does not mean you don’t have a persistent workspace as the program cleverly lets you dedicate a folder from your computer to be the ‘bridge’ or the home folder of your mobaxterm.
Here’s my take on building a mobaxterm plugin, it is very basic and relies heavily on a separate instance of cygwin which you can download from here.
1. install a cygwin instance, keep it minimal but with some tools you’ll probably want for the packaging process (I used tar, gz, wget, apt-cyg…)
2. in your cygwin prompt install apt-cyg. it is helpful for installing packages to your cygwin, and to get list of prerequisites for your op to work.
wget http://apt-cyg.googlecode.com/svn/trunk/apt-cyg
chmod +x apt-cyg
mv apt-cyg /usr/local/bin/
3. install the program you would like to pack ( technically you could build rather then pull ready packages, you may have to use it for some software cygwin/redhat had not patched and optimized to cygwin but my guess is you won’t need it anytime soon )
apt-cyg install package
4. after the package is installed, we know that all of our dependecies are met since apt-cyg is a good package manager. we would like to compile the list of software to pack. I do it by querying the apt-cyg db. i.e.
$ apt-cyg describe perl_vendor
Working directory is /home/dave/plugins/libglib2.0_0
Mirror is ftp://mirror.mcs.anl.gov/pub/cygwin
2013-01-12 01:18:58– ftp://mirror.mcs.anl.gov/pub/cygwin/setup.bz2
=> `.listing’
Resolving mirror.mcs.anl.gov (mirror.mcs.anl.gov)…, 2620:0:dc0:1800:214:4fff:fe7d:1b9
Connecting to mirror.mcs.anl.gov (mirror.mcs.anl.gov)||:21… connected.
Logging in as anonymous … Logged in!
==> SYST … done. ==> PWD … done.
==> TYPE I … done. ==> CWD (1) /pub/cygwin … done.
==> PASV … done. ==> LIST … done.

[ <=> ] 1,043 –.-K/in 0.003s

2013-01-12 01:19:00 (323 KB/s) – `.listing’ saved [1043]

Removed `.listing’.
2013-01-12 01:19:00– ftp://mirror.mcs.anl.gov/pub/cygwin/setup.bz2
=> `setup.bz2′
==> CWD not required.
==> PASV … done. ==> RETR setup.bz2 … done.
Length: 364917 (356K)

100%[=============================================================>] 364,917 186K/in 1.9s

2013-01-12 01:19:03 (186 KB/s) – `setup.bz2′ saved [364917]

Updated setup.ini

sdesc: “Additional vendor modules previously packaged in perl-5.10
ldesc: “Common modules to installtest, debug and report CPAN modules:
Pod::Escapes Pod::Simple Test::Pod Devel::Symdump Pod::Coverage
Test::Pod::Coverage Compress::Bzip2 IO::String Archive::Zip
Term::ReadKey Term::ReadLine::Perl Term::ReadLine::Gnu
XML::NamespaceSupport XML::SAX::Base XML::SAX XML::LibXML::Common
XML::LibXML XML::Parser Proc::ProcessTable YAML Config::Tiny
File::Copy::Recursive IPC::Run3 Probe::Perl Tee IO::CaptureOutput
File::pushd File::Which File::HomeDir Digest::SHA IPC::Run
Module::Signature URI HTML::Tagset HTML::Parser HTTP::Date
Encode::Locale LWP::MediaTypes HTTP::Message File::Listing
HTTP::Daemon WWW::RobotRules HTTP::Negotiate HTTP::Cookies Net::HTTP
WWW::RobotRules LWP Net::IP Digest::HMAC_MD5 Net::DNS Test::Reporter
Net::SSL common::sense JSON::XS JSON Metabase::Client::Simple
Sub::Install Data::UUID Sub::Exporter Params::Util Data::OptList
Data::GUID CPAN::DistnameInfo Metabase::Fact Test::Tester
Test::NoWarnings Config::Perl::V CPAN::Testers::Report
Test::Reporter::Transport::Metabase Capture::Tiny Devel::Autoflush
IPC::Cmd CPAN::Reporter Module::ScanDeps PAR::Dist Socket6
IO::Socket::INET6 B::Generate PadWalker Data::Alias”
category: Perl
requires: perl libssp0 libgcc1 libbz2_1 zlib0 libopenssl100 libncurses10 libreadline7 libncursesw10 libxml2 libiconv2 libexpat1 _autorebase cygwin
version: 5.14.2-3
install: release/perl/perl_vendor/perl_vendor-5.14.2-3.tar.bz2 2536307 5b3f36321cff52dc66c8048aa06f905a
source: release/perl/perl-5.14.2-3-src.tar.bz2 17627943 8a1d9eb3fd927b45b8c75fa8f590bcf0
I would then go through with this for every pack then recursively in the bold list. and when I have compiled a list of libraries, and apps I will continue to compiling a list of the needed files.
vi liblist
save the liblist file in a clean empty folder, we are getting to the secret sauce now.
for g in `cat liblist`do zcat /etc/setup/$g.lst.gz; done >filelist
for i in `cat filelist` ; do if [[ -d /$i ]]then echo mkdir -p ./$ielse echo cp –parents /$i ./fidone | sort -r> createlist
chmod +X createlist
Thats it, (hopefully XD) now just zip the files and folders (leave out liblist filelist and createlist) and rename the zip into .mxt3 and you should be good with your packed software.
This worked for me for building an irssi plugin which you can download here.