My dig into unix based systems

Process for buildin new kernel

2010-09-10T14:53:00.000-07:00

Build the kernel

a. Download kernel

b. Untar it
c. make menuconfig
d. This Command builds the images in the *parent directory*
/usr/bin/fakeroot make-kpkg --initrd --append-to-version=-custom kernel_image kernel_headers

Install the kernel
a. Move the packages to target machine and install

sudo dpkg --purge linux-headers-2.6.35.4-custom
sudo dpkg --purge linux-image-2.6.35.4-custom
sudo rm -f /boot/initrd.img-2.6.35.4-custom*

sudo dpkg -i linux-headers-2.6.35.4-custom_2.6.35.4-custom-10.00.Custom_amd64.deb
sudo dpkg -i linux-image-2.6.35.4-custom_2.6.35.4-custom-10.00.Custom_amd64.deb
sudo update-initramfs -c -k 2.6.35.4-custom

b. Update initramfs
sudo update-initramfs -c -k 2.6.35.4-custom

c. Update grub.conf.
Add initrd file path.
Edit the path of root device in kernel options from /dev/hda1 to /dev/sda1.

Reboot.. Hopefully it should come up with no kernel panics.

2010-02-17T15:56:00.001-08:00

I would like to start a linux kernel tutorial. The idea of this blog is to provide me with a drive to complete the book. Linux Kernel Development by Robert Love.

You can access the pdf here.

ScreenCast in Ubuntu

2010-01-05T12:38:00.000-08:00

a. Install gtk-recordmydesktop
Record your first screen cast using this software

b. Install Audacity: To remove noise
Get a wav file to be used with Audacity
ffmpeg -i out.ogv out.wav

c. Merge the edited wav file with original video file(Also convert it to avi)
mencoder -audiofile out.wav -oac mp3lame -ovc xvid -xvidencopts fixed_quant=4 -o out.avi out.ogv

Postgres from python

2009-08-07T12:22:00.000-07:00

This snippet contains no validation whatsoever.

import psycopg2

connection = psycopg2.connect('dbname=product user=postgres host=127.0.0.1')
mark = connection.cursor()
query = """INSERT INTO PRODUCT(product_id, product, attribute) VALUES(%d, '%s', '%s')"""
statement = query % (1, "name", "attribute")
mark.execute(statement)
connection.commit()

GeoIP with Maxmind's Geolite

2009-08-06T09:43:00.000-07:00

I was searching for a python tool to determine approximate location from IP address.

You need to download 3 tars:

a. Geolite C library- Python module internally uses this C library
http://geolite.maxmind.com/download/geoip/api/c/GeoIP.tar.gz

tar -xvzf GeoIP.tar.gz
cd GeoIP-1.4.6
./configure --libdir=/usr/lib/python2.5/site-packages/
make
make install

b. Python api:
http://geolite.maxmind.com/download/geoip/api/python/GeoIP-Python-1.2.4.tar.gz

tar -xvzf GeoIP-Python-1.2.4.tar.gz
cd GeoIP-Python-1.2.4

# Edit setup.py to adjust the library_dirs and include_dirs
library_dirs = ['/usr/lib/python2.5/site-packages/'],

python setup.py build
sudo python setup.py install

c. Download Data file (This you probably need to update periodically)
http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz

Test your setup
--------------------------
(From: http://stackoverflow.com/questions/1163136/how-to-detect-the-country-and-city-of-a-user-accessing-your-site)

I do not completely understand how variables in library are accessed from python.. So I export this variable
export LD_LIBRARY_PATH=/usr/lib/python2.5/site-packages/

import GeoIP
gi = GeoIP.open("GeoLiteCity.dat", GeoIP.GEOIP_INDEX_CACHE | GeoIP.GEOIP_CHECK_CACHE)
print gi.record_by_name("74.125.67.100") # a www.google.com IP

{'city': 'Mountain View', 'region_name': 'California', 'region': 'CA', 'area_code': 650, 'time_zone': 'America/Los_Angeles', 'longitude': -122.05740356445312, 'country_code3': 'USA', 'latitude': 37.419200897216797, 'postal_code': '94043', 'dma_code': 807, 'country_code': 'US', 'country_name': 'United States'}

Simple unit convertor in python

2009-07-06T14:23:00.001-07:00

This function uses the google calculator api
http://www.google.com/ig/calculator?hl=en&q=1EUR%3D%3FUSD


import urllib, urllib2
import socket

# Crawl header #
default_header = { 'User-Agent' :
    'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.5) Gecko/2008121622 Ubuntu/8.10 (intrepid) Firefox/3.0.5;'}

def getpage(pageurl, pageargs = None, pageheaderargs = {}):
  fail = 0
  content = reply_header = None
  pageheaders = default_header
  pageheaders.update(pageheaderargs)
  while True:
    print pageurl
    retry = 0
    try:
      params = urllib.urlencode(pageargs) if pageargs else None
      req = urllib2.Request(pageurl, params, pageheaders)
      page = urllib2.urlopen(req)
      content = page.read()
      reply_header = str(page.headers).split('\r\n')
      page.close()
    except urllib2.HTTPError, e:
      retry = 1
    except IOError, e:
      retry = 1
    except socket.timeout, e:
      retry = 1
    except socket.sslerror, e:
      retry = 1
    if retry:
      fail += 1
      if fail == 5:
        print 'Failed to retrieve' + pageurl + ': ' + str(e)
        return (reply_header, content)
      else:
        continue
    break
  return (reply_header, content)

def convert(unit1, unit2):
  query = '1%s=?%s' % (unit1, unit2)
  (header, content) = getpage('http://www.google.com/ig/calculator?hl=en&q=' + query)
  reresult = re.search('rhs: \"(.*?) ', content)
  result = reresult.expand(r'\1') if reresult else 'Error'
  return result if result != '' else 'Error'

Python string strip function..

2009-06-24T07:17:00.000-07:00

I just realized a super blunder, I have committed all my life

url = 'abcdc.com'
print url.strip('.com')
Expect: abcdc
Resut: abcd

Instead try
url.rsplit('.com', 1)

strip function takes in an array of characters and chops any pattern of characters in the array.. :(

Lambda function and class in python

2009-06-12T18:04:00.000-07:00

I had a hash with hash value an object. I wanted to sort by one of the elements (integer) of the class.. Here is neat way..


class cl:
  def __init__(self, val):
    self.val = val

arr = {'a' : cl(10), 'ab' : cl(75), 'bd' : cl(50)}

g = lambda (x, y): y.val
for (x, y) in sorted(arr.iteritems(), key = g):
  print x + ' ' + str(y.val)

Mercurial on code.google

2009-06-03T13:56:00.000-07:00

I played with mercurial on code.google.. Here are some simple instructions

a. Create the project in the interface
b. Create a local repository

hg clone https://USERNAME@PROJECT.googlecode.com/hg PROJECTDIR

When it prompts for a password, key-in the password available on
http://code.google.com/hosting/settings (Profile > Settings)

I created trunk directory and stored all the code.. (to allow branching).. This probably not mercurial way and is svnish..

c. I prefer to edit wiki through mercurial..

hg clone https://wiki.PROJECT.googlecode.com/hg wiki-PROJECTDIR

"reset is not a function" - Firebug error

2009-03-17T13:22:00.000-07:00

I am a newbee to javascripting and I was stuck at reseting a form to default values for 2 full days..

Firebug kept saying
$("#addpropertyForm")[0].reset is not a function

Turned out that I had named the reset button as reset...!!
input type="reset" name="reset".....

Django serialization - Deserialization

2009-03-13T13:16:00.000-07:00

I have a table
desc property;
+--------------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------------+--------------+------+-----+---------+----------------+
| property_id | int(11) | NO | PRI | NULL | auto_increment |
| property_name | varchar(30) | NO | | NULL | |
| property_address_1 | varchar(120) | NO | | NULL | |
| property_address_2 | varchar(120) | YES | | NULL | |
| property_city | varchar(30) | NO | | NULL | |
| property_state | varchar(30) | NO | | NULL | |
| property_pincode | int(7) | NO | | NULL | |
| property_country | varchar(30) | NO | | NULL | |
| property_user_id | int(11) | NO | MUL | NULL | |
| property_price | int(20) | NO | | NULL | |
+--------------------+--------------+------+-----+---------+----------------+

Serialize

python manage.py shell
Python 2.5.2 (r252:60911, Oct 5 2008, 19:24:49)
>>> from users.models import Property
>>> from django.core import serializers

>>> json = serializers.serialize("json", Property.objects.filter(pk=2)
>>> print json
[{"pk": 1, "model": "users.property", "fields": {"property_user_id": 1, "property_state": "aa", "property_country": "aa", "property_pincode": 323, "property_city": "aa", "property_price": 1, "property_address_1": "aa", "property_name": "aa", "property_address_2": "aa"}}]

Deserialize
>>> json = '[{"pk": 2, "model": "users.property", "fields": {"property_user_id": 2, "property_state": "aa", "property_country": "aa", "property_pincode": 323, "property_city": "aa", "property_price": 1, "property_address_1": "aa", "property_name": "aa", "property_address_2": "aa"}}]'

>>> for obj in serializers.deserialize("json", json):
... obj.save()

I finally dint use it because I dint know how to convert a UTF encoded dictionary to string.. shame on me..!!

foreign key across different mysql engines

2009-03-11T12:44:00.000-07:00

I had a pre existing table(MYISAM) and I was creating new table innodb.
Dint realize for a long time that you cant have a foreign key relationship across 2 tables of different engines.
Error that you would see are

mysql> alter table property add (FOREIGN KEY(id) REFERENCES AUTH_USER(id));
ERROR 1005 (HY000): Can't create table 'sportland.#sql-134d_a' (errno: 150)
mysql> show engine innodb status;
------------------------
LATEST FOREIGN KEY ERROR
------------------------
090311 15:36:27 Error in foreign key constraint of table sportland/#sql-134d_a:
FOREIGN KEY(id) REFERENCES AUTH_USER(id)):
Cannot resolve table name close to:
(id))

took me sometime to figure out the issue..!!

Qemu and Debian

2008-03-22T10:01:00.000-07:00

After a few hiccups, I had a new debian machine running on Qemu in my Ubuntu Gutsy machine

1. sudo apt-get install qemu
2. wget http://cdimage.debian.org/debian-cd/4.0_r3/i386/iso-cd/debian-40r3-i386-CD-1.iso
Ensure the md5 check sum matches with the site(md5sum debian-40r3-i386-CD-1.iso of the file should match what is there on the server). This would save a lot of trouble. Don't skip it.
3. Create the image file
dd if=/dev/zero of=rootfile bs=1M count=2048
4. qemu -cdrom debian-40r3-i386-CD-1.iso -hda rootfile -net user -net nic -boot d
Install only the base system if you want to save time
5. If you complete the above step... Voila you can now boot into your brand new debian machine
qemu -hda rootfile -net user -net nic
Boot into it and get excited about seeing the shell prompt.

No wonder I soooooo love debian..

New kubuntu system..

2008-03-15T01:46:00.000-07:00

Here is something I have to brag out. I scrapped the default (only prescribed) OS on my office laptop and installed Kubuntu- gutsy... What a relief..

Must haves in no particular order
1. Amarok
2. Firefox
3. Compiz
4. Pidgin
5. KNetworkManager
6. Ktorrent (Yes in office laptop.. cant live without it..)
7. Kontact
8. AventailLinuxconnect..

Voila.. I am up and running (Running.. no flying high)...

Life has never been better..
apt-get... never ending list..

Torn between C and Java

2008-03-12T09:48:00.000-07:00

I started my career 3 years back.. Fresh out of college, I stepped into the wonderful world of C and Linux with a great mentor. I dint realise how lucky I am. I have thrown all that away for a few more bucks and great company name. Tomorrow I shall be committing the sin of opening an Eclipse for the first time. Shame on me...

Eternal love - Assembly lines

2008-02-24T11:32:00.000-08:00

I decided to brush some of basics, before I get on with my life... I picked up random stuff and started reading from middle.

I. First stop
http://asm.sourceforge.net/articles/linasm.html
Assembly
1. AT & T syntax and Intel

a. AT & T prefix % for registers and $ for immeds
movl $1 %eax
movl $0xff %ebx
int $0x80
b. Intel does not use prefix for registers.. mark immeds with b (binary and hexa)
mov eax, 1
mov ebx, 0ffh
int 80h

Note: directions are opposite .. AT&T src dst .. Intel dst src

2. Memory operands
a. Intel
mov ebx, [ebx]
mov ebx, [ebx + 3]
b. AT&T
movl (%ebx), eax
movl 3(%ebx), eax

a. Intel
instr foo segreg:[base + index * scale + disp]
sub eax, [ebx + ecx * 4h - 20h]
b. AT&T
instr %segreg:disp(base, index, scale), foo
sub -0x20(%ebx, %ecx, 0x4), eax

3. Suffix indicate length of operands
l - long, w - word, b - byte

Intel
mov al, bl
mov ax, bx
mov eax, dword ptr [ebx]

AT&T
movb %bl %al
movw %bx, %ax
movl %eax, %ebx

I dint understand something here... free

II. Next stop
http://www.scribd.com/doc/6900/PC-Assembly-Tutorial

1.
6897 - In nibbles (bits) - 1 1010 1111 0001
6897 - In hex 1 a f 1
(Group from right.. This is why hex is important... Hex compresses bits and there is a direct conversion possible)

2. ASCII - Maps simple characters like A - 65 (I guess).. B-66
Uses 1 Byte

To accommodate more characters (many other languages) unicode is used
Uses 2 bytes
(Thats all there to unicode.. and the word sounded like a jargon to me all these days.. which stone was I living under)

3. Compilers convert high level language programs into machine specific instructions for that particular architecture (duh 4 years of education.. you should know it by now...)

4. I remember wagely, the joy of tying instructions in hex on a 8085 / 8086.. and getting the result. World was lot more simple atleast in college

8088 / 8086-
Must read - http://www.cpu-world.com/Arch/8088.html
16 bit registers - Hence segments of size 64 KB possible
Hence to access more than 64KB(1 MB), the special segment registers CS - code, DS - Data, SS - Stack, Extra ES
1. 8088 16 bits
(address) - Lower order byte
(address + 1) - Higher ordere byte
2. 80386 32 bits
(address) - segment lower order
(address + 1) - segment higher order
(address + 2) - offset lower order
(address + 3) - offset higher order
address (segment * 16) + offset
Code segment -
64KB segment, Jump can be short jump within the segment or long jump outside the segment

Reserved locations
0000h - 03ffh ( Interrupt vectors - 32 bit format segment:offset)
fff0h - ffffh (reset starting point)

Code segment: holds the code
All instructions that uses the IP (instruction pointer) retrieves the instruction using CS * 16 + IP

Stack segment:(address) -
All access to Base pointer (BP) and stack pointer(SP) are offsets in segment pointed to SS register.

Data segment:
All access by AX, BX, CX, DX, index registers (SI, DI) are offsets into segment pointed to be DS register

Extra Segment
DI by default is used with data segment. During string manipulation this might be used.. (not very clear - free)

AX (Accumulator)
BX (Base registers)
CX (Counters)
DX (Data)
SP (Stack pointer)
BP (Base pointer)
SI (Source Index)
DI (Destination index)
IP (Instruction pointer)

Flags
OF - Overflow
DF - Direction flag (For string manipulation)
IF - Interrupt enable flag
TF - Single-Step flag
SF - Sign flag
ZF - Zero flag
AF - Auxiliary flag
PF - Parity flag
CF - Carry flag

8086/ 8088 - 16 bit + segments
80286 - Protected mode
80386 - Extends registers 32 bit protected mode - Hence 4GB segments possible.. Bulb.. 4GB process space in linux..
80486/Pentium - MMX
AX - 16 bits EAX - 32 bits

Career advice C or Java

2008-02-17T12:33:00.000-08:00

For the past three years, I have been a hard core C programmer, working on a variety of open source technologies like qmail, netfilter... Six months back I got an offer from a highly respected company. Good company, nice pay, I couldn't resist the offer. Now only problem, everything is Java.. at least in my locale. I am probably the only person trying find my through Java code using vim, ctags and cscope. I cant bring myself to open a eclipse. So I don't know how it works. Anyways, now I am ready to re look at my decision. Should I let my interest in system programming, C, linux kernel pass and start taking up work in Java.. or should I stick to my interests.

If I take up Java.. I am scared I will loose touch with system side.. like how do I filter packets of a particular kind.. How many connections are going to server...

If I don't take up Java.. I might earn the wrath of management. Probably cost me the job, I am now used to the comforts provided here..

I badly want to learn assembly language programming... Work on those for my bread.. Am I living in a well... Do people have such cool jobs.. Am I asking for an non feasible job position. People around me are into Java.. Is this a problem specific to India.. Do these jobs don't get outsourced to India..

This is probably one helluva situation.. and I am not sure which way to go..

Miscellaneous scripts

2007-12-15T23:37:00.000-08:00

Initrd image
1. To create initrd image on debian / ubuntu based systems
update-initramfs -k 2.6.23.9 -c -v

2. To unpack initrd image
cpio --extract < initrd file

Ubuntu Gutsy 7.10 from USB disk...

2007-12-12T07:31:00.000-08:00

This article is not intended to be a step by step installation process.

Why? I wanted to just carry around a usb harddisk around, plug on to any CPU(with BIOS support to boot from external USB disk) and boot from it.

Hardware. I got a 320 GB seagate harddisk (Requires external power supply) and Zippy cover that connects the hard disk as a USB device on to the machine. Costs totally around Rs.4300

Some installation considerations
1. Inspite of the fact that my processor is 64 bit, installed 32 bit, as 32 bit machines are ubiquitous.
2. I created a swap partition on the USB, as the host machine can be a W$ only machine. Probably not efficient, as access over USB is definitely slow.
3. I installed on my laptop and dint want the internal harddisk to be touched. I tried booting by disabling it in BIOS, but Ubuntu installer was still able to see it. Wonder how and why?
4. Two problems with MBR grub installation
a. Ubuntu by default installs the MBR on first disk
b. During installation, the USB disk appears second (hd1) and during boot (after installation) appears first (hd0)
How to overcome it -
a. The advanced option on the last screen of installer, allows changes to the partition where MBR is to be installed (change from hd0 to hd1)
b. Once the installation is complete, the /boot/grub/menulist needs to be updated back to hd0.

Well its worth all the pain.. I have a brand new kubuntu (with Compiz fusion) on it working.....
Kubuntu (Gutsy) + Compiz just rocks...
Going crazy over super + tab and cubes......