New Blog

I'm quite lazy... and to keep this blog clean, I decided to create a now one solely dedicated to security and programming. Please check it out: dakhma.net

Oh by the way, a dakhma is construction for the purification of dead bodies in Zarathustrian religion, i.e. the bodies get cleaned by birds.


Last notice: I hope to post more language/linguistic related news soon. Though I'm currently working on quite a lot of projects.

Zipf's law

Last friday I gave presentation on Zipf's law primarily concerned with processing frequency lists and spectra in R and zipfR. The scripts contain two Python programs for extracting frequencies out of NLTK's internal Gutenberg selection corpus and the section J of the ACL Anthology corpus. If you don't have access to the ACL, I provide the processed TFL and SPC files for both corpora in the ZIP file.

Download:
[Slides]
[Scripts]

Some NLP-related Python code

1. A program that counts the Flesch Score of a text. Code here. Don't know if the syllables are computed correctly.

2. A program that searches a wordlist for minimal pairs. Code here. Example here. The format of the wordlist is restrictive and the minimal pairs are printed twice!

3. A program that obfuscates the input, which means that first and last letter are the same but everything in between is mixed around. Code here.

4. A program that constructs a tree from a file and searches for the common minimal ancestor of two nodes. Code here. Example here.

Python-based RPN Evaluator

This program evaluates logic expressions out of a textfile with Reveresed Polish Notation (RPN) syntax.

Example world file:
wind
/sun
/rain
red

wind and red have the value of 1, sun and rain 0 since they are prefixed by "/".
Here's the syntax to run the program: "python log.py myworld.world".
It quits when an empty expression occurs.

Example usage:
C:\Python26>python log.py myworld.world
Logical Expression: rain sun &
0
Logical Expression: sun red |
1
Logical Expression: sun wind ^
True
Logical Expression: winter sun &
*** Error while evaluating: Bad name: 'winter'.
Logical Expression: sun red
*** Error while evaluating: Unbalanced expression: 'sun red'.
Logical Expression: sun red red |
*** Error while evaluating: Unbalanced expression: 'sun red red |'.
Logical Expression:

C:\Python26>



Find the source code here.

Download all SMBC Comics

Simple regex-based bruteforce program to save all comics from http://www.smbc-comics.com/. You'll need http://commons.apache.org/io/ and my source code.

These little annoying and surprising things...

...concerning Python as language of "very clear syntax which emphasizes code readability" (Wikipedia):

A = Annoying == Anti-Python

1. The naming conventions of...:

a) ...package-files/magic members: __init__.py, def __del__, __name__
b) ...visibility modifiers: _protected and __private

So _ and __ in general. Really, why did Guido do this? Is there an explanation? Perhaps it's inherited from another language?

2. The verbose...:

a) ...object inheritance declaration of each class: class Standard(object)
b) ..."self"-reference of each class-value/-constructor/-function just to indicate that it's non-static: def compute(self, number), self.Radius, def __init__(self)


(3. Multiple inheritance: It's no coincidence that most languages don't support multiple inheritance. Normally, you don't need it and it is a trap which makes debugging almost impossible. It is definitely not a feature for a language which emphasizes code readability and clear syntax.)




S = Surprising

(Powerful ability to handle and process strings in general.)

1. Lambda/Annonymous functions: (lambda x, y : x + y)
2. Managed Attributes: property([fget[, fset[, fdel[, doc]]]])
3. Great modification abilities due to magic members/methods and type-emulation.
4. List comprehension, generator expressions and yield.
5. Function decorations. Java has this one too.
6. Localization module. It's neat and easy to localize your programs.
7. Parallel computing module!
8. Awesome network protocol capabilities.
9. Unit tests.
10. The best documantation I've ever seen.

Blagh

1. Blagh:

• I read the Post "Crying" again and must say that Java is definitely my favourite programming language.
• I totally lost every knowledge of Python which's a bad thing since I'll have to use it in a exam in a few months.
• I got attached to Twitter. It's so convenient and you can watch it on the panel of my blog.
• I read Russell's Introduction to Mathematical Philosophy. Very interesting.
  

2. Pathfinder:

I had to write a small Java program that implements Dijkstra's shortest path algorithm in order to find the shortest path between two cities. You can download the source code here. You can add nodes and edges to the data.txt and the user interface is console-based.


3. Imageboardsave:

Currently, I'm writing a program which downloads every image and Rapidshare URL in a thread of an imageboard. The program works for AnonIB at the moment and the user can specify how many pages he want to search. The program scans the page for threads and download the content to:

/threadID/picturefilename.something


and the Rapidshare-URLs to:

/threadID/rapidshare.txt


Subthreads, i.e. more than a one-page thread, are considered as well. It even has a graphical user interface programmed in Java Swing. Actually, it runs pretty good right now, but I don't want to release it yet.

Language Guesser and OpenNLP Pipe

1. The first program I wrote estimates the language of a document, based on a simple statistical bigram model. It takes 5 arguments from commandline based on following syntax:

java Main trainfile_lang_1 trainfile_lang_2 trainfile_lang_3 trainfile_lang_4 testfile_lang

e.g.

java Main ./europarl/train/de/ep-01-de.train ./europarl/train/en/ep-01-en.train ./europarl/train/es/ep-01-es.train ./europarl/train/fr/ep-01-fr.train ./europarl/test/de/ep-01-de.test

and writes to the system standard output: "Die Sprache ist wahrscheinlich Deutsch." (The language is probably German.)

You see, this is very static and perhaps I'll make it more dynamic in the future. It was originally created to estimate the language of a protocol from the European parliament. You can download it here, but please be aware that the comments are in German.


2. The second program is more interesting. It takes a XML file with tags and everything and writes the sentences to a file in following format:

token TAB tag TAB chunk

e.g. for the sentence "Is this love?"

Is VBZ O
this DT B-NP
love NN O
? . O

It just takes the path to the XML file as commandline argument. To run the program you'll need 2 things. Ant and the OpenNLP models. You can download the program here

/*
* To ensure the functionality of the program, the models EnglishChunk.bin.gz,
* EnglishSD.bin.gz, EnglishTok.bin.gz, tag.bin.gz and tagdict.htm have to be
* in the models directory.
*
* IMPORTANT:
* The models have to be downloaded and placed in the models directory
* otherwise the program won't work. The download links can be found in
* ./model/readme.txt.
*
* Install:
*
* 1. Download the models at OpenNLP: http://opennlp.sourceforge.net/
* 2. Run Ant
* 3. Start the program with java -jar NLPipe.jar XMLfile
* 4. Optionally you can run the program with: ant -Dargs="XMLfile" run
* 5. Optionally you can archive the basedir with: ant tar
*/

Crying

Vacation! Last semester was very intense, no time for anything. Gosh, and now I'm trying to learn Java... It's far more complicated than imagined. Python was neat and easy - in comparison - but Java is far more potent, so they say. I don't quite understand why I've to write code which seems to be obsolete - at least for a beginner like me , e.g.

public class HelloWorld
{
 public static void main(String argv[])
 {
  System.out.println("Hello World!");
 }
}

is equivalent to

print "Hello World!"

in Python.

And next semester I'm going to die for sure... Here're my courses:

Formal Syntax: I'm more the Morphology, Phonology, Semantics type. Actually Syntax is the only thing which gives me problems.
Java: Well, it's going to be hard I'd say.
Logic: I like formal logic, really.
Artificial Intelligence: Very very interesting. Inference, neuronal networks, genetic algorithms and that stuff - you know...
Acoustic Phonetics: This sounds good: Reading spectrograms and get in touch with VoiceXML.

Python Madness

My Night at 3 AM: Hacking Python code to learn the language. That's a life! Here's what I've done so far:

1. Decimal to dual conversion:

import sys

def bd(x):
 n = []
 if x < 0: 
  return "Positive integer required"
 elif x == 0:
  return [0]
 else:
  while x > 0:
   n.insert(0,x%2)
   x = x/2
   bd(x)
 return n

if __name__ == "__main__":
 try:
  number = int(raw_input("Number: "))
  print bd(number)
 except ValueError:
  sys.stderr.write("Integer required\n")

2. Basic truthtables:

def logicalAnd():
 for valueOne in range(2):
  for valueTwo in range(2):
   print "%d %d %d"%(valueOne, valueTwo, valueOne and valueTwo)

def logicalOr():
 for valueOne in range(2):
  for valueTwo in range(2):
   print "%d %d %d"%(valueOne, valueTwo, valueOne or valueTwo)

def logicalConditional():
 for valueOne in range(2):
  for valueTwo in range(2):
   print "%d %d %d"%(valueOne, valueTwo, not valueOne or valueTwo)

def logicalBiconditional():
 for valueOne in range(2):
  for valueTwo in range(2):
   print "%d %d %d"%(valueOne, valueTwo, valueOne is valueTwo)

if __name__ == "__main__":
 op = raw_input("Connective: ")
 if op == "and":
  logicalAnd()
 elif op == "or":
  logicalOr()
 elif op == "conditional":
  logicalConditional()
 elif op == "biconditional":
  logicalBiconditional()
 else:
  print "Connective not known"

3. ASCII table. First column is the ASCII value, second column is the local interpretation, third column is the raw UTF-8 interpretation, fourth column is the hexadecimal value:

for element in xrange(256):
 print "%s \t %s \t %s \t %s"%(element,%%
 chr(element), str(tuple(chr(element)))%%
 .strip("()'',"), chr(element).encode("hex"))

4. Perhaps a complicated dual to decimal program:

def reverseRange(input):
 n = []
 for i in range(len(input)-1,-1,-1):
  n.append(i)
 return n

def singleValues(input):
 m = []
 for  i in input:
  m.append(i)
 return m

if __name__ == "__main__":
 input = raw_input("Number: ")
 rR = reverseRange(input)
 sV = singleValues(input)
 dN = 0
 for i in range(len(sV)):
  dN += int(sV[i])*2**int(rR[i])
 print dN

This code works in 2.6.1