Coding Life

My Weblog has moved...

2010-01-10T17:01:00.001-08:00

Back to Hosting it myself: http://trevelyn.com/

Please refer to that for your RSS of my fantasical posts!!!1 :D

Regular Expressions, expanding Taming the Electron Ep 1 blurb.

2010-01-06T08:27:00.000-08:00

What a good topic for a post eh? I spoke about Regular Expressions (RE) in my first episode of Taming The Electron and felt it was only fair to write a full intro for newbies to the subject. I have been reading "Mastering Regular Expressions" off and on for a good while now, but recently have been reading it with more focus on the material. This is probably due to my Kindle DX purchase, which you can read or watch about in a previous post Here. Regular Expressions are amazing. They are "magic." They can make matches with EXTREME precision.

Now, most people, including most of my friends, think that languages that do operations on text are lame. Well, OS's that are UNIX based, revolve around text. Operations on said files are what customize your OS. Customization is key to creating synergy between you and your OS. It makes you want to come back or entices you.

Most file in Unix based OS's that I deal with are in /usr/share or /etc or sometimes "live" files, like those in /proc. /proc has a lot of files that are dynamic and is called a "virtual file-system." They change constantly to with the changes of your overall system. If you wanted to write a cool application that checks your memory or cpu, you can use the files in /proc (/proc/meminfo, /proc/cpuinfo) to do that. You can even use them for error messages, bus messages, and more. In fact, many applications used in the administration of Unix based OS's, such as ps, dmesg, top, and more utilize "live" files in /proc. Here is a great article for anyone new to /proc: /proc

When you open a file and read it in an application, you will most likely want to search for useful information. Sometimes you are simply handed raw data that you need to "parse" (or change in some way) to turn the data into information. The best way to search is line by line. The best way to match is character by character (like grep, or egrep). egrep is a powerful matching tool that allows you to create extremely complicated yet powerful matching expressions.

The word expression, in Regular Expressions has the same meaning that it does in plain Algebra. Regular Expression syntax was actually developed back in the 1950's by Stephen Kleene as formal language theory, and automata theory (theoretical machines and problem solving). These are both basic sub-sets of computer science and theoretical computer science. the syntax can act like algebra in some ways, and even many languages that deal with "lame" text have their own regular expression syntax built right into them. RE should be a ANSI standard, but I hear a lot about how Perl 5 vs. Perl 6, and python, and Ruby all have slight differences in their Regular Expression syntax.

Okay, so that's what they are good for and a bit about their history. Now, let's try some matching patterns, meta characters, and meta-sequences. Think about characters and put them into classes. Class alpha will be your alphabet A-Z and lowercase a-z. You can specify a "range" in Regular Expressions with the square brackets "[]" say we have a text file that has a few lines with numbers and a few lines with digits, or phone numbers like so:

abcdefg
abc
12345
ghostbusters
007
31337
drums

Now, say we cat the file (show it's contents) and we only want to get (filter) the lines that have letters only. We can use egrep or grep like so:

cat filename.txt | egrep '[A-Za-z]'

And this will display the lines:

abcdefg
abc
ghostbusters
drums

Now, we can change the class to Numeric (numbers) and do [0-9].

cat filename.txt | egrep '[0-9]'

This will print the lines that have numbers in them only (we filter OUT the alpha class). This is a very basic example. We can further filter our output by using "anchors." Anchors will display what we are looking for ONLY if it is at the beginning of a line or the end of the line with the meta-characters (special word for operators (usually not of alpha or numeric classes)) "caret," or "^" and "Dollars" or "$" respectively.

so say we add:

cat filename.txt | grep '^[0-9]'

This will display all lines with numbers only that start with numbers. If we switch the "^" with a "$" and put it after the [0-9] range like '[0-9]$' This will match all lines that END with numbers and contain ONLY numbers. What if we put the meta-character "^" into the square brackets before our range? Well, usually meta-characters lose their meta when put into those brackets. for example the period "." which usually means "any one character" becomes a simple period. The "^" means that it negates the range. So '[^0-9]' means match any character that is NOT in our numeric class. Here's a cool tip: the range meta-character "-" only is a meta-character INSIDE of the square brackets. Yep, that means outside it's just a plain old "-" character.

There are parenthesis in the language, just like Algebra, that group together "expressions." For instance, here is an example from "Mastering Regular Expressions - O'Reilly" that searches for all instances of July 4th. The question mark meta-character searches for "one or more instance of the following character or group, so:

'(July?) (Four|4)(th)?'

Will find ALL values like:

July 4th
Jul Fourth
Jul 4
July 4
July Fourth

Their example in the book was:

'July? (Fourth|4(th)?)'

which seems to have over-looked the "th" string at the end of "fourth." Yeah, you get picky like that, and once you get into the swing of computer languages and Regular Expressions, you start to look for the most efficient way to code.

These small examples can help you get into the flow I'm sure, or at least pique your interest in Regular Expressions. Just like any language, Regular Expressions will open up a lot of doors for you as a developer or system administrator. They too can help you with problem solving. Sometimes you can perform large, usually complicated tasks with them and sometimes small tasks would be large tasks without them. Mastering Regular Expressions is a good book. It also covers a few Awk, Sed, and Grep topics as well and brings everything together in one cool place; a bunch of papers wrapped in heavier paper. :)

I, without a doubt, realize that my code isn't always efficient and that sometimes I do things in non-efficient ways. Perl's motto is TIMTOWTDI "There's more than one way to do it." The Perl community accepts newbie code and more importantly, the interpreter Perl accepts newbie code. It's a good language to start programming with and I am still a beginner programmer. If you find anything wrong with what I have stated above, simply let me know and I will fix it. I am forever a student of Awk, Sed, Grep, Vi and Regular Expressions.

Bashing DOS

2010-01-01T16:46:00.000-08:00

I hate DOS. I don't like how I have to switch back and forth between things from Linux's Bash to Windows' DOS. Usually what I end up doing is this:

1. Create a file in C:\WINIDOWS called "ls.bat" in that file put one word "DIR"
   now, when you accidentally type "ls" instead of "dir" you get the same ouput :)
2. Download GNU nano. Yeah cos that "thing" you get when you type "edit"
   is just ugly, and awful. Get the zip file and put all DLL's and EXE's into the same    directory you put the ls.bat: C:\WINDOWS (this works even with Windows 7)
3. Install Strawberry Perl and Ruby. Yeah, the windows versions are weird, but they do    exist!
4. Get the Win32 version of WGET, put the EXE file into C:\WINDOWS
5. Get the Win32 Versoin of NMAP, put the files form the zip into the C:\WINDOWS directory.

As you may have figured out, C:\WINDOWS is in your environment's $PATH. So any command typed, sends that to DOS which searches your $PATH then executes any file or executable with the commands name. (quite easy eh?)

You can now make any ".bat" file you want and put whatever you want into it and then stuff it into your C:\WINDOWS directory! You can do things that even take arguments, so say you wanted something to repeat your arguments, well, they are like this: "%1, %2, ... %n"

echo "You gave me the argument %1!"

Now, don't go and get lazy or forget the real commands [application names e.g. "DIR", "EDIT", etc] for DOS in case you are sitting at a Windows machine that has a non-Bashed DOS environment! ;)

Resources:
Win32 NMAP - http://users.ugent.be/~bpuype/wget/
Win32 WGET - http://nmap.org/book/inst-windows.html
GNU Win32 Nano - http://www.nano-editor.org/
StrawBerry Perl - http://strawberryperl.com/
Ruby for Win32 - http://www.ruby-lang.org/en/downloads/

Taming The Electron

2009-12-31T11:57:00.000-08:00

I will be hosting a show on CactiRadio at 7PM EST each Sunday night for WeakNet Labs called "Taming the Electron." It will primarily be about programming, hacking, and phreaking. I will try to get a few interviews in, and play a lot of electronic music. If you have any suggestions about future shows, please feel free to Email me - Douglas [at] WeakNetLabs [dot] com

If you'd like to listen in, here is the URL for CactiRadio

It's a great station that plays The Phone Show, PLA Radio episodes, and often plays my music! I will host the episodes as well, at http://weaknetlabs.com/tamingtheelectron

This weeks Episode will be themed "Inspiring people to start programming"

Perl Nano (pnano)

2009-12-31T11:43:00.000-08:00

I was messing around with some lurking [extremely quiet] geniuses on IRC and said "How can I assign a whole bunch of keystrokes to one keystroke? So maybe I could hit Windows [super key] and it would print out "#!/usr/bin/perl -w" for me, so I don;t have to type it each time."
Well, I got no answer, as usual. So then I just thought up my own way, I could edit the way I call nano.
In my real life example, I used an "alias" in my ~/.bashrc file since I didn't have root access to the box I was using. But I will show you two ways here:

1. create a file "pnano" and in it type:

#!/bin/bash
echo "#!/usr/bin/perl -w" > $1
nano $1
CT=`cat $1 | wc -l`
if [ $CT == 1 ]; then
rm $1
fi

2. chmod it so it's executable:

chmod +x pnano

3. put it into your $PATH, I usually put things into bin:

sudo cp pnano /usr/bin/

This will create a new "executable" program. Let's look at the code in the script above, line by line. This tyoe of scripting is called "shell scripting" Line 1. shows the shell what to use, as the interpreter, to "run," or "execute" the application. In our case this will be bash. Line 2. put the line "#!/usr/bin/perl -w" into the file that you gave as an argument "$1". This means that if you do:

pnano lolhi.pl

This will put the shebang line right into the file "lolhi.pl" bringing us to our next line. Line 3. runs nano and opens the file [from your argument] $1, or as in our example "lolhi.pl"
Once opened, you will see the shebang line in the file, and you can begin coding.

You may want to add a few arguments to nano that are extremely useful:

-w means "no wrap" meaning when you hit the end of your terminal screen with a long line, it won't automatically hit RETURN for you giving you a new line. I think that should be default and has made me hate nano in the past. Next you should add to line 3:

nano +2 $1

The "+2" will put you at line two so you don't have to hit down before coding. Line 4. puts the value evaluated by "cat FILENAME | wc -l" which means "how many lines are in the file you just opened with pnano" into the variable $CT. We do this so that if you close nano without typing any code, you aren't left with a file named $1 [whatever you put after pnano] with one line it it: "#!/usr/bin/perl -w"

so there's an if statement: if it's "empty" [only shebang line exists] rm it. Then "fi" ends the if block. That's a fun exercise.

If you do not have root access to the box you code in you can make an alias statement in your ~/.bashrc file like so:

alias pnano='/home/USERNAME/pnano'

And put that anywhere in your ~/.bashrc file.
Another cool tip for those who use nano to write their Perl applications is the /etc/nanorc file. This was recently pointed out to me by Tully. What it does is automatically color your code so you can read it easier. To use it, simply uncomment the line that looks like so:

# include "/usr/share/nano/perl.nanorc"

Just take away the "#" and save it. next time you open a Perl .pl file, it will be in color. <3EDIT:I have already clobbered a file with the ">" in the echo system call :-( So what I did was use the

if [ -f $1 ]; then

syntax to check to see if the file already exists or not before clobbering it! check it out:

#!/bin/bash
FILE=$1
if [ -f $FILE ]; then
        nano $FILE
else
        echo "#!/usr/bin/perl -w" > $1
        nano +2 $1
        CT=`cat $1 | wc -l`
        if [ $CT == 1 ]; then
                rm $1
        fi
fi

Enjoy ;)

Perl for Multiple Homicide Comparisons

2009-12-26T10:34:00.000-08:00

I was watching Dexter last night. It kind of scares me how I feel so connected to his thought process. Maybe I should see a shrink? Speaking of shrinks, he [Dexter] was comparing profiles of suicide victims in his area. After looking through them multiple times, a light bulb went off in his head.

He realized that they [suicide victims] had the same psychiatrist! Well, they were computer print outs, and "Word" documents I'm sure, so why not leave them digital and search through them like "grep?"

I thought up an extremely easy application as I was watching the show (ADD, I know):

#!/usr/bin/perl -w
my @prof0 = ("bobby", "luke", "test", "wheel", "root", "hacker");
my @prof1 = ("computer", "luke", "bobby", "root", "hacker");
my @dict = ("computer", "test", "wheel", "root", "hacker");
foreach my $var (@prof0) {
        if (grep(/$var/, @prof1)) {
                if (!grep(/$var/, @dict)) {
                        print $var . " is in both!\n";
                }
        }
}

This will compare 2 profiles (The real implementation will need to slurp the lines of each profile FILE into the corresponding arrays.)

Then (once again slurping a dictionary file into the @dict array in the real life implementation) checks to see if the word is a plain word or proper noun. Proper noun was the whole 'EUREAKA' cusp for me. "Cusp" being our mathematical peak(/pique? - curious distinction in our context eh?) for finding the last clue to solving the problem in brain activity :P

So everything from the cusp to the code above was downhill as I adjusted my thoughts into (simple) boolean logic. I am sure scientists have such applications in forensics labs, or police stations, but it was a nice exercise and taught me that I think outside of what is really being taken in by my eyes.

~Douglas.

Bash Phasing out the good ol' CTRL+r History Search?

2009-12-22T12:16:00.000-08:00

Well, CTRL+r will still be good for things used pretty far in your past, but now Bash seems to have implemented something new. When playing around with Debian I noticed that history accessed by the up and down keys was de-duplicated!

Say for instance I am writing a new PHP application and each time I want to test it, I FTP it to a server. If say, as I usually do, I use nano or vi to edit the application, then run the application locally with php5-cli about 10 to 15 times before deciding to upload it. That means with old Bash I would have had to press up to find my old FTP command 25 to 30 times! 2 for each run-and-debug sequence between.

That's where CTRL+r became a life saver. especially when working with loooooooong SQL passes to Bash. In SQL I could include my username and password so I don't have to pass it each time with "--password= --user="

Well, now with History de-duplication, that's all in the past! This will compact ALL similar commands send to Bash in the past, and make it so the most I would have to press the up key to find the old FTP command is like 3 or 4 times!

I love Bash, I love Linux, I love programming!

~Douglas.

What nano did to me.

2009-12-22T12:07:00.000-08:00

Nano
Yeah, It's not just nano. After weeks of using vi I do the same thing: try to close a windows or web based editor with CTRL+X,Y,ENTER or ESC,:,w,q,ENTER For instance Notepad, Dreamweaver, Wordpad, Openoffice.org, even Blogspot.. When I think the post looks fine, i accidentally hit CTRL+X and look up to realize I am wired.

I once noticed that nano (GNU at least), adds newlines to the end of your document, where vi doesn't. Chartreuse from WeakNet said the same thing before too, so I know I am not crazy. Sometimes I use vi, sometimes I use nano. I like them both. I have heard people whine about vi being hard to use, I don't understand their argument. It's simply knowing what keyboard strokes to use, as opposed to seeing little icons to click on that eat up your system's resources.

Another thing I do, is CTRL+W to search in a browser in stead of CTRL+F, which closes the tab or window. I feel like a train when i am coding and getting out of the state of mind takes longer to slow down and stop.

Re-cap on last post
Another thing I'd like to add to my last post about Microsoft being my nemesis:

Windows Mobile... What a joke that OS is. My phone boots Windows each time it turns on, sometimes taking like 4 minutes before I can make a call. God-forbid I have an emergency and need to dial 911, but I need to boot Windows first! The boot time is not the only issue. Check this out: Windows Mobile absolutely SUCKS at memory re-alloc.

Yeah I said it, if I start Task-Manager and see 20MB of RAM open, then reun 3 or 4 applications and watch the RAM go in TaskManager. Then stop those applications, and see only about %40 of the RAM freed??? How? What? When?

I need to make a new years resolution to stop talking about Windows and that stupid company.

~Douglas

Dear Microsoft

2009-12-19T11:58:00.000-08:00

I Hate you and you are my nemesis. I used to be one of those guys who would slander you and make fun of the people who used you, jokingly, simply because I used Linux. It became a more serious hatred when I was forced against my will to understand you and your code writing "ability." Working in IT, I have to remove malware, virii, fix your OS from your poorly written updates, etc ad absurdum. While dealing with these "issues" I found who absurdly fragile your OS is. It's party due to user end misconfiguration, but mostly it's your fault.

Explorer.exe is, in all honesty, the worst File Manager I have ever used. Permissions are a joke. Doing data backups daily, I find that using a Linux live disk is much faster, more efficient, and simply gets the job done. Hell, even OSX's file manager Finder is a little bit better than Explorer.exe Doing things form the command line has become my life. It's how to do things with a computer. If you would have realized that people like I existed, you may have made your CMD.exe able to do a higher resolution (maximize much?) Developing, editing, doing ANYTHING that requires CMD.exe is a joke. In fact, I usually move the shortcut link from system tools to the Accessories Games menu.

Okay, besides most of your native executables locking up most of the time, there's more. Recently I was given a project at work that relied on your Event Logger. Wow, what a joke. Your event logger for Windows XP, sucks. You simply cannot edit your code to make login/logout events on the same line, let alone in the same order? You can't NOT fill the "Security" table with garbage when a user only logs Logon/Logoff? I don't get it. I hate you. Your [right click on my computer->Manage->Events->Security] log is fake. Your raw log data is in 24hr time, and your data in said window is parsed to 12 hour time and seconds are truncated. What? They don;t mean anything to you? NOT ONLY THAT, when I bring the raw data (viewed as 24 hour time) into a fresh installation of Microsoft Access, YOU PARSE IT AGAIN VIA SOME MACRO THAT IS ENABLED BY DEFAULT?

Why are there sometimes 10 events labeled "551" (logout) between "528" (login) and "551?"

I was truly baffled by this, I felt as if I were overthrown by Microsoft by this. I hated so much that I was spewing out curse words in front of my coworkers and family - I'm sorry, if you look at it on my side, this was too - non-trivial, to just calm down and re-sync into the corporate brainwash.

I don't care, except that Windows XP WON'T GO AWAY. Here I have constructed a curve at which functionality/productivity would change over time as the world were forced to stop using Microsoft Windows, let's say it just disappeared one day..

At point "A" our low productivity drops to zero for all users of Windows. (please take in the fact that even medical facilities, science departments, government and federal, all unfortunately still use windows XP (some do)). After the drop to zero, a few years go by, where highly efficient Linux developers are in the spotlight blazing away at their keyboards and mice, re-writing all software used in said fields. This is a big economic crash. Things get chaotic, i'll be honest. But look what happens to the curve after a few years, development/productivity goes up exponentially until finally, after a few years - it sky-rockets.

Children of your children of your children will be smarter than ever. Linux/Unix jobs are everywhere. Our world, has completely changed. People start to exercise more, take care of themselves, we avoid the inevitable plague that is hinted to us in the movie "Idiocracy." We become a world united once again by more than just the internet. By teaching, learning, respecting, hacking, etc.

Computers, once more, can start relaxing system resources as we move towards the Linux based cloud. We can rely simply on smart cards, or fingerprints to log us into our cloud via a Sun thin client.

An almost perfect world (technologically speaking).

~Douglas.

The Holidays.

2009-12-18T10:55:00.000-08:00

Christmas time is here! There's a winter advisory for places around me. I'm sure it will just turn to rain by the time it gets to Pittsburgh though. Our climate is assinine.

My Forums were hacked!! YAY! WeakNetLabs.com/forums is now gone. Whatever.

Tully posted something about someone that was hilarious on FaceBook. I'd tell you but, I have something better. I took a pic of it when we were using TeamViewer!

I got an arduino for christmas, so I can be a hardware hacker soon! I have a ton of neat ideas about what to make and I am excited to work with the code that goes into the chip itself.

Happy Holidays..

~Douglas.

Fibonacci and Tullywacker.

2009-12-15T10:40:00.000-08:00

Tully has a new blog, http://buffer0verflow.com/ and he posted about a C program that produces the nth Fibonacci number. This made me dig up a piece of code I wrote in PHP a long time ago from the old.html page in http://weaknetlabs.com/BACKUP-NEW/old.html

This is one piece of code I am really proud of. It will only list Fibonacci numbers up to whatever you desire, but it is elegant:

$m = 1; $n = 1; $i = 1; $h = 0;<br />echo "$m"; echo "$n";<br />while ($h <= 10) { $i = ($m + $n); echo "$i"; $m = ($n + $i); echo "$m"; $n = ($i + $m); echo "$n"; $h++; }echo "...";

What I think is so elegant about the code is if you look closely you can see that no variable is repeated until all have cycled: "m", "n", and "i." :)

You may want to add "<br />" to the echo's (if they don't show up as I use the xmp tags{which they did}), so they break a new line separating the integers. Here is the code in action http://weaknetlabs.com/temp/fibonacci.php

Tully's is faster of course since it is compiled.
This sequence is closely related to the Golden ratio and you can learn more about it here:
http://en.wikipedia.org/wiki/Fibonacci_number

Kindle Review

2009-12-14T14:56:00.000-08:00

I recently reviewed the differences between the two new Kindles.

The Kindle DX is well worth the extra money if you think you will use the Kindle as often as I do. I found that it fits nice and snug in a 10" netbook carrying case.

Long Weekend

2009-12-14T08:33:00.000-08:00

So, I started playing the new Zelda, "Spirit Tracks" yesterday morning. It's a good game. I put maybe a few hours into it, and so far I like it much more that the "Phantom Hourglass." I really want to get my hands on the new DS XL. There's an article I just read on IGN about Spirit Tracks on the DS LL and the author said it was fantastic.

I got around to releasing my first Ruby application; a D&D dice roller. It allows the user to give the type (# of sides) of dice, plus multipliers and modifiers. Getting the negatives to work right was weird. I feel like I am getting the hang of Ruby the more I type. I took a different approach to learning the new language. I read through a book and a half of it before even writing any code. Well, it worked! I was able to remember a LOT of syntax and rules and such.

CodeRepo Link to D&D Dice Roller

I had a few guests over this weekend and I got to see Terminator Salvation! That movie was amazing, but I am a die-hard Terminator fan, so don't listen to me.

I was going to play D&D tonight, but my friend got off work like 1hr too late. Seems to take about 3 hrs per session and I need to get up early as it is, a school night.