Command-line Bayesian Twitter reader with ad-blocking

UPDATE: The method below and the correspond script recently stopped working since Twitter changed their login method to "OAuth". The actual necessary changes will be a) switching to an updated Twitter client like TTYtter (which looks pretty awesome) and b) maybe revising the parsing of the Twitter client output. I've just started working on this and don't know when or if I will release updates. The script still works for Facebook updates, as is.

---

Following up on my initial post on command-line Bayesian filtering and the subsequent update, I have now generalized the script to incorporate not just Facebook status updates but Twitter posts as well.

I've been refining this script over the last year. In addition to providing a convenient way of incorporating Bayesian filtering in the tracking of posts from Twitter and Facebook, it now has the ability to review a particular Twitter user's posting history (via the "rewind" option) and to whitelist or blacklist arbitrary terms, which can be useful in permanently filtering out undesirable subjects or ads. And it tries to auto-expand those annoying shortened URLs and display the actual title of the web page being referenced.

This script requires that your system has the following programs: links (terminal-based web browser), curl, dbacl (Bayesian decision engine), twyt (command-line Twitter interface).

(I actually use another command line Twitter client (twixer) for following and unfollowing other people's accounts. And, while Twyt should work for posting on Twitter, I often use a third program for posting (twerp) which I started using before I found Twyt.)

Typical usage of the Bayesian twi script:

Teapot:~ surly$ twi
0: [5325528218] StephenAtHome: wearing a mask is 
so sweaty. i don’t know how those scooby-doo 
villians did it. (Sun Nov 01 00:40:05 2009 via web)
o)k, b)ad, u)rgent, open l)ink, c)opy, add to q)uotes, r)ewind, <CR>=next

The number "0" is just an index that counts down to zero to help identify the status messages you will be processing in this session. The number in brackets comes from the twyt program and is a unique Twitter message ID.

The options presented after the post are:
o)k: Add the post to the "ok" data file (for stuff you want to see more of in the future) and go to the next item.
b)ad: Add the post to the "bad" data file (stuff you don't want to see) and go to the next item.
u)rgent: Add the post to the "urgent" data file and go to the next item. This is a category I came up for for messages that say things like "Party at my place tonight" or "Free T-shirts for the first ten respondents". All I am really doing with them at the moment is highlighting them in a brighter color, but my idea is that in some future revision, these might be actively fetched and brought to my attention. At the moment, so few messages qualify as "urgent" that it's still a half-baked idea.
open l)ink: Opens the first extracted hyperlink in Safari.
c)opy: Copies the text of the post to the OS X system clipboard.
add to q)uotes: Appends the post to a text file with my collection of quotes. In the future, I might cause this to also mark the post as a "favorite" on Twitter, but this is not a priority for me.
r)ewind: The rewind feature was designed to compensate for situations where your Bayesian filter causes posts to filter below your radar and then suddenly another one from the same person bubbles up to the surface, but you don't understand it because you are missing the context. Rewinding causes the program to start going backwards through the locally cached history of all posts (in the "all.d" file) searching for posts from the user in question. It will keep feeding you previous posts as you keep hitting keys (or until you type q for "quit"). This feature only works on Twitter accounts at the moment, due to the way it is extracting the name to grep for.
<CR>=next: You've got to hit the carriage return to advance to the next post. If you type "o" and hit enter, the post will be categorized as "ok". If you just hit enter, your Bayesian filter will learn nothing from this post. Initially, it's good to give your filter as much definite feedback as you can, but once you are happy with its performance, you can just hit enter all the way through if you like.

Importantly, but perhaps not obviously, any set of these commands can be entered for a given post, and they will all be executed. Example:

o)k, b)ad, u)rgent, open l)ink, c)opy, add to q)uotes, r)ewind, <CR>=next

olq 

will categorize the item as "ok", open the associated hyperlink, and add the item to the quotes file. There is nothing stopping you from categorizing the item as ok, bad, and urgent, but this will almost certainly just confuse the Bayesian filter.

Finally you will be prompted to "Categorize which other posts as ok?"


When the program is finished processing all the new status messages, it will post an accuracy rating for the current session (basically, if you don't mark any of the "ok" messages as bad or correct any of the "bad" ones to be ok, it thinks it has 100% accuracy). You can use this as a rough metric for how good the current training level is.

Limitations: The current version of Twyt does not seem to be picking up the newfangled "Retweet" posts. Based on the messages that I am missing, this is something of a net advantage.

The twi script can be found here.

Automated self-monitoring

Looking for a way to self-monitor my working habits, I decided to write a script to analyze my notebook file and determine on which dates I had made an entry. Ideally, I ought to be working and making notes on my progress every day.

I decided to grep through the file for datestamps and form something like an ASCII sparkline, indicating whether I had made at least one entry on a given day.

The output for the last couple of months looks like this:

.|.|...|......|...|...||.|..||||....||........|..|..|||.

It looks even more badass displayed on my desktop via Geektool. I am also displaying the actual number of entries for each day immediately below the corresponding sparkline symbol. Admittedly, this contradicts the idea of displaying the information in tiny graphical form. I may eventually switch to something like this:

.|.-...:......-...-...--.:..-::-....::........-..-..---.

This is almost like Benjamin Franklin's system for reinforcing his 13 virtues. Admittedly, I am only reinforcing one virtue now, but I have a few other habits that I have been monitoring with my all-purpose tracking script, and these are ripe for display and correlation with other behaviors.

My 1mm-thick PDA backup system

You may have heard of the "Hipster PDA". It started as a joke that a bunch of notecards could be used like a PDA without all the electricity and technology. Then it caught on. There are elements of this that I like (particularly the creativity and do-it-yourself-ness), but it wasn't what I wanted.

The problem that motivated me: Every once in a while my PDA runs out of charge and/or loses all its data. As everything is backed up on my laptop, this is mainly just inconvenient. But when I suddenly need some vital bit of information that is on said PDA, it suddenly shifts to serious nuisance.

My first solution was to simply print out my calendar (which I keep in the plain text remind format) in 4-up form, fold it up, and keep it in my back pocket. And I found that I liked this because it is often easier to pull out and immediately access the information that I want. (It is particularly useful when going on a trip and having flights, times, hotels, and rental car places all listed on a little rectangle of paper.)

This eventually developed into a full script. This script currently collects the following things into a single text file. 0) The current date, to ensure freshness. 1) The output of my remind calendar, excluding boring things. 2) A little address book. I started by extracting specific contact information for specific people from the OS X Address Book via a little command line program called contacts. Then I wised up and started extracting everyone who has an asterisk after their name, allowing me to easily switch people in and out of my short list. (This is something I would like to automate someday.) 3) Some things that I have hard-coded into the program, including public transportation information, phone numbers for taxis, directions to places that I rarely go to, and business hours for a few places. 4) A tiny calendar. I initially thought of setting this up to grab the last, current, and next months from the UNIX cal program, but then I thought of the ingenious thumb calendar which allows a bunch of months to be compactly fit into a small area. And I realized that I could make my own. 5) And finally, my shopping list as extracted from my todo.txt to-do list.

[The only thing that I would like to get on there that I haven't figure out yet is a ruler, along the edge. This will likely require monkeying with PostScript or something.]

Here's the pared down script:

#!/usr/bin/perl -w

$fname="/Users/Surly/remind.txt";
open(INFO,$fname);
@lines = ;
close(INFO);
$outfile="hpda.txt";

open(OUTFO, ">$outfile");

$datestamp=`/bin/date "+%y%m%d"`;
print OUTFO $datestamp;


$i=0;
while($lines[$i])#assuming we initially have a numbered subtitle
{ $rem=(($lines[$i]=~/^rem/)|($lines[$i]=~/^REM/));
 $pass=((($lines[$i]!~/^[;#\n]/)&$rem)&($lines[$i]!~/growlnotify/));
 $pass=$pass & ($lines[$i]!~/boring/);
 
 if ($pass) 
 #ignore comments and growlnotifys and commands and only pass REMs
 {$lines[$i]=~s/^rem //;$lines[$i]=~s/^REM //;
 $lines[$i]=~s/ msg/:/;$lines[$i]=~s/ MSG/:/;
 print OUTFO $lines[$i];   
 }
 $i++;
}

print OUTFO "===contacts=========\n";
$a1=`contacts -H -f "%n %mp %wp %hp" Cleo`;
$a1=~s/-/./g; print OUTFO $a1;
$a1=`contacts -H -f "%n %wp %op" Popeye`;
print OUTFO $a1;
$a1=`contacts -H -f "%n %mp %e" Betty`;
print OUTFO $a1;
$a1=`contacts -H -f "%n %mp %wp %op %a" Snidely`;
$a1=~s/\n\n/\n/g; print OUTFO $a1;  
$a1=`contacts -H -f "%n %mp" "*"`;
$a1=~s/-/ /g;print OUTFO $a1;
$a1=`contacts -H -f "%n %mp %a" God`;
$a1=~s/\n\n/\n/g; print OUTFO $a1;

print OUTFO "===random stuff====\n";
print OUTFO "The Hamburglar's place: Practice, practice, practice, and take a left.\n";
print OUTFO "Library: open until 8pm.\n";
print OUTFO "===thumb calendar=====\n";#thumbcalendar.com
print OUTFO "            Su Mo Tu We Th Fr Sa JUL (31)\n";
print OUTFO "               Su Mo Tu We Th Fr Sa SEP (30)\n";
print OUTFO "                     01 02 03 04 05 06 07\n";
print OUTFO "01 02 03 04 05 06 07 08 09 10 11 12 13 14\n";
print OUTFO "08 09 10 11 12 13 14 15 16 17 18 19 20 21\n";
print OUTFO "15 16 17 18 19 20 21 22 23 24 25 26 27 28\n";
print OUTFO "22 23 24 25 26 27 28 29 30 31\n";
print OUTFO "29 30 31\n";
print OUTFO "                  Su Mo Tu We Th Fr Sa JUN (31)\n";
print OUTFO "   Su Mo Tu We Th Fr Sa AUG (31)\n";
print OUTFO "===\@shopping========\n";

$fname="/Users/Surly/todo.txt";
open(INFO,$fname);
@lines = ;
close(INFO);

$i=0;
while($lines[$i+1]){$i++;}

while($i>=0)
{ $pass=(($lines[$i]!~/^x /));
 $pass=$pass & (($lines[$i]!~/\@office/));
 $pass=$pass & ($lines[$i]=~/buy/);
 $pass=$pass & ($lines[$i]!~/\@online/); 
 
 if ($pass) 
 {$lines[$i]=~s/ \@shopping//;
 $lines[$i]=~s/^buy //;
 print OUTFO $lines[$i];   
 }
 $i--;
}

close(OUTFO);