Improving Twitter signal-to-noise ratio with TTYtter extensions

I've been using TTYtter as my primary interface to Twitter for almost a year, and it is still great. In fact, the new releases have made it even better. And the depth of configurability is amazing.

One Twitter user that I follow posts some very cool stuff, but also lots of side-chatter that doesn't interest me. Twitter lets users turn off retweets on a per-user basis, but not the messages to other users (indicated by the presence of the "@" symbol). TTYtter's extensibility makes this easily remedied with a short bit of Perl code that is run during the processing of each Twitter status.
$handle = sub {
my $ref = shift;
my $text = &descape($ref->{'text'});
my $name = &descape($ref->{'user'}->{'name'});
return 0 if (($text =~ /@/i) && ($name =~ /Mr. Noisy/i));
&defaulthandle($ref); #actually print the Twitter message
return 1;
};
The first two bolded lines extract the status text and the name of the person who posted it. The third line checks whether the text is from Mr. Noisy and whether it contains an "@". If so, the function returns a 0 before the code that prints that message to the screen is executed.

Then when I run TTYtter, I include the "-exts=..." flag among the command line arguments:
ttytter -exts="/Users/surly/lib/denoise.pl"
And all of Mr. Noisy's Twitter postings that contain the "@" symbol are filtered out.

Another option is to only include postings from a given user when they receive a certain number of retweets. Here, the bolded lines also show how to use the user's actual Twitter handle (because it's the more common and convenient name to use):

$handle = sub {
my $ref = shift;
my $text = &descape($ref->{'text'});
my $name = &descape($ref->{'user'}->{'name'});
return 0 if (($text =~ /@/i) && ($name =~ /Mr. Noisy/i));
my $rtcount = &descape($ref->{'retweet_count'});
my $sname = &descape($ref->{'user'}->{'screen_name'});
return 0 if (($rtcount < 10) && ($sname =~ /verboseguy/i));
&defaulthandle($ref); #actually print the Twitter message
return 1;
};


I worked out most of this from the TTYtter page on extensions and advanced topics. A partial list of fields that you can extract and filter on are listed in the Twitter documentation on the JSON data representation of a single status object. There is a lot of metadata that could be conceivably used for filtering.

Simpler filtering on the text of Twitter statuses can be carried out using the command-line -filter argument, but these extensions allow far more sophisticated ways of controlling the flow of Twitter information to your screen.

---

Other extensions that I like:

• deshortify.pl - which deobfuscates all of those "t.co" hyperlinks.

TTYtter: a turbo-charged console Twitter client

I finally got around to updating to a real Twitter client again. Twitter now requires that applications use OAuth authentication instead of a standard username/password login, rendering my old command-line Twitter programs obsolete.

But then I found the awesome Perl-based TTYtter. First, unlike a lot of Perl-based applications, you don't have to recursively install lots of other modules to get TTYtter to work. I just installed the single file Perl script, and it worked. The OAuth set-up is clearly described and easy. The benefits of this program are huge! It supports the new-fangled retweets, favoriting, direct-messaging, tracking arbitrary keywords, and searching (I visit search.twitter.com a lot less often). It can retrieve the entire thread for any given Twitter post. If your version of Lynx or cURL supports SSL, TTYtter can use SSL encryption on your Twitter traffic.

There is one Perl module that I would recommend you install though... Term::ReadLine::TTYtter fixes the readline behavior AND gives a cool dynamically-updating character count for what you have typed at the prompt

TTYtter supports Growl notifications, so you can set it up to inform you when you get a direct message or just when there is a new update of any kind. It is highly configurable.

Mac users: Make a file called ".ttytterrc" in your home directory containing just this one line:

urlopen=open -g %U

and then when you use the /url command, TTYtter will open the corresponding URL in your default browser. The "-g" switch opens the page in the background, keeping your Terminal window in the foreground.

One downside to the high degree of configurability is that there is a lot to learn to fully exploit the powers of TTYtter. (I run the program with the following set of command-line switches: ttytter -ssl -readline -notifytype=growl -notifies=dm,reply -notifyquiet -timestamp -filter="/spn\.tw/||/\#FF/".) But the documentation is well-written, and what you get is worth the learning curve.

Command-line Bayesian Twitter reader with ad-blocking

UPDATE: The method below and the correspond script recently stopped working since Twitter changed their login method to "OAuth". The actual necessary changes will be a) switching to an updated Twitter client like TTYtter (which looks pretty awesome) and b) maybe revising the parsing of the Twitter client output. I've just started working on this and don't know when or if I will release updates. The script still works for Facebook updates, as is.

---

Following up on my initial post on command-line Bayesian filtering and the subsequent update, I have now generalized the script to incorporate not just Facebook status updates but Twitter posts as well.

I've been refining this script over the last year. In addition to providing a convenient way of incorporating Bayesian filtering in the tracking of posts from Twitter and Facebook, it now has the ability to review a particular Twitter user's posting history (via the "rewind" option) and to whitelist or blacklist arbitrary terms, which can be useful in permanently filtering out undesirable subjects or ads. And it tries to auto-expand those annoying shortened URLs and display the actual title of the web page being referenced.

This script requires that your system has the following programs: links (terminal-based web browser), curl, dbacl (Bayesian decision engine), twyt (command-line Twitter interface).

(I actually use another command line Twitter client (twixer) for following and unfollowing other people's accounts. And, while Twyt should work for posting on Twitter, I often use a third program for posting (twerp) which I started using before I found Twyt.)

Typical usage of the Bayesian twi script:

Teapot:~ surly$ twi
0: [5325528218] StephenAtHome: wearing a mask is 
so sweaty. i don’t know how those scooby-doo 
villians did it. (Sun Nov 01 00:40:05 2009 via web)
o)k, b)ad, u)rgent, open l)ink, c)opy, add to q)uotes, r)ewind, <CR>=next

The number "0" is just an index that counts down to zero to help identify the status messages you will be processing in this session. The number in brackets comes from the twyt program and is a unique Twitter message ID.

The options presented after the post are:
o)k: Add the post to the "ok" data file (for stuff you want to see more of in the future) and go to the next item.
b)ad: Add the post to the "bad" data file (stuff you don't want to see) and go to the next item.
u)rgent: Add the post to the "urgent" data file and go to the next item. This is a category I came up for for messages that say things like "Party at my place tonight" or "Free T-shirts for the first ten respondents". All I am really doing with them at the moment is highlighting them in a brighter color, but my idea is that in some future revision, these might be actively fetched and brought to my attention. At the moment, so few messages qualify as "urgent" that it's still a half-baked idea.
open l)ink: Opens the first extracted hyperlink in Safari.
c)opy: Copies the text of the post to the OS X system clipboard.
add to q)uotes: Appends the post to a text file with my collection of quotes. In the future, I might cause this to also mark the post as a "favorite" on Twitter, but this is not a priority for me.
r)ewind: The rewind feature was designed to compensate for situations where your Bayesian filter causes posts to filter below your radar and then suddenly another one from the same person bubbles up to the surface, but you don't understand it because you are missing the context. Rewinding causes the program to start going backwards through the locally cached history of all posts (in the "all.d" file) searching for posts from the user in question. It will keep feeding you previous posts as you keep hitting keys (or until you type q for "quit"). This feature only works on Twitter accounts at the moment, due to the way it is extracting the name to grep for.
<CR>=next: You've got to hit the carriage return to advance to the next post. If you type "o" and hit enter, the post will be categorized as "ok". If you just hit enter, your Bayesian filter will learn nothing from this post. Initially, it's good to give your filter as much definite feedback as you can, but once you are happy with its performance, you can just hit enter all the way through if you like.

Importantly, but perhaps not obviously, any set of these commands can be entered for a given post, and they will all be executed. Example:

o)k, b)ad, u)rgent, open l)ink, c)opy, add to q)uotes, r)ewind, <CR>=next

olq 

will categorize the item as "ok", open the associated hyperlink, and add the item to the quotes file. There is nothing stopping you from categorizing the item as ok, bad, and urgent, but this will almost certainly just confuse the Bayesian filter.

Finally you will be prompted to "Categorize which other posts as ok?"


When the program is finished processing all the new status messages, it will post an accuracy rating for the current session (basically, if you don't mark any of the "ok" messages as bad or correct any of the "bad" ones to be ok, it thinks it has 100% accuracy). You can use this as a rough metric for how good the current training level is.

Limitations: The current version of Twyt does not seem to be picking up the newfangled "Retweet" posts. Based on the messages that I am missing, this is something of a net advantage.

The twi script can be found here.

Upgrading the Bayesian-filtered status reader

In a previous post, I described my assembly of a command-line program for Bayesian filtering and presentation of Facebook statuses. After a while, constantly pulling down the RSS feed (via curl) and seeing the same statuses for certain people, day after day until they expired, kind of wore on me, so I went looking for a simple feed reader. There were command-line feed readers that looked more appealing to me (Canto and Newsbeuter) but they required newer versions of Python or some series of installations of things that were not available in a sufficiently upgraded form in Fink. So I tried rawdog which is written in Python and only requires the easily installed feedparser. It seemed like it was modifiable to do what I wanted, and I even went so far as to post a triumphant blog post about my success with getting it to properly parse my Facebook feed and deliver it to my Bayesian-filtering/displaying script. And then the next day, it went haywire and started reshowing me things I had already seen and dismissed.

So.

I decided to do everything myself. I was already partially parsing the RSS feed, so I just went a little further and parsed out the post date and compared it with the date when I last looked through the posts. (This seemed daunting until I found out that Perl already has time functions built in.) Finally I coded in dumping the output to a text file, as well as to the screen. Now the command can work on its own or have its output easily combined with my separate Twitter Bayesian filter which is a bit more sophisticated and which does a nice job of increasing my local Twitter signal-to-noise ratio. I intend to post about it someday.

The code for stati.pl is here.

In search of the ideal Twitter command-line client

After testing out a bunch of candidates, my new default command-line Twitter client is Twyt. Do not be fooled by the spartan front page. Active development is taking place. It's written in Python and (as of version 0.9.2) has the virtue of offering the following command-line options for the "check stati of friends" command:

-s SINCE, --since=SINCE
The date or ID of a message to list status messages
from.
-P PAGE, --page=PAGE Lists the PAGEth page of status updates
-n COUNT, --count=COUNT
The number of statuses to retrieve (max 200)

Unfortunately, it seems to be suffering from the same malady as other programs: No matter what value I put in for the --since flag, I still get the same 20 stati... The --page flag does work though, and the --count flag is doing something...

However, I also like the twixer client because it has implemented a few commands that I have not found in any other command line program. According to its command line options, it can 1) list your friends, 2) list your followers, 3) show the little information box on a user, and 4) delete stati and direct messages.

Unfortunately (and maybe I am doing something wrong here), but as of version 0.1.1, it will only show the little information box on me, no matter who I put in. But status deletion works great!