When the new Retweet API came along, I wrote an RT module which uses the Twitter Search API to find relevant tweets and users, then retweets those messages or just somehow interacts with a user. As soon as the Twitter Lists API was announced, I started working on the Lists module, which eventually became a simple “sorting-the-tweeps” based on their keywords – web design, design, wordpress, etc. Soon enough, I found out that Lists couldn’t contain more than 500 members (although some glitch made 501 possible). Web design 2, wordpress 2, etc wouldn’t be as fancy. I also tried building a conversation list of tweeps that by any means talked to the robot, but then again, the 500 limit broke all my hopes ;)

There were other modules which I worked on really hard, such as RThx module or Random Buzz, DM Control. Some of them worked, some of them were turned off after a few days (yes, you guessed it – Random Buzz, that really made some noize ;)

So, what did I achieve? Me – nothing. Michael did though, in 6 months he went up to 4500+ followers, while following a little more than 200 people himself, has been featured in ~ 250 lists, sent ~ 55,000 tweets and retweets all based on four keywords (or hashtags) – design, web design, wordpress and jquery. Built 4 lists based on these keywords, 500 members in each. Total list followers is a little less than 150 (which is quite good actually).

For comparison take a look at my account – (@kovshenin), in a little bit more than a year I got ~ 1700 followers. It took Michael a couple of months to reach that. The chart below illustrates the followers growth during the last three months. Human (me – blue) vs Robot (Michael – red). Yeah, I added a new module in mid December ;)

I manually logged into Michael’s account recently to check out how he’s doing, and I was kind of surprised to see that people really are talking to the guy, thanking him for retweets, asking him for further reads, wishing him a great day and handing over some coffee. Michael doesn’t usually reply to these and he’s a little bit shy sometimes, besides, he never drinks coffee ;)

A few days ago I decided to give Michael a rest, so tuned his backend to a new Twitter account with a few different settings, especially in timing. I’ll be switching to other keywords and feeds in the next few weeks. So let’s see if he’s as good as Michael, or perhaps better? ;)

I’m not sure what I’ll do with Michael’s account. TweetValue said it’s worth over $5k … anyone? ;) Or should I just throw it away.. Or run a contest for his password? …

• Tweets around the clock
• Tweets from RSS feeds, supporting prefix and postfix text (for adding hashtags)
• Retweet via the Twitter Search API and build conversation lists
• Shoot random sentences at users who mention you, thank them for retweets
• Control your robot via your own Twitter account by sending him direct messages
• All this is Twitter OAuth powered, no password required
• Such robots are called Twibots

Now, before downloading the code, I have to warn you that it’s completely unorganized. The code is horrible, comments are awkward, the database being used is SQLite (just for the fun of it) and it’s very very glitchy. Be prepared for Twitter suspending your account for ‘strange activity’ and use this at your own risk, don’t run here blaming me for that ;) I also suggest you’d contact Twitter to get your IP addresses and Twitter account white-listed before you start, especially if you plan to tweet very often (which I wouldn’t recommend). Use this at your own risk, and please keep my copyrights and preferably the OAuth application IDs.

Download: here (version 0.1)

Operation Instructions.. To say the truth it’s pretty tough, no web interface, not buttons, no config files. There are a bunch of files there, some of them useless. There’s the Snoopy class for reading and parsing RSS, there’s the Twitter OAuth class, and two core php files – cron.php and oauth.php. Open up cron.php, there are some comments and examples there. Make sure you get your own bit.ly API key and secret. Also make sure you get a connection with the twibots.sqlite database which has a couple of empty tables. Those will be used for tokens and dump data for unrepeated tweets.

Once you’re done configuring, use the command-line php in order to make it work. It goes something like this:

# php cron.php oauth register
# Please browse to https://twitter.com/...
# php cron.php oauth validate 123465
# Authentication successful, greetings @ev ;)

# php cron.php random
# tweeting a random RSS feed ...

# php cron.php reply
# sending replies...

# php cron.php dm
# reading direct messages

# php cron.php retweet
# retweeting...

You’ll have to put that in your crontab file and launch by schedule. Don’t run them too often though, as Twitter doesn’t like flooding, especially from newly created accounts. Any questions or suggestions are welcome in the comments below, but please, don’t tell me the code is horrible, I know it is, and I wouldn’t have posted it if you didn’t ask ;) Cheers!

So, direct messaging the robot’s twitter account with the text “update status text” would make him tweet “status text” to the public timeline. Remember we had three branches of actions – feed, reply and rthx? Let’s add a fourth one and call it dm. This branch will simply scan through the account’s latest direct messages, find those sent by you and tweet them out loud. Again, as I said in the first tutorial, this is simply a prototype, just to get things up and running. You’ll have to polish this off for actual use and yeah, storing the access keys, dm_since_id, etc on disk is not such a good idea, you should probably use the database. Here’s the code:

// The id of the latest read direct message will be stored in
// a file called dm_since_id, just like mentions_since_id
// in the previous examples
$since_id = @file_get_contents("dm_since_id", true);
if ($since_id > 0) { }
else { $since_id = 1; }

// Retrieve the direct messages into $dms and parse the xml string
$dms = $oauth->OAuthRequest("http://twitter.com/direct_messages.xml" ,
	array("count" => 10, "since_id" =>  $since_id), "GET");
$dms = simplexml_load_string($dms);

// If it's valid read the latest id and store into dm_since_id
if (count($dms))
{
	$last_id = ($dms->direct_message[0]->id > $since_id) ?
		$dms->direct_message[0]->id : $since_id;
	file_put_contents("dm_since_id", (string)$last_id, FILE_USE_INCLUDE_PATH);
}

// Loop through the messages
foreach ($dms->direct_message as $dm)
{
	// Make sure you're the sender
	$sender = $dm->sender->screen_name;
	if ($sender == "kovshenin")
	{
		// What should we do
		if (strtolower(substr($dm->text, 0, 7)) == "update ")
		{
			// Construct the message, tweet and wait a few seconds
			$message = substr($dm->text, 7);
			$oauth->OAuthRequest('https://twitter.com/statuses/update.xml',
				array('status' => $message), 'POST');
			echo "Tweeting: " . $message;
			sleep(rand(5,30));
		}

		// Add more actions here
	}
}

Read through the comments in the code and you should be able to get the idea. The direct messages Twitter API method is documented here. Test it out a few times through your SSH client by sending a direct message to your robot with the text “update Updating my status” or whatever, then run:

# php robot.php dm
Tweeting: Updating my status

If everything works fine you might as well add the action to your cron, say 5 minutes:

*/5 * * * * php /home/youruser/twibots/robot.php dm

And done! Your robot is now remote controlled. A few suggestions to more advanced remote controlled operations would be:

• Retweet capabilities by regular expression
• Adding and removing feed sources, prefixes and postfixes
• Turning on and off other operations (feeding, replying, rthxing)
• Adding people to “reply ignore lists” (ones that talk to your robot too much)
• Adding people to “rthx ignore lists” (ones that retweet too much, twitterfeed for instance)

I think that’s enough for a start, oh and please don’t build dumb and annoying spammish robots. Stick to intelligent, smart twibots! ;)

Upd. Continued: Twitter Robot in PHP: Twibots Draft

Here’s a brief list of features we will implement:

• Runs in console, no HTTP access
• Authentication via OAuth, tweeting via OAuth
• RSS lookup, parsing, forming tweets in bound of 140 characters including a postfix (hashtag or RT)
• Tweeting ‘thank you for retweeting’ to users that retweet the robot
• Following people that retweet the robot
• Acting strange on users that mention the robot

All this is going to be setup on a linux box running crond and acting every 15 minutes or so. Are you ready? Let’s do it!

Registering an App at Twitter OAuth Clients

This is mandatory if we’re using OAuth (don’t go with basic authentication because Twitter will be closing that down sooner or later, plus we’re making a personalized robot, so the ‘from’ string in the tweets is a must). Browse to the Twitter OAuth Clients page (assuming you’re logged into Twitter) and register a new application. Make sure you pick Read and Write access as our robot will use POST calls to update its status. Also note that we’re preventing all HTTP access to the robot control, thus we’ll be using PIN based OAuth, so make sure you pick Client application and not web.

Now, don’t lose your Consumer key and Consumer secret – we’ll be using those in our app for identity. Don’t give them out to anybody you don’t trust unless you’d like people tweeting “from YourApp”. Note that you can ask Twitter to reset the values for you in case somebody got unwanted access to them. The request and access URLs are not important, they’re the same for everyone.

Setting up the Environment

What environment? Umm, we’ll be using Magpie RSS to parse RSS feeds, Curl to access the Twitter API and the Twitter OAuth Class by Abraham Williams (we used this a bunch of times in my earlier postings) which is dependent on the Curl functions. Make sure you install everything right – the magpie functions work, curl functions are available and the Twitter OAuth class is defined.

What you also need is a bit.ly login and API key, go get yours somewhere here. Feel free to use a different URL shortener if you feel like, but we’ll stick to bit.ly.

Authentication: Storing the Access Tokens

We’ll divide our application in two parts, one will be running every 15 minutes, the second one will run once for authentication. Create a new PHP file, call it oauth.php or whatever and also make sure you got your consumer key and secret next to you and visible by oauth.php. It’s up to you where to keep them, I like storing them in a config.php which I include/require in auth.php and robot.php.

If you haven’t read my previous articles about Automated Serverside Tweeting Using OAuth and PIN-based OAuth Using PHP then make sure you do, because I wouldn’t like to go through the same code again. What’s important here is that oauth.php will run on its own and will accept parameters via command line. Like this:

# php oauth.php register
Request tokens aquired, proceed to this link: https://twitter.com/oauth/authorize?oauth_token=whatever

# php oauth.php validate 123456
Access tokens stored, identified as @twittername

So after you’ve validated your OAuth session, access tokens are stored somewhere on disk, I just call my files access_token and access_token_secret. If the files are present, then your robot will be able to tweet, otherwise ask for registration. I hope everything’s clear enough here. The main point of this file is to get your access tokens written to disk.

Feeding Twitter from RSS

We need to distinguish different actions which the robot is supposed to do, through an action parameter passed by the command line. I leave it up to you, but make sure your control flows similar to this:

$action = $argv[1];
if ($action == "feed")
{
	$feed_name = $argv[2];
	// Read the feeds, tweet the feeds
}
elseif ($action == "rthx")
{
	// Read the names, tweet the names!
}
elseif ($action == "reply")
{
	// Read the names, umm.. Say something!
}

So the robot control will look something like this:

# php robot.php feed wordpress
Feeding my public timeline with wordpress articles
# php robot.php rthx
Thanking peeps for the retweets and following them all
# php robot.php reply
Writing strange stuff to peeps that mentioned me

Good, now let’s assume we’re in the feed action. Define an associative array of feeds:

$feeds = array(
	"wordpress" => array("url" => "http://wordpress.org/development/feed/",
		"postfix" => "#wordpress"),
	"mashable" => array("url" => "http://feeds2.feedburner.com/Mashable",
		"postfix" => "via (@mashable)"),
);

Two is enough for starting. We’ll make our robot tweet #wordpress tweets and read out mashable’s website and tweet headlines via @mashable. Make sure that you’ve already written a check for access_token and access_token_secret and gave out an error message if they didn’t exist or were expired. We cannot tweet without those, remember? I’m assuming you’ve stored them into $access_token and $access_token_secret respectively and initialized the $oauth object:

$oauth = new TwitterOAuth($oauth_consumer_key, $oauth_consumer_secret,
	$access_token, $access_token_secret);

The second parameter (“wordpress” in the control above example) would be stored into a $feed_name variable. From there on we’ll go with code:

// Store the feed settings into $feed
$feed = $feeds[$feed_name];

// Fetch the feed and store the prefix
$rss = fetch_rss($feed["url"]);
$postfix = $feed["postfix"];

// Loop through the feed items
foreach ($rss->items as $item)
{
	// All simple enough here
	$title = trim($item["title"]);
	$url = $item["link"];

	// Let's make sure our feeds are in English, allow spaces and punctuation
	if (ereg('^[[:alnum:][:blank:][:punct:]]+$', $title))
	{
		// Escape the URL for bit.ly shortening and then shorten the link
		// This is the place where you have to use your bit.ly login
		// And the API key
		$url_escaped = urlencode($url);
		$bitly_url = "http://api.bit.ly/shorten?version=2.0.1";
		$bitly_url .= "&longUrl=$url_escaped";
		$bitly_url .= "&login=$bitly_login&apiKey=$bitly_key";

		$shortened_url = json_decode(file_get_contents($bitly_url));

		// If everything went okay, go on
		if ($shortened_url->errorCode == 0)
		{
			// Retrieve the shortened url from the json object
			foreach ($shortened_url->results as $key => $value)
				$shorturl = $value->shortUrl;

			// Form a new message from the short URL and the postfix
			$message = " $shorturl $postfix";
			$length = strlen($message);

			// We should trim down the title if it's too long
			// So that our tweets are 120 characters or less
			if (strlen($title) > 120-$length)
				$shorttitle = substr($title, 0, 117-$length) . "...";
			else
				$shorttitle = $title;

			// Add the title to the message
			$message = $shorttitle.$message;

			// Post the message to Twitter
			$oauth->OAuthRequest('https://twitter.com/statuses/update.xml',
				array('status' => $message), 'POST');

			// Wait a couple of mintes before the next tweet
			// Don't try and flood Twitter, remember, you have
			// Only 150 API calls per hour, use them wisely.
			sleep(rand(60,120));
		}
	}
}

Read carefully through the comments, you should be able to understand what we’re doing here. This is the RSS part, (almost) all clean and shiny. Proceed to the retweets part.

Automatically Thank Your Retweeters and… Follow them!

Be very very careful here as we don’t want to thank the retweeters too much. You’ll need to think of a mechanism to store the id of the latest retweet that you already thanked and use it in the since_id parameter when calling the Twitter API. I’ll leave this part up to you, but in general the code should look something like this:

// Read the since_id from a file
$since_id = @file_get_contents("retweets_since_id");
if ($since_id > 0) { }
else { $since_id = 1; }

// Send the Twitter API request for the latest 50 mentions
// That were posted after the since_id parameter we read earlier
$mentions = $oauth->OAuthRequest("http://twitter.com/statuses/mentions.xml" ,
	array("count" => 50, "since_id" =>  $since_id), "GET");

// Make the XML an object
$mentions = simplexml_load_string($mentions);

// Setup an array which will contain users to retweet and follow
$users_to_rthx = array();

// Read the last tweet's id and store it into the retweets_since_id file
$last_id = ($mentions->status[0]->id > $since_id) ? $mentions->status[0]->id : $since_id;
file_put_contents("retweets_since_id", (string)$last_id);

// Loop through the tweets
foreach ($mentions->status as $status)
{
	// Let's see if somebody retweeted you.
	// Err, remember to replace @yourname
	if (strpos(strtolower($status->text), "rt @yourname")
		|| strpos(strtolower($status->text), "via @yourname"))
	{
		// Add the guy to the retweeters array
		$users_to_rthx[] = $status->user->screen_name;
	}
}

// Remove duplicates (we don't thank somebody twice in a tweet)
$users_to_rthx = array_unique($users_to_rthx);

// Setup the tweet prefix and initialize the mentions variable
// The tweet_prefix is just in case ;)
$tweet_prefix = "Thanks for the retweets! ";
$tweet_mentions = "";
$tweet_prefix = "";

// Loop through the retweeters popping variables out of the array
while ($mention_this_guy = array_pop($users_to_rthx))
{
	// If the popped guy fits into our brand new tweet, add him
	if (strlen($tweet_prefix . $tweet_mentions .
		$tweet_postfix . "@".$mention_this_guy) < 140)
	{
		$tweet_mentions .= "@".$mention_this_guy . " ";

		// Send the friendhips create (follow) request to the API
		echo "Following: " . $mention_this_guy;
		$oauth->oAuthRequest("https://twitter.com/friendships/create/" .
			$mention_this_guy . ".xml",	array(), "POST");
	}
	// If he doesn't push him back into the variable, tweet and reset
	// The $tweet_mentions variable for the other retweeters
	else
	{
		array_push($users_to_rthx, $mention_this_guy);

		// Format the message and output to screen (for debugging)
		$message = $tweet_prefix . trim($tweet_mentions) . $tweet_postfix;
		echo "Tweeting: " . $message . " (" . strlen($message) . ") n";

		// Send the status update request to the Twitter API
		$oauth->OAuthRequest('https://twitter.com/statuses/update.xml',
			array('status' => $message), 'POST');

		// Wait a few seconds before the next round
		sleep(rand(5,30));

		// Reset
		$tweet_mentions = "";
	}
}

// If we've got something left in the mentions, we need to tweet that
if (strlen($tweet_mentions) > 0)
{
	$message = $tweet_prefix . trim($tweet_mentions) . $tweet_postfix;
	echo "Tweeting: " . $message . " (" . strlen($message) . ") n";

	$oauth->OAuthRequest('https://twitter.com/statuses/update.xml',
		array('status' => $message), 'POST');
	sleep(rand(5,30));

	$tweet_mentions = "";
}

The comments explain it all.

Did Somebody Say Anything? Random Replies

This is the fun part. Whenever somebody mentions your name (and it is not a retweet), we send them a strange message mentioning their name. The code looks exactly like the retweeters thank code above, except that we don’t search for “rt @yourname” and “via @yourname”, instead we work with ones that don’t contain “rt” and “via” at all.

Don’t forget to store the replies_since_id too as we don’t want to message users multiple times (infinite number of times actually) because they’ll block the frustrating robot. Here’s the while loop that actually tweets. In this part the names are stored in the $users_to_reply array and goes like this:

// The loop
while ($mention_this_guy = array_pop($users_to_reply))
{
	// Some random quotes ;) You can add your own
	$random_quotes = array(
		"Wha?", "Interesting ...", "Affirmative sir!", "Hmm, makes me think ..",
		"So you really think I'm not human? Well.. Umm.. *sigh*"
	);

	// Format the message and tweet a random quote
	$message = "@".$mention_this_guy . " " .
		$random_quotes[array_rand($random_quotes)];

	echo "Tweeting: " . $message . " (" . strlen($message) . ") n";
	$oauth->OAuthRequest('https://twitter.com/statuses/update.xml',
		array('status' => $message), 'POST');

	// Wait a little
	sleep(rand(5,30));
}

That’s about it. The last part is adding the whole stuff to crontab for automated work. Open up /etc/crontab and add a few lines:

# Feed mashable and wordpress tweets once every 15-16 minutes
*/15 * * * * php /home/youruser/twibots/robot.php feed wordpress
*/16 * * * * php /home/youruser/twibots/robot.php feed mashable

# Thank and follow the retweeters once every half an hour
*/30 * * * * php /home/youruser/twibots/robot.php rthx

# Reply to people hourly
01 * * * * php /home/youruser/twibots/robot.php reply

Restart your cron daemon and voila! I’m used to having full control over my virtual private server so if you use simple shared hosting you should access the crontab (also called cron jobs) via your CPanel. Don’t forget to actually write the php command because .php files aren’t actually executable by linux. Also note that my files are stored in a non-accessible by apache part of the hard drive, so omit putting them into public_html, www or whatever. If you do though, make sure you define a “deny from all” rule in .htaccess. We don’t want other people messing with our newly born robot. Robots, robots, robots… What should we call them? Twibots? ;)

Upd. Continued: Automated Twitter Bot in PHP: Remote Control and Twitter Robot in PHP: Twibots Draft

Y’all know Russian, don’t you? Kidding.. The guys in the video talk about the development of robotics in Russia, why people are afraid of robots and why they shouldn’t be (the three laws of robotics by Asimov). This “thug” (AR-600) can now make some steps back and forth then side to side (though they don’t show them on this exhibition, cause it requires a special surface). He can also recognize people’s faces and greet them individually. Oh, and almost forgot, he can read numbers from papers and add or subtract them! Smart ass! ;)

P.S. Available in HQ (click the play button, then HQ next to the full-screen button)

Sorry for the quality, I hadn’t had a camera back then so I caught those on my Nokia cellphone. It was fun though, our Russian president Dmitry Medvedev shook hands with the android, and then the newspapers here posted out some good photos of them both, stating that Dmitry’s standing on the right, in case somebody got confused LOL. BTW that’s me in the orange shirt. It was almost midnight and we had a bunch of technical problems, that’s why I looked so sad.

P.S. I’m on 12seconds.tv, follow me!

If anybody’s interested, be sure to check out the photos and videos right here: androidov.net. The blog’s in russian, but you should get the most of it ;)

Regards.