Tag Archives: mysql

Working With Amazon EC2: Tips & Tricks

It’s been a while now since I’ve been hosting on Amazon Web Services and I’d just like to point out some issues I had and quick ways of solving them. We’re gonna talk about setting up a server that would serve not only you, but your clients too, cause $100/mo is quite expensive, isn’t it? So let’s begin and keep this as straightforward as possible. If you don’t understand something, it’s probably because you haven’t read the official EC2 docs and haven’t searched the forums. This is not a tutorial, it’s just a set of rules you may want to follow to make things right.

Once you start a new instance from an Amazon predefined AMI (Fedora Core 8 for example) I suggest you start building your structure right straight away. Attach an EBS volume to you instance (I mount it to /ebs) and start creating your users with their home directories in /ebs/home/kovshenin not the regular /home/kovshenin. Also point your MySQL server to keep your database files in /ebs/mysql. There are plenty tutorials out there on how to do that.

Now, edit your httpd.conf, add your vhosts, point them to the right users dirs, install an ftp server and make sure you chroot the users to their home directories. That way they won’t be able to mess up with eachothers files and folders, peek passwords etc. You might want to change the root user’s home directory to / instead of /root in case you’ll want to use ftp via your root user (which is quite dangerous).

Now comes the fun part. The HTTP server runs under the apache user by default in FC8 and I recommend you don’t touch this. Damn it took me quite some time to figure out how the heck can the apache user execute and write to files not belonging to apache. I messed up big time with the groups, adding apache to all my client’s users groups, but thank god I found mod_suphp in the end. Install that one and make sure you use it and there’s no need to change the users umasks anymore.

Note: There’s a little issue with the mod_suphp in Fedora as far as I know, which doesn’t let you use the suPHP_UserGroup directive in the httpd.conf yelling that it does not exist. Most of the man pages on the net say you have to use that directive, but I’m good without it. It seems that suphp can figure out what user to run on its own, look closely at the config files, and also make sure you’re running php-cgi, not the CLI version. By the way, this is the part where WordPress stops asking you your FTP credentials on plugins/themes update, install, remove and core upgrade too. Speeds up the whole process ;)

I used the following code to test how mod_suphp works (or doesnt):

<?php echo system("id"); ?>

Which should output what’s the current user. Make sure you check everything works before going public, and do not set your min_uid and min_gid in suphp lower than 50. It’s safer to chown -R files and folders than to let suphp run your scripts via root or some other powerful user.

Backing up your EC2 and EBS

This is very important. Once you have everything set up and running, DO backup. Backing up the EBS is quite simple, just create a snapshot from the Amazon EC2 Management Console. Backing up the running AMI (instance) is a little bit mroe complex. You have to use the ec2 command line tools to bundle a new volume, upload it to an Amazon S3 bucket and register the AMI. There are plenty tutorials on the net on how to do that. Shouldn’t take you more than half an hour to figure it out.

Just make sure you have copies of all the major config files (httpd.conf, crontab, fstab, ..) backed up on your /ebs/config for instance. You might need them in the future (when you loose everything, haha ;) Restoring a backed up AMI instance is simple. Launch a new instance using the AMI you generated, attach the Amazon Elastic IP address to it and voila. Way too simple.

About the EBS, there are quite a few things you should be able to do with it before continuing. Restoring a backed up Snapshot: Create Volume from Snapshot, umount /ebs, deattach old volume, attach new volume, mount /ebs. Cool? Be careful when you’re resizing your EBS. The xfs filesystem automatically grows as far as I know, but in my case I use the ext3 filesystem. So if you need to grow your ext3 EBS you’ll go:

  1. Create a Snapshot
  2. Create a new EBS Volume from that Snapshot you created (say 10 GB if you were running 5 GB)
  3. Attach it to your Instance, say /dev/sdg
  4. Use the resize2fs command to resize the partition to 10GB
  5. Mount it to /ebs2 or whatever
  6. Check to see if everything’s in place
  7. Unmount /ebs2, deattach /ebs2, unmount /ebs, deattach /ebs
  8. Attach the 10GB volume to where /ebs was attached (/dev/sdf)
  9. Mount /ebs and start your services

There you go, back to work, server. By the way, when working with Amazon AWS, note that you should be working in the same region where your AMI is (us, eu, east, 1c, …) otherwise some of the options (when attaching, etc) might just not come up. Beware of that.

Well, I guess those are pretty much all the basics. Don’t forget to read the Amazon S3 tutorials and API, pretty sweet stuff! Good luck.



Foller.me: MySQL Tweaking & Optimization

As I mentioned in the interview with @enked on his website Chidimar.com, I had serious problems with MySQL database optimization on the Foller.me project. The current public stable version (beta-1) is using the MyISAM engine and it’s not holding much data – profiles, locations, geo points for the followers geography.

In the new version (currently dev-1 and hopefully beta-2 in a few days) I changed most of the old tables and added new ones, and using InnoDB this time. You see, it’s not very easy to scan through ~1,000,000 relations for the @mashable account ;) and I bumped into a ~10 second delay before the @mashable profile showed up at Foller.me. Slow query showed that one of the simplest queries caused that slow-mo – it took 6 seconds to execute! The guys at Stack Overflow helped me optimize the query and the two tables I was having problems with, so I came down to ~2 seconds for that query, neat!

Digging further I managed to tune the MySQL server up a little bit (caching, all sorts of buffers, etc – you should definitely take a look at MySQLTuner, it’s a perl script that helps you tune pretty much all the MySQL config) which decreased the query execution time to 1 s. The peeps at Stack Overflow said it’s pretty okay for that kind of query to execute for 3 seconds on over 2 million rows, so I thought that 1 second is final. Phew! :)

Now, think about the MySQL query cache. It doesn’t work in my situation, simply because I shoot UPDATEs and INSERTs at the relations table every five minutes or so (via a cron job), so there actually is a way to perform even higher. Thought of temporary tables, views and triggers (and even stored procedures). Nah.. Simply caching that query would be good, right? I mean if I cache the whole profile for an hour, why wouldn’t I cache the relations result set? Cache the query.. Aha, but I thought slightly further. Why not cache the whole page with memcached? I’ll keep you updated with the results.



Three Linux Commands You Can't Live Without

Okay, we’re not going to talk about the shutdown, yum, etc. commands, though THEY are probably the ones that nobody can live without. I’m talking about the web here, remember? And we all know that not everybody owns a VPS, a VDS or a dedicated server. Virtual hosting plans are quite cheap today, and most of them are tuned to provide you with SSH access and basic privileges (although I DO highly recommend you get yourself a VPS, it’s not much more expensive than virtual hosting and I’m sure it’s worthed). These linux commands are extremely helpful when doing backup or moving from one server to another.

1. Create a compressed archive of the current directory:

tar -cvzf backup.tar.gz ./*

2. Create a compressed archive of a MySQL database dump:

mysqldump -u username -p password -h host -P port databasename | gzip -c > mysql.sql.gz

3. Get directory contents from a remote FTP server:

wget -r ftp://username:password@domain.com/directory/*

Hope that helped ;)



PHP: Thinking Ahead

Hey. Some of you might have already heard about me launching a computer graphics web 2.0 thingy here in Russia and that is basically the reason why I’m working on this series of php classes. Why didn’t I choose to run some WordPress, WordPress MU or BuddyPress? Not flexible enough. Zend Framework? Too flexible and time-consuming. You can check out the designs on my DeviantArt profile I’ll be very glad to have some feedback.

Okay, back to PHP. Now, what I did was I wrapped up the User and Art classes together into a class called Item, which can now contain almost any piece of information together with its meta data (if needed). The User and Art classes now extend the Item class with a custom constructor (and maybe other functions in the future).

About the Art class, I know I haven’t written anything about it yet and that’s because there’s nothing interesting about it. It’s very similar to the User class, just uses different tables. Anyways they’ve now merged so you should just forget about it. One more interesting class is the ItemSet. Remember I talked about the upcoming UserSet class? ItemSet can now deal with both UserSet and ArtSet that extend it, and another sweet thing about it is that it implements Iterator so you can use the objects in foreach loops.

Here are a few examples:

// Load the #1 user
$user = new User($db);
$user->load(1, true);

// Change some meta and save it
$user->meta_meta1 = "nice";
$user->save(true);
foreach($user->data as $key => $value)
	echo $key . ": " . $value . "<br />";

// Load all the arts
$arts = new ArtSet($db, "1=1", true);

// Print them out
foreach ($arts as $art)
{
	foreach($art->data as $key => $value)
	{
		echo "$key => $value<br>";
	}
}

And yeah, still supporting the method I wrote about in the previous posts about the User class. You should play with these new ones, though they could be tricky sometimes, especially when you’re working with meta variables and foreach loops, so be careful!

Here are the classes:

Have fun!



PHP: The User Class Snippet

Okay, this is gonna be fun. I just finished my own version of the User class in php. I know there are millions of others around, but I didn’t find one that would suit my project – they’re all too complicated and unflexible. I’ll start off by explaining some of the requirements and a couple of examples. I’ll introduce you to the class at the end. I had to slice the article with a cut in case someone isn’t interested at all. There’s way too much code here to display on my main page. So if you’re interested, read on…

The User class uses two tables in a MySQL database – the users table and the users meta table. You can go ahead and use just one (omitting the meta functions of the class), but I do encourage you to split up data that is needed on every run, and data that’s only required when viewing a user’s profile for instance. This will give you better performance. Just a short example for fields in the users table: id, username, fullname and avatar. Ones that may go into the users meta table: id, password, email, location, icq, url, etc. See the difference?

Anyways, my class assumes that you have the id and username fields in your users table, and id field in the users meta table (in case you want to use that). The ids must be uniqe in tables scopes and valid cross-table (i.e. user1′s id in both users and users meta tables should be 1). I prefer using a MySQL UNIQUE index on both tables and a AUTO_INCREMENT parameter for the users table. Kick me if I’m wrong.

Off with the examples:

// just an instance of the class, passing the mysql link
// as the first and only parameter
$user1 = new User($db);

// we initialize a user here whos id is 1
$user2 = new User($db, 1)

// same as above, but we user the username instead, and we load his meta data too
// the last two parameters indicate that we'll be using the my_users and my_users_meta
// tables instead of the default ones (you may change the defaults in the class)
$user3 = new User($db, "some_username", true, "my_users", "my_users_meta");

// load some_username into $user1 together with his meta
$user1->load("some_username", true);

// $user2 had user #1 loaded, we load in his meta this way
$user2->loadMeta();

// forget user #3
$user3->clear();

// output some data
echo $user1->username . "'s id is #" . $user1->id;
echo "He is " . $user1->age . " years old."; // in case you have an "age" field in the users table

echo "Some meta information perhaps: " . $user1->meta["icq"]; // must have an "icq" field in users meta table

// change some data
$user1->fullname = "Jack Sparrow";
$user1->meta["profession"] = "Pirate";

// save it to database
$user1->save();
$user1->saveMeta();

// or just
$user1->save(true); // which saves meta as well

// Let's create a couple of new users
$user1->create("john");
$user1->fullname = "John Smith";
$user1->save();

$user2->create("george", true);
$user2->fullname = "George Black";
$user2->data["meta"]["icq"] = 12345;
$user2->save(true);

I guess that’s enough. Let’s look at our class now.

class User
{
	protected $db;			// Ugh, database link
	protected $tb_users;	// The users table, duh
	protected $tb_meta;		// The users meta table, jeez

	protected $save_valid;		// Do we have valid user data? Used for saving and loading meta
	protected $save_meta_valid;	// Valid meta data (same as above)

	public $data;			// This is the data array (containing a "meta" array as the last element)

	function __construct($db, $query = false, $load_meta = false, $tb_users = "users", $tb_meta = "users_meta")
	{
		$this->db = $db;
		$this->tb_users = $tb_users;
		$this->tb_meta = $tb_meta;
		$this->save_valid = false;
		$this->save_meta_valid = false;

		if ($id) $this->load($query, $load_meta);
	}

	function __destruct() { }

	// Overloading
	public function __set($name, $value) {
		$this->data[$name] = $value;
	}

	public function __get($name) {
		if (array_key_exists($name, $this->data)) {
		return $this->data[$name];
	}

        $trace = debug_backtrace();
        trigger_error(
            'Undefined property via __get(): ' . $name .
            ' in ' . $trace[0]['file'] .
            ' on line ' . $trace[0]['line'],
            E_USER_NOTICE);
        return null;
 	}

	// I don't think there'll be much trouble here. We just load in a user (w/ or w/out meta). $query can be either a username or a user id.
	function load($query, $load_meta = false)
	{
		if (is_numeric($query))
			$sql = "SELECT * FROM `{$this->tb_users}` WHERE `id` = {$query} LIMIT 1";
		else
			$sql = "SELECT * FROM `{$this->tb_users}` WHERE `username` = {$query} LIMIT 1";

		$rs = mysql_query($sql, $this->db);
		if (!$rs || @mysql_num_rows($rs) == 0) return false;

		$this->clear();

		$row = mysql_fetch_assoc($rs);
		$this->data = $row;

		$this->save_valid = true;
		if ($load_meta) return $this->loadMeta();
		return true;
	}

	// Load the meta for the current user
	function loadMeta()
	{
		if ($this->save_valid)
		{
			$sql = "SELECT * FROM `{$this->tb_meta}` WHERE `id` = {$this->data["id"]} LIMIT 1";
			$rs = mysql_query($sql, $this->db);
			if (!$rs || @mysql_num_rows($rs) == 0) return false;

			$row = mysql_fetch_assoc($rs);
			$this->data["meta"] = $row;
			$this->save_meta_valid = true;

			return true;
		}
		else
			return false;
	}

	function save($save_meta = false)
	{
		if ($this->save_valid)
		{
			$id = $this->data["id"];
			$query = "";
			foreach($this->data as $key => $value)
			{
				if ($key != "meta" && $key != "id")
				{
					if (!is_numeric($value)) $value = "'{$value}'";
					$query .= "`{$key}` = {$value}, ";
				}
			}
			$query = rtrim($query, " ,");
			$sql = "UPDATE `{$this->tb_users}` SET {$query} WHERE `id` = {$id} LIMIT 1";
			$return = mysql_query($sql, $this->db);
			if ($save_meta) return $this->saveMeta();
			else return $return;
		}
		return false;
	}

	function saveMeta()
	{
		if ($this->save_meta_valid)
		{
			$id = $this->data["id"];
			$query = "";
			foreach($this->data["meta"] as $key => $value)
			{
				if ($key != "id")
				{
					if (!is_numeric($value)) $value = "'{$value}'";
					$query .= "`{$key}` = {$value}, ";
				}
			}
			$query = rtrim($query, " ,");
			$sql = "UPDATE `{$this->tb_meta}` SET {$query} WHERE `id` = {$id} LIMIT 1";
			return mysql_query($sql, $this->db);
		}
		return false;
	}

	function create($username, $load_meta = false)
	{
		$sql = "SELECT `id` FROM `{$this->tb_users}` WHERE `username` = '{$username}'";
		$rs = mysql_query($sql, $this->db);
		if (mysql_num_rows($rs) > 0) return false;

		$sql = "INSERT INTO `{$this->tb_users}` (`username`) VALUES ('$username')";
		if (!mysql_query($sql, $this->db)) return false;
		$id = mysql_insert_id($this->db);
		$sql = "INSERT INTO `{$this->tb_meta}` (`id`) VALUES ($id)";
		if (!mysql_query($sql, $this->db)) return false;

		return $this->load($id, $load_meta);
	}

	function validate()
	{
		if (isset($this->data["id"]))
		{
			$this->save_valid = true;
			if (isset($this->data["meta"]))
				$this->save_meta_valid = true;
			else
				$this->save_meta_valid = false;
		}
		else
			$this->save_valid = $this->save_meta_valid = false;
	}

	function clear()
	{
		$this->data = array();
		$this->save_meta_valid = $this->save_valid = false;
	}
}

Don’t worry about the validate() function, it’s just used for checking if a user (and his meta) are valid and setting the appropriate variables. This will be used later on, when I’ll publish the UserSet class, which will be able to hold an array of users and iterate through them.

Alright, I guess that’s it. If you have any questions, or perhaps suggestions about the class and possible improvments, I’ll hear them out in the comments below. Thank you.



MySQL Charset Issues

And yet another MySQL character set problem solved. I had a remote FreeBSD server running MySQL 4.0 and a local Windows server running MySQL 5.0 (the XAMPP web servers package). I had no problems doing an export from MySQL 4.0 (running on cp1251 charset) to MySQL 5.0 (running utf8 charset) with something like this:

mysqldump -u -p --default-character-set=cp1251 dbname tablename > table.sql

Adding a few lines using vi at the beginning of the created file:

SET NAMES cp1251;
SET CHARACTER SET cp1251;

I imported the table dump into MySQL 5.0 and was able to see windows-1251 encoded characters using MySQL Query Browser. I’ve deleted two rows from the resultset, applied, then dumped back using mysqldump.exe:

mysqldump -u -p dbname tablename > dump.sql

When I tried to import the dump back to my remote MySQL 4.0 server

mysql -u -p dbname < dump.sql

I got some errors concerning the unavailable in MySQL 4 and lower SET NAMES and SET CHARACTER SET commands, which was fixed using the compatible flag during export:

mysqldump -u -p --compatible=mysql40 dbname tablename > dump.sql

That kept me going. Anyways, the dump back to MySQL 4 succeeded, but the character set didn’t match. I got utf8-coded cp1251 characters which looked weird, so I tried the following:

mysqldump -u -p --compatible=mysql40 --default-character-set=cp1251 dbname tablename > dump.sql

And got an error: mysqldump: Character set ‘cp1251′ is not a compiled character set and is not specified in the ‘C:mysqlsharecharsetsIndex.xml’ file. MySQL 5 was installed in C:/xampp/mysql, so I tried changing the [mysql] and [mysqldump] sections in the my.cfg file, but that didn’t help. It seems that mysql.exe, mysqldump.exe and other mysql utilities for windows are pre-compiled using a specific configuration.

The following solved my problem:

mysqldump -u -p --compatible=mysql40 --default-character-set=cp1251 --character-sets-dir=C:/xampp/mysql/share/charsets/ dbname tablename > dump.sql

Which got me a plain cp1251 encoded MySQL 4 compatible dump containing all my data. I’m not sure, but this seems to be a MySQL for Windows issue, cause I’ve seen posts stating same errors but using a standalone MySQL 5.0 server installation (unlike the XAMPP built-in). There you go ;)