Storing large values in MySQL fields with PHP

Tue, 29 Sep 09 22:55:00 +0200

Today a colleague at work ran into a problem when trying to store file contents into a MySQL database table. We are using this solution for quite some time now and it has always worked pretty good for documents and files up to 15 MB. The main reason for us to put files in the database is that it is easier to migrate a website to another server because the database contains all the data that we need. The problem appeared at a customers' webserver that is running Fedora. First we thought there must be something wrong with our PHP code. But as the code worked fine on our servers (Debian) we soon found out that we couldn't store more than 1 MB in our content field in the database. After a little more investigation the only solution to the problem seems to be raising the mysql system variable max_allowed_packet to a value that is bigger than the largest content that should go into a single field. For Debian the default max_allowed_packet size is 16 MB, but since the max_allowed_packet is not set for Fedora, MySQL defaults to 1 MB. We have no access to the MySQL server configuration so we had to solve problem this on the client side (our PHP code). An example of how this can be done is given here.


$fileName = '/file/name/of/uploaded/file.bin';
$fileSize = filesize($fileName);

// $db holds the (ADODB) database connection
// Fetch current packet size
$packetSize = (int) $db->GetOne('SELECT @@max_allowed_packet');
if ($packetSize < $fileSize + 2048) {
	$sql = sprintf('SET @@max_allowed_packet=%d', $fileSize + 2048);
	$db->Execute($sql);
}

$db->Execute('INSERT INTO `file`(`filename`, `filesize`) VALUES(?, ?)', array($fileName, $fileSize));
$fileId = $db->Insert_ID();

// Here you can store the file contents in database
$handle = fopen($fileName, 'r');
while (($data = fread($handle, 524288)) && strlen($data) > 0) {
	// Update record with chunks of 512K, until the file is completely in the database
	// This prevents PHP from running out of memory
	$sql = sprintf(
		'UPDATE `file` 
		SET `contents` = CONCAT(`contents`, ?) 
		WHERE `id` = ?');
	$this->db->Execute($sql, array($db->Quote($data), $fileId));
}
fclose($handle);

// now set the  max_allowed_packet back to old value
$sql = sprintf('SET @@max_allowed_packet=%d', $packetSize);
$db->Execute($sql);

It is possible to store values up to 1 GB in a field in MySQL (as 1 GB is currently the maximum allowed packet size). It is however not advisable to use the above method using CONCAT to put large files in the database, CONCAT gets very slow when the size of the blob gets bigger. If you want to put large files in the database you probably want to take a look at the LOAD_FILE function. This is however more complex if MySQL doesn't run on the same server as your PHP code is running on. As you can see in the above PHP code the following MySQL queries can be used to retrieve and set the max_allowed_packet variable for the current connection.


-- Fetch maximum allowed packet size (in bytes)
SELECT @@max_allowed_packet;

-- Set maximum allowed packet size (in bytes)
SET @@max_allowed_packet=33554432;

We use the following table definition to store the files (I have left out some unimportant fields).


CREATE TABLE `file` (
  `id` int(10) unsigned NOT NULL auto_increment,
  `filename` varchar(255) NOT NULL,
  `filesize` varchar(50) NOT NULL,
  `contents` longblob NOT NULL,
  PRIMARY KEY  (`id`)
) ENGINE=InnoDB CHARSET=utf8

About me

Fri, 14 Aug 09 21:30:52 +0200

Hi, my name is Joris van de Sande. From the moment that I was born in 1983, I have always been interested in technology. My first experience with a PC was back in 1991 when my parents bought their first PC (based on an Intel 80286 processor). In these days I used the computer to play games like Tetris, Digger and Space Invaders and to create documents in Word Perfect 5.1 (for DOS). Fast forward a few years (I think it must be 1996) and I was playing around with DBase III databases. A couple of months later my dad learned me how to interact with the DBase databases via the programming language Clipper and how to create a simple user interface with it. After some time my first useful program called "Taal" (which means "Language" in Dutch) was born. I used it to learn foreign languages like English and German. In 1999 I created a simple HTML website for my group of friends, to let the world know about our existence. That was one of my first experiences with building websites. One year later I redesigned the website (with Dreamweaver) and I wanted the website to be more interactive, so I stumbled upon PHP (version 3.x at that time) and started writing PHP scripts. Soon a MySql database was in use too.
© geeksaresexy.net A couple of years, programs and websites later I went to college and I received my bachelor degree in Computer Science in 2005. In November 2005, I started working as a Junior Application Programmer at Prezent Internet. At that time I was mainly working on customer websites (build in PHP), creating frontends and new modules for the CMS that was build in house. Today I work as a Software Architect at Prezent Internet. I am responsible for the architecture of most web applications that are build by Prezent Internet today. Besides that I spend my days coding (PHP, JavaScript, HTML, CSS and MySql), teaching my colleagues about software design and object oriented programming, maintaining the CMS and sometimes I even act as a system administrator (for Windows and Linux). When I have some spare time I like to listen to music, watch (action) movies or one of my favorite TV series (Numb3rs, NCIS and House), drink some beer with my friends, visit a dance festival, browse the Internet for interesting (open source) projects and other stuff that I am interested in, play with different operating systems and I like programming.

Devplace

Storing large values in MySQL fields with PHP

About me