Scripting MySQL

How To Sort Columns of MySQL Data on a Web Page With Perl

Tony Darnell — Fri, 02 Dec 2011 11:00:19 +0000

A friend of mine was building a web site so his customers could view his current inventory of transportation containers, and he asked me for help on how to sort the rows of information that appeared on his site. So, in this post, I will give you a quick example on how to sort columns on a web page.

First, let’s start with an inventory database that we will build in MySQL:

CREATE TABLE `inventory` (

  `id` int(6) NOT NULL AUTO_INCREMENT,

  `item_name` varchar(30) NOT NULL,

  `item_SKU` varchar(20) NOT NULL,

  `item_cost` decimal(4,2) NOT NULL,

  PRIMARY KEY (`id`)

) ENGINE=InnoDB AUTO_INCREMENT=100000 DEFAULT CHARSET=latin1

Next, here are some SQL statements to populate the MySQL database with some sample data.

use inventory;

insert into inventory (item_name, item_SKU, item_cost) values ('Apple', '10001', '1.04');

insert into inventory (item_name, item_SKU, item_cost) values ('Peach', '10004', '1.28');

insert into inventory (item_name, item_SKU, item_cost) values ('Plum', '10301', '1.17');

insert into inventory (item_name, item_SKU, item_cost) values ('Apricot', '13033', '1.92');

insert into inventory (item_name, item_SKU, item_cost) values ('Grapes', '20422', '1.34');

insert into inventory (item_name, item_SKU, item_cost) values ('Kiwi', '98561', '2.78');

insert into inventory (item_name, item_SKU, item_cost) values ('Mango', '56231', '0.99');

insert into inventory (item_name, item_SKU, item_cost) values ('Strawberry', '24689', '1.52');

insert into inventory (item_name, item_SKU, item_cost) values ('Banana', '65213', '0.39');

insert into inventory (item_name, item_SKU, item_cost) values ('Tangerine', '47112', '1.22');

For this example, I am not going to show you how to create a new record, edit or delete your information in a MySQL database, as I did that in an earlier post.

Now that we have our data, we are going to need a Perl script that will retrieve all of the data and create our web page at the same time. I have named the Perl script inventory_view. Don’t forget to change all of the variables to match your system. I will explain the variables that we will use in the Perl Script after the script below:

#!/usr/bin/perl

#--------------------------------------------------------------------------

# inventory_view.pl

#--------------------------------------------------------------------------
use DBI;

use DBD::mysql;

use CGI qw(:standard);
my $Database = "scripting_mysql";
# get the sort parameters from the URL

my $query = new CGI;

# $sort is the order by which to sort - ascending or descending

# you only need to set a value if it is descending (desc)

$sort = $query->param("sort");

# $sortby is the field name to be used for sorting

$sortby = $query->param("sortby");
# check for sort order - ascending or descending

if ($sort =~ "asc")
{

	$order_by = "order by $sortby";

}
else
{

	$order_by = "order by $sortby $sort";

}
# if the sort isn't set to a value, set it to a default sort of item_name

if (length($sort) < 1)
{

	$order_by = "order by item_name";

}
# print HTML header

print header;
	# connect to the database and pull every record

	$dbh = ConnectToMySql($Database);

	$query = "select item_name, item_SKU, item_cost from inventory $order_by";

	$sth = $dbh->prepare($query);

	$sth->execute();
# print the table header

print <







HTML
# set the second line background color

$background_color = "#FFFFFF";
			# loop through the data fetched from the query

			while (@data = $sth->fetchrow_array()) {
				$item_name = $data[0];

				$item_SKU = $data[1];

				$item_cost = $data[2];
# alternate the background colors

if ($background_color =~ "#FFFFFF")
{

	$background_color="#FFFFCC";

}
else
{

	$background_color="#FFFFFF";

}
				# print the table rows, one for each item from the database

				print <
				






HTML

			}
# close the table

print "


  
  
  

Item Name
Item SKU
Item Price









				$item_name 
				$item_SKU 
				$item_cost 
				
";
exit;
# more on what I am doing with the accessSM file may be found at:

# http://scriptingmysql.wordpress.com/2011/07/27/connecting-to-mysql-with-perl/

#----------------------------------------------------------------------

sub ConnectToMySql {

#----------------------------------------------------------------------
   my ($db) = @_;
   open(PW, "<..\/accessSM") || die "Can't access login credentials";

   my $db= ;

   my $host= ;

   my $userid= ;

   my $passwd= ;
   chomp($db);

   chomp($host);

   chomp($userid);

   chomp($passwd);
   my $connectionInfo="dbi:mysql:$db;$host";

   close(PW);
   # make connection to database

   my $l_dbh = DBI->connect($connectionInfo,$userid,$passwd);

   return $l_dbh;
}

When you run the script for the first time in a browser, you should see something like this:

In our Perl script, we are using the $query variable to select the three columns from our MySQL database – item_name, item_SKU and item_cost. And we will decide which one to sort by with the variable $sortby. And we will assign ascending or descending order to the variable $sort. This will give us a total of six options for sorting columns and in which order:

$sortby	$sort
item_name	ascending order
item_name	descending order
item_SKU	ascending order
item_SKU	descending order
item_cost	ascending order
item_cost	descending order

To build the URL for each, we simply use the script name – inventory_view.pl – and add the variables $sortby and $sort along with their values, using an ampersand for the delimiter.

inventory_view.pl?sortby=[column_name]&sort=[asc or desc]

Example:

inventory_view.pl?sortby=item_name&sort=asc

We will then use an image of an arrow pointing upwards (for ascending order) and downwards (for descending order). Here is an example of the HTML for “sort by item_name and ascending order”: (don’t forget to add the full path names for both the location of the Perl script and the location of your image file)

You simply have to build a link for each of the six sorting options (two per database column) to correspond to each of the blue up/down arrows. All six links would look like this:

When the inventory_view.pl script is executed the first time, we have not set the $sortby column or $sort order, so it will default to sorting by item_name and ascending order (which really doesn’t have a value, as ascending order is the default sort order). Each time a sort request is made, a new connection will be made to the database, so that is something to consider when you decide which columns you want to be available to sort. And we aren’t putting a limit on the number of rows that are retrieved, so that you have multiple pages of items – but I will try to cover that in a future post.

While there are other solutions that don’t require you to hit the database each time, this should give you a quick (and dirty) way to sort columns of information in a table on a web page.

—————————————–

Tony Darnell is a Principal Sales Consultant for MySQL, a division of Oracle, Inc. MySQL is the world’s most popular open-source database program.

Tony may be reached at info [at] ScriptingMySQL.com and on LinkedIn.

Automatically Download MySQL Enterprise Monitor Graphs as PNG Files Using Perl

Tony Darnell — Fri, 18 Nov 2011 10:00:08 +0000

I was giving a presentation of the MySQL’s Enterprise Monitor* application to a client recently. I was demonstrating the “graphs” section of MEM, where you can monitor MySQL sessions, connections, replication latency and more with 60+ graphs. Usually, you view the graphs from within the MEM Enterprise Dashboard (via a web browser). But the client asked if there was a way to automatically download graphs. I wasn’t sure why he wanted to download the graphs (I didn’t ask), but I knew it wasn’t possible by using MEM alone. However, in the past I have written Perl scripts to automatically download files from web sites, so I thought I would see if it was possible with MEM.

*The MySQL Enterprise Monitor (MEM) continuously monitors your MySQL servers and alerts you to potential problems before they impact your system. Its like having a “Virtual DBA Assistant” at your side to recommend best practices to eliminate security vulnerabilities, improve replication, optimize performance and more. As a result, the productivity of your developers, DBAs and System Administrators is improved significantly. (from: http://www.mysql.com/products/enterprise/monitor.html)

Of course, you have to install MEM and at least one agent. Let’s assume that you have already accomplished this task, and that MEM is running properly. Open MEM in your browser, login, click on the graphs tab, and then you will see a list of all of the available graphs.

For this example, we are going to automatically download the Agent Reporting Delay and the Disk IO Usage graphs. We will download the first graph for all of the servers in a particular group, and the second graph for an individual server. First, click on a server group in your server list on the left side of MEM.

Next, we will need to change the Time Range settings to “From/To”, so that we can enter the a timeline for the graph in our script. Don’t worry about the time settings that are in MEM, as we will change these settings later, but we need them so that they will be included in the URL that we will use (more on this later). After you have changed the Time Range settings, click on the “Filter” button.

Next, click on the plus sign for the graph that you want to use so that MEM will draw the graph. For this example, we will click on the “Agent Reporting Delay” graph:

You will notice two icons to the right of the graph name. The first icon (on the left) allows you click on the icon to download the graph as a .csv file. The second icon (on the right) allows you to click on the icon and download the graph as a PNG image file.

We need some information from the actual link that is used when you click on the PNG icon. So, we will need to right-click on the icon to get the URL link location information for the Agent Reporting Delay graph:

The URL for this graph is then copied to your clipboard. This is the URL location (which is for all servers in the group that I selected):

http://192.168.1.2:18080/Graph.action?dims_height=300&dims_width=800&graph=f924cb42-fed5-11df-923c-a6466b4620ce&locale=en_US&noDefaults=false&servers_group=0&style=NORMAL&time_fromDate=2011-11-16&time_fromTime=11%3A24&time_toDate=2011-11-16&time_toTime=11%3A54&time_type=FROMTO&tzName=America%2FNew_York

As you can see in the URL above, there are several variable values that we will include in our script to produce our graphs (in blue text above). In this example, we will only be working with the following variables:
- dims_height
- dims_width
- time_fromDate
- time_fromTime
- time_toDate
- time_toTime
- graph name/ID (which is a UUID and is constant)
- servers_group and servers_server
(the servers_server variable and value are not shown in the above example, but will be in the next example below)

We will be using a text file named files.txt to store some of the graph variable values that will be used by the script. Now that you know how to copy the URL for a graph, you will need to extract the value for the graph variable and the value for the servers variable and place the values into your files.txt file. The graph value for the above URL (shown again below) is in blue text, and the value for the server variable is in red text: (notice that all values are separated on the left by an equal sign “=” and on the right by an ampersand “&”)

http://192.168.1.2:18080/Graph.action?dims_height=300&dims_width=800&graph=f924cb42-fed5-11df-923c-a6466b4620ce&locale=en_US&noDefaults=false&servers_group=0&style=NORMAL&time_fromDate=2011-11-16&time_fromTime=11%3A24&time_toDate=2011-11-16&time_toTime=11%3A54&time_type=FROMTO&tzName=America%2FNew_York

In the above example, we had selected a group of servers in our server list (on the left side of MEM), and therefore the URL will not have a value for the individual server (variable named servers_server). The graph that we will extract will be for this group of servers (in this case servers_group has a value of zero, which is still a value). This is what we had chosen under our Servers list:

Now, we want to select an individual server. In this case, we will click on “iMac-Tony”:

Now that we have chosen an individual server, in the URL for that graph, you will have a value for the variable named “servers_server”, as well as a value for servers_group – and you will need both values together. So, if you want a graph for an individual server, you will need to click on that individual server in your servers list, reselect the “Time Range” value of “From/To”, click “Filter”, and re-copy the PNG graph URL. Once we have copied the URL for this graph for an individual server, you will see a different value for the graph variable (in red) and a value for servers_group and servers_server (in blue) like this:

http://192.168.1.2:18080/Graph.action?dims_height=300&dims_width=800&graph=6d9c8ac0-7a3b-11df-9df0-f30c5eb77a3c&locale=en_US&noDefaults=false&servers_group=0&servers_server=111&style=NORMAL&time_fromDate=2011-11-16&time_fromTime=15%3A27&time_toDate=2011-11-16&time_toTime=15%3A57&time_type=FROMTO&tzName=America%2FNew_York

We will use the above URL information for our second graph – the Disk IO Usage graph. You will need to copy all of the graph and server values for the graphs that you want to download. For the above URL, we will grab these values, to be placed in our files.txt file:
graph = 6d9c8ac0-7a3b-11df-9df0-f30c5eb77a3c
server group and server name = servers_group=0&servers_server=111

This is a tedious process at first, but you should only have to do this once for each graph. Mark Leith from the MySQL software development team in the UK gave me some great information for finding out the graph names along with the graph UUID value (be sure to also check out Mark’s blog). Mark stated “The uuid per graph does not change over time/versions – it’s how we maintain the constant between them in fact.” The graph name and graph value (UUID) is stored in the MEM Service Manager repository database, which contains all of the statistical information for MEM. To access this database, simply login to your MEM Service Manager MySQL repository database with the following syntax:

mysql -uroot -p -P13306 -h127.0.0.1

During the installation of MEM, if you decided to use your current MySQL database to store the repository information, you will need to just login to that database. Once you have logged into the repository database, you can get a list of the graph names and graph UUID’s with this command:


SELECT title, uuid FROM mem.graphs;

mysql> SELECT title, uuid FROM mem.graphs;

+--------------------------------------------+--------------------------------------+

| title                                      | uuid                                 |

+--------------------------------------------+--------------------------------------+

| agent_lua_mem_usage.name                   | 545e6c5e-ccab-457e-89ca-cc6e5eeb1e1d |

| agent_reporting_delay.name                 | f924cb42-fed5-11df-923c-a6466b4620ce |

| avg_row_accesses.name                      | f289dae0-82be-11df-9df0-f1c3fca44363 |

| binlog_cache.name                          | f84b270e-7a21-11df-9df0-f30c5eb77a3c |

.....

This should make it easier than copying the variables in the links as described above, but I wanted to show you how to get the information and explain all of the variables in the graph URL links. (Thanks Mark!)

In the files.txt file, we also want a name for the graph (which will also be used for the PNG image file name), the graph value, the servers value and the server or server group values from the above URLs, as well as the name of the server group or individual server. You should separate the values with a delimiter of three tildes "~~~".

So, for the two example graphs above, your files.txt file should contain the following values - Graph Name~~~graph value~~~server information~~~server or group name: (please note that the graph values that I have here may not be the same values that you would have for the same graph)

Agent Reporting Delay~~~f924cb42-fed5-11df-923c-a6466b4620ce~~~servers_group=0~~~All Servers

Disk IO Usage~~~6d9c8ac0-7a3b-11df-9df0-f30c5eb77a3c~~~servers_group=0&servers_server=111~~~iMac Tony

The first line above will produce an "Agent Report Delay" graph for "All Servers". The second line will produce a "Disk IO Usage" graph for only the server named "iMac-Tony".

Now that we have our files.txt file in place (it should be placed in the same folder as the Perl script - or you may modify the Perl script for a different file location), we will use this Perl script to download our graphs as PNG image files. In case you want to place this script in a cron job to run every X number of minutes, we will include a variable to allow you to select the previous number of minutes to include in your graph. For example, in the Perl script, if you set the value of the variable $time_interval to 60 (minutes) and run the job at 30 minutes past the hour, the script will retrieve a graph for the past 60 minutes from the time of script execution.

For this example, we will name the Perl script "get_graphs.pl". There will be some variables in the script that you will have to change once, to match your system's information. The variables that you need to change are highlighted in blue text in the script:

#!/usr/bin/perl
use WWW::Mechanize;

use Date::Calc qw(Add_Delta_DHMS); 
# file name for input - this contains the Graph Name and Graph URL

$filename = "files.txt";
# time interval must be in minutes

$time_interval = '60';
# the width of your graph

$dims_width = "800";

# the height of your graph

$dims_height = "300";
# IP and port number of your MEM server

$server = "192.168.1.2:18080";
# get the current time using the display_time_now subroutine

$unixtimenow = &display_time_now();

($time_toDate, $time_toTime) = split(" ",$unixtimenow);
# get the past time using the display_time_past subroutine

$unixtimepast = &display_time_past();

($time_fromDate, $time_fromTime) = split(" ",$unixtimepast);
# fool the web server into thinking we are a person

my $mech = WWW::Mechanize->new();

# look like a real person

$mech->agent('User-Agent=Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5');

# we need cookies

$mech->cookie_jar(HTTP::Cookies->new);
# Login to the MySQL Enterprise Monitor

$mech->get('http://192.168.1.2:18080/Auth.action');

$mech->success or die "login GET fail";
# you will need to substitute your user name and password for MEM here

my $user = 'tonydarnell';

my $pass = 'tonyd999';
# find a fill out the login form

my $login = $mech->form_name("DoAuth");

$login->value('username' => $user);

$login->value('password' => $pass);

$mech->submit();

$mech->success or die "login POST fail";
open(line, "$filename") || die (print "\nERROR - could not open file: $filename\n");

while ()
{
chomp $_;
print "\n$_\n";
($imagefilename, $graph_to_get, $servers_to_get, $servers_name) = split(/~~~/);
if (length($imagefilename) > 2)
{
	$time_toDate_for_filename = $time_toDate;

	$time_toDate_for_filename =~ s/\-/_/g;
	$time_toTime_for_filename = $time_toTime;

	$time_toTime_for_filename =~ s/\:/_/g;
	$servers_name =~ s/ /_/g;
	$imagefilename =~ s/ /_/g;

	$imagefilename = $servers_name . "_" . $imagefilename . "_" . $time_toDate_for_filename . "_" . $time_toTime_for_filename . ".png";
	# you will need to change your settings here to match your URL for your graphs

	$graph = "http://". $server . "/Graph.action?dims_height=" . $dims_height . "&dims_width=" . $dims_width . "&graph=" . $graph_to_get . "&locale=en_US&noDefaults=false&" . $servers_to_get . "&style=NORMAL&time_fromDate=" . $time_fromDate . "&time_fromTime=" . $time_fromTime . "&time_toDate=" . $time_toDate . "&time_toTime=" . $time_toTime . "&time_type=FROMTO&tzName=America%2FNew_York";
	print "\n$graph\n";
	#exit;
	# Get the PNG image file from the URL

	$mech->get($graph);

	$mech->save_content($imagefilename);
}
}
exit;
close($filename);
# ------------------------------------------------

# sub-routines
sub display_time_now {

  my ($sec,$min,$hour,$mday,$mon,$year,undef,undef,undef) = localtime time();

  $year += 1900;

  $mon += 1;

  return "$year-".sprintf("%02d-%02d %02d:%02d",$mon,$mday,$hour,$min);

}
sub display_time_past {

  my ($sec,$min,$hour,$mday,$mon,$year,undef,undef,undef) = localtime time() - ($time_interval*60);

  $year += 1900;

  $mon += 1;

  return "$year-".sprintf("%02d-%02d %02d:%02d",$mon,$mday,$hour,$min);

}
# ------------------------------------------------

When we executed the script, two files were created and downloaded - All_Servers_Agent_Reporting_Delay_2011_11_16_16_17.png and iMac_Tony_Disk_IO_Usage_2011_11_16_16_17.png:

All_Servers_Agent_Reporting_Delay_2011_11_16_16_17.png

iMac_Tony_Disk_IO_Usage_2011_11_16_16_17.png

You could also create a similar script to download the information as a .csv file, but the syntax is very different (maybe I will do that in a future post). But for now, I have a possible solution for the client - and I hope that he likes it.

-----------------------------------------

Tony Darnell is a Principal Sales Consultant for MySQL, a division of Oracle, Inc. MySQL is the world's most popular open-source database program.

Tony may be reached at info [at] ScriptingMySQL.com and on LinkedIn.

Using MySQL and Perl to Create, Edit and Delete Information Via a Web Page

Tony Darnell — Fri, 11 Nov 2011 15:10:11 +0000

A friend of mine was asking me for my recommendation of a good desktop database program to use to keep track of his inventory of cargo containers. I suggested to him that he should use MySQL and write a web page interface to do everything that he needed. He then reminded me that he is a lawyer by trade, and that he doesn’t have any computer programming experience. Then I remembered that he has almost zero computer skills. And his Texas Hold-Em skills are even worse, but I don’t mind taking his money. In his case, he should just use a notepad and a pencil. (As for the question – what is a lawyer doing with cargo containers? – that is a different story.)

If he did decide to broaden his horizons a bit, he could easily write his own software web application for creating and storing almost any kind of data. In this post, I will show you how to create a MySQL database and then the web pages needed to create new addresses, edit the same data and delete the data as well.

Of course, you will need to download and install MySQL. There are a ton of resources on the web for doing this, so let’s assume that you have already this part completed – and that you know how to use MySQL. First, let’s create a MySQL table. In this example, we will create a simple address book, and populate it with a few fake names. Here is the SQL, complete with the fake data to be inserted:

SET NAMES latin1;

SET FOREIGN_KEY_CHECKS = 0;
CREATE TABLE `address` (

  `serial` int(4) NOT NULL AUTO_INCREMENT,

  `name_first` varchar(30) NOT NULL,

  `name_last` varchar(30) NOT NULL,

  `address_01` varchar(40) NOT NULL,

  `address_02` varchar(40) NOT NULL,

  `address_city` varchar(30) NOT NULL,

  `address_state` varchar(2) NOT NULL,

  `address_postal_code` varchar(10) NOT NULL,

  PRIMARY KEY (`serial`)

) ENGINE=InnoDB AUTO_INCREMENT=7 DEFAULT CHARSET=latin1;
insert into `address` values('1','1','Clark','Kent','344 Clinton St','Apt. #3B','Metropolis','NY','10001'),

 ('2','2','Dave','Jones','500 Second Avenue','Suite 100','Atlanta','GA','30303'),

 ('3','3','Tom','Watson','123 Golf Course Lane','Suite A','Macon','GA','31066'),

 ('4','4','Jack','Nicklaus','400 Laurel Oak Dr','Suite 49','Suwanee','GA','31044'),

 ('5','1','Betty','Smith','100 Main Street','Suite 500','Buffalo','NY','14201'),

 ('6','2','Bruce','Wayne','1007 Mountain Drive','','Gotham City','NY','10000');

SET FOREIGN_KEY_CHECKS = 1;

We will be creating several Perl scripts as follows. To keep it simple, we will use Perl to create all of the pages, even though the “add_entry.pl” script could be a standard HTML page. This will allow you to put all of the scripts in your cgi-bin directory. Here are the scripts that we will be creating and using:
  — dashboard.pl -> This is your “home page”, and will list all of the entries in your address database.
  — add_entry.pl -> This page will allow you to complete a form containing the new address information.
  — add.pl -> This Perl script will save the information from the add.html web page.
  — edit.pl -> This Perl script will allow you to edit an entry from your address database.
  — edit_save.pl -> This will save the changes made from the edit.pl page.
  — delete.pl -> This is step one in deleting a record, it takes you to the delete_confirm.pl page.
  — delete_confirm.pl -> This will confirm the deletion of a record, or allow you to cancel the request.

For our home page (called dashboard), we need to create a web page (via a Perl script) that will list all of the addresses in the database. In this example, we aren’t using any logic to restrict the number of entries that are displayed, so this page will just display all of them (adding limits to a web page like this is a bit more complicated, and I might try to cover this in a future post). We will use a Perl script to display the HTML, as we will need to pull data from the database to be displayed in the web page. So, you will need to put the “directory.pl” script (home page) and all of the other scripts in your cgi-bin directory (and don’t forget to make all of your Perl scripts executable – via “chmod 755″). You could create a regular HTML page (.htm) using frames, with the top frame using HTML and then use a Perl script to pull the data for the bottom frame, but let’s just stick with this example for now.

Here is what the opening home page (dashboard.pl) should look like:

In this script, we will also create a link that will allow us to edit the information for a single address entry, as well as a link to delete the entry. We will use the serial database field to point to the record that we want to edit or delete. We will also display a link at the top for creating a new address entry, and this Perl script will be named “add.pl”. But first, here is the “dashboard.pl” script:

#!/usr/bin/perl
#--------------------------------------------------------------------------

# dashboard.pl

#--------------------------------------------------------------------------
# DBI is the standard database interface for Perl

# DBD is the Perl module that we use to connect to the MySQL database

use DBI;

use DBD::mysql;

use CGI qw(:standard);
my $Database = "scripting_mysql";
print header;

print <








Scripting MySQL - Address Book - Dashboard





HTML

# leave the above line to the left of the page		
# list my addresses
	$dbh = ConnectToMySql($Database);
	$query = "select serial, name_first, name_last, address_01, address_02, address_city, address_state, address_postal_code FROM address order by serial";
    $sth = $dbh->prepare($query);
    $sth->execute();
    $dbh->disconnect;
# list all of the addresses in a table

# be sure that the link for add.html is the correct path according to your system

print "
My Addresses - Create New";

print "
";

# print table row headers

print "";
# set your initial row background color

$bgcolor = "#EEEEEE";
          while (@data = $sth->fetchrow_array()) {

			$serial = $data[0];

			$name_first = $data[1];

			$name_last = $data[2];

			$address_01 = $data[3];

			$address_02 = $data[4];

			$address_city = $data[5];

			$address_state = $data[6];

			$address_postal_code = $data[7];
# print rows of data

# for the $serial, we include the link to the edit.pl script - edit.pl?$serial

print "
";
# alternate background colors

if ($bgcolor =~ "#EEEEEE") { $bgcolor = "white"; }
else
{$bgcolor = "#EEEEEE";}
# end while (@data = $sth->fetchrow_array())

			}

# print bottom of page

print <
ID First Last Address Address City ST ZIP


$serial $name_first $name_last $address_01 $address_02 $address_city $address_state $address_postal_code





HTML

# leave the above line to the left of the page		
exit;
# more on what I am doing with the accessSM file may be found at:

# http://scriptingmysql.wordpress.com/2011/07/27/connecting-to-mysql-with-perl/
#----------------------------------------------------------------------

sub ConnectToMySql {

#----------------------------------------------------------------------
   my ($db) = @_;
   open(PW, "<..\/accessSM") || die "Can't access login credentials";

   my $db= ;

   my $host= ;

   my $userid= ;

   my $passwd= ;
   chomp($db);

   chomp($host);

   chomp($userid);

   chomp($passwd);
# or, you may comment out the above nine lines - starting with open(PW...

# and just put in your database connection information below

# and un-comment these lines

# but I don't like putting my connection information in a Perl script

#   my $db = "database_name";

#   my $host = "host_name";

#   my $userid = "user_name";

#   my $passwd = "password";
   my $connectionInfo="dbi:mysql:$db;$host";

   close(PW);
   # make connection to database

   my $l_dbh = DBI->connect($connectionInfo,$userid,$passwd);

   return $l_dbh;

}
















The add_entry.pl web page will look like this (with the data for a new entry already entered):

And here is the script to create the add_entry.pl web page:


#!/usr/bin/perl
#--------------------------------------------------------------------------

# add_entry.pl

#--------------------------------------------------------------------------
use CGI qw(:standard);
print header;
print <








Scripting MySQL - Address Book - Add Entry




My Addresses - Dashboard













HTML
exit;




Name First 
Name Last 
Address 1 
Address 2 
City 
State 
Zip 





The form on this web page calls a Perl script named “add.pl”, which will insert the information to the MySQL database.
Here is the Perl script add.pl:


#!/usr/bin/perl
#--------------------------------------------------------------------------

# add.pl

#--------------------------------------------------------------------------
use DBI;

use DBD::mysql;

use CGI qw(:standard);
read(STDIN, $buffer, $ENV{'CONTENT_LENGTH'});
$buffer =~ tr/+/ /;

$buffer =~ s/\r/ /g;

$buffer =~ s/'/ /g;

$buffer =~ s/\n/ /g;

$buffer =~ s/%([a-fA-F0-9][a-fA-F0-9])/pack("C",hex($1))/eg;

$buffer =~ s// /g;

$buffer =~ tr/\\|[|]|<|!|"|$|{|}|*|#|'|>|||;|%/ /;
@pairs = split(/&/,$buffer);

foreach $pair(@pairs){

($key,$value)=split(/=/,$pair);

$formdata{$key}.="$value";

}
# here are the values from the HTML form

$name_first = $formdata{'name_first'};

$name_last = $formdata{'name_last'};

$address_01 = $formdata{'address_01'};

$address_02 = $formdata{'address_02'};

$address_city = $formdata{'address_city'};

$address_state = $formdata{'address_state'};

$address_postal_code = $formdata{'address_postal_code'};
my $Database = "scripting_mysql";
print header;

print <








Scripting MySQL - Address Book - New





HTML

# leave the above line to the left of the page		
# list my addresses
	$dbh = ConnectToMySql($Database);
	$query = "insert into address (name_first, name_last, address_01, address_02, address_city, address_state, address_postal_code) values (?,?,?,?,?,?,?)";
    $sth = $dbh->prepare($query);
    $sth->execute("$name_first", "$name_last", "$address_01", "$address_02", "$address_city", "$address_state", "$address_postal_code");
    $dbh->disconnect;
print "$name_first $name_last - was added to the database.
";
print "Return to the Dashboard";
# print bottom of page

print <


HTML

# leave the above line to the left of the page		
exit;
# more on what I am doing with the accessSM file may be found at:

# http://scriptingmysql.wordpress.com/2011/07/27/connecting-to-mysql-with-perl/
#----------------------------------------------------------------------

sub ConnectToMySql {

#----------------------------------------------------------------------
   my ($db) = @_;
   open(PW, "<..\/accessSM") || die "Can't access login credentials";

   my $db= ;

   my $host= ;

   my $userid= ;

   my $passwd= ;
   chomp($db);

   chomp($host);

   chomp($userid);

   chomp($passwd);
# or, you may comment out the above nine lines - starting with open(PW...

# and just put in your database connection information below

# and un-comment these lines

# but I don't like putting my connection information in a Perl script

#   my $db = "database_name";

#   my $host = "host_name";

#   my $userid = "user_name";

#   my $passwd = "password";
   my $connectionInfo="dbi:mysql:$db;$host";

   close(PW);
   # make connection to database

   my $l_dbh = DBI->connect($connectionInfo,$userid,$passwd);

   return $l_dbh;

}
















After you add an entry to the database, this is the confirmation web page after a record has been added.  

The add.pl page gives you a link to go back to the dashboard, which will then display all of your address database entries, including the one you just entered.
To edit an entry, from the dashboard web page, you simply click on the ID of the entry that you want to modify.  For example, when you click on the first entry with a serial number of “1″, you will see this link:


http://192.168.1.2/cgi-bin/scripting_mysql/edit.pl?id=1




Clicking on an serial number link will call the Perl script named “edit.pl”.  This will create a web page where you can edit the information and save it back to the database.  The edit.pl web page looks like this:

Here is the edit.pl script:


#!/usr/bin/perl
#--------------------------------------------------------------------------

# edit.pl

#--------------------------------------------------------------------------
use DBI;

use DBD::mysql;

use CGI qw(:standard);
$query = new CGI;
$id = $query->param("id");
print header;

print <








Scripting MySQL - Address Book - Edit




HTML
# grab the information for this id
	$dbh = ConnectToMySql($Database);
	$query = "select serial, name_first, name_last, address_01, address_02, address_city, address_state, address_postal_code FROM address where serial = '$id'";
    $sth = $dbh->prepare($query);
    $sth->execute();
    $dbh->disconnect;
# list all of the addresses in a table

# be sure that the link for add.html is the correct path according to your system

print "My Addresses - Dashboard";

print "
";
          while (@data = $sth->fetchrow_array()) {

			$id = $data[0];

			$name_first = $data[1];

			$name_last = $data[2];

			$address_01 = $data[3];

			$address_02 = $data[4];

			$address_city = $data[5];

			$address_state = $data[6];

			$address_postal_code = $data[7];

			}
print <















Name First   
Name Last   
Address 1   
Address 2   
City   
State   
Zip   






HTML
exit;
# more on what I am doing with the accessSM file may be found at:

# http://scriptingmysql.wordpress.com/2011/07/27/connecting-to-mysql-with-perl/
#----------------------------------------------------------------------

sub ConnectToMySql {

#----------------------------------------------------------------------
   my ($db) = @_;
   open(PW, "<..\/accessSM") || die "Can't access login credentials";

   my $db= ;

   my $host= ;

   my $userid= ;

   my $passwd= ;
   chomp($db);

   chomp($host);

   chomp($userid);

   chomp($passwd);
# or, you may comment out the above nine lines - starting with open(PW...

# and just put in your database connection information below

# and un-comment these lines

# but I don't like putting my connection information in a Perl script

#   my $db = "database_name";

#   my $host = "host_name";

#   my $userid = "user_name";

#   my $passwd = "password";
   my $connectionInfo="dbi:mysql:$db;$host";

   close(PW);
   # make connection to database

   my $l_dbh = DBI->connect($connectionInfo,$userid,$passwd);

   return $l_dbh;

}




The edit.pl script will create the web page for you to make your changes.  After you have made the changes, the form will call another Perl script named “edit_save.pl” to apply those changes to the database.  Here is the edit_save.pl script:


#!/usr/bin/perl
#--------------------------------------------------------------------------

# edit_save.pl

#--------------------------------------------------------------------------
use DBI;

use DBD::mysql;

use CGI qw(:standard);

use LWP::UserAgent;
my $Database = "address";
read(STDIN, $buffer, $ENV{'CONTENT_LENGTH'});
$buffer =~ tr/+/ /;

$buffer =~ s/\r/ /g;

$buffer =~ s/'/ /g;

$buffer =~ s/\n/ /g;

$buffer =~ s/%([a-fA-F0-9][a-fA-F0-9])/pack("C",hex($1))/eg;

$buffer =~ s// /g;

$buffer =~ tr/\\|[|]|<|!|"|$|{|}|*|#|'|>|||;|%/ /;
@pairs = split(/&/,$buffer);

foreach $pair(@pairs){

($key,$value)=split(/=/,$pair);

$formdata{$key}.="$value";

}
$id = $formdata{'id'};

$name_first = $formdata{'name_first'};

$name_last = $formdata{'name_last'};

$address_01 = $formdata{'address_01'};

$address_02 = $formdata{'address_02'};

$address_city = $formdata{'address_city'};

$address_state = $formdata{'address_state'};

$address_postal_code = $formdata{'address_postal_code'};
	$dbh = ConnectToMySql($Database);
	$query = "update address set name_first = '$name_first', name_last = '$name_last', address_01 = '$address_01', address_02 = '$address_02', address_city = '$address_city', address_state = '$address_state', address_postal_code = '$address_postal_code' where serial = '$id'"; 
    $sth = $dbh->prepare($query);
    $sth->execute();
    $dbh->disconnect;
print header;

print <








Scripting MySQL - Address Book - Edit Saved






My Addresses - Dashboard
The following information was updated:











Name First   $name_first
Name Last   $name_last
Address 1   $address_01
Address 2   $address_02
City   $address_city
State   $address_state
Zip   $address_postal_code

HTML

# leave the above line to the left of the page		
exit;
# more on what I am doing with the accessSM file may be found at:

# http://scriptingmysql.wordpress.com/2011/07/27/connecting-to-mysql-with-perl/
#----------------------------------------------------------------------

sub ConnectToMySql {

#----------------------------------------------------------------------
   my ($db) = @_;
   open(PW, "<..\/accessSM") || die "Can't access login credentials";

   my $db= ;

   my $host= ;

   my $userid= ;

   my $passwd= ;
   chomp($db);

   chomp($host);

   chomp($userid);

   chomp($passwd);
# or, you may comment out the above nine lines - starting with open(PW...

# and just put in your database connection information below

# and un-comment these lines

# but I don't like putting my connection information in a Perl script

#   my $db = "database_name";

#   my $host = "host_name";

#   my $userid = "user_name";

#   my $passwd = "password";
   my $connectionInfo="dbi:mysql:$db;$host";

   close(PW);
   # make connection to database

   my $l_dbh = DBI->connect($connectionInfo,$userid,$passwd);

   return $l_dbh;
}




After you have edited your record, the edited results web page will look like this:

You can then click on the Dashboard link to go back to the main dashboard page.
If you want to delete a record, from the dashboard.pl page, you simply click on the red X image on the row that you want to delete.  For example, when you click on the X for the serial number of “1″, you will see this link:


http://192.168.1.2/cgi-bin/scripting_mysql/delete.pl?id=1




This link takes you to the delete.pl page, where you are asked if you want to confirm the deletion, or you may cancel and go back to the dashboard.  When you click on an X to delete a row, you will see a page like this:

And here is the delete.pl script:


#!/usr/bin/perl
#--------------------------------------------------------------------------

# delete.pl

#--------------------------------------------------------------------------
use DBI;

use DBD::mysql;

use CGI qw(:standard);
$query = new CGI;

$id = $query->param("id");
my $Database = "scripting_mysql";
print header;

print <








Scripting MySQL - Address Book - Delete Record





HTML
# grab the information for this id
	$dbh = ConnectToMySql($Database);
	$query = "select serial, name_first, name_last, address_01, address_02, address_city, address_state, address_postal_code FROM address where serial = '$id'";
    $sth = $dbh->prepare($query);
    $sth->execute();
    $dbh->disconnect;
# list all of the addresses in a table

# be sure that the link for add.html is the correct path according to your system

print "My Addresses - Dashboard";

print "Are you sure you want to delete this record?
";

print "
";
          while (@data = $sth->fetchrow_array()) {

			$id = $data[0];

			$name_first = $data[1];

			$name_last = $data[2];

			$address_01 = $data[3];

			$address_02 = $data[4];

			$address_city = $data[5];

			$address_state = $data[6];

			$address_postal_code = $data[7];

			}
print <














Name First   $name_first
Name Last   $name_last
Address 1   $address_01
Address 2   $address_02
City   $address_city
State   $address_state
Zip   $address_postal_code





HTML

# leave the above line to the left of the page		
exit;
# more on what I am doing with the accessSM file may be found at:

# http://scriptingmysql.wordpress.com/2011/07/27/connecting-to-mysql-with-perl/
#----------------------------------------------------------------------

sub ConnectToMySql {

#----------------------------------------------------------------------
   my ($db) = @_;
   open(PW, "<..\/accessSM") || die "Can't access login credentials";

   my $db= ;

   my $host= ;

   my $userid= ;

   my $passwd= ;
   chomp($db);

   chomp($host);

   chomp($userid);

   chomp($passwd);
# or, you may comment out the above nine lines - starting with open(PW...

# and just put in your database connection information below

# and un-comment these lines

# but I don't like putting my connection information in a Perl script

#   my $db = "database_name";

#   my $host = "host_name";

#   my $userid = "user_name";

#   my $passwd = "password";
   my $connectionInfo="dbi:mysql:$db;$host";

   close(PW);
   # make connection to database

   my $l_dbh = DBI->connect($connectionInfo,$userid,$passwd);

   return $l_dbh;

}




If you choose to not delete the record, you are simply taken back to the Dashboard.  If you choose to delete the record, then the delete-confirm.pl script is executed, and the record is deleted.  Here is the confirmation web page for a deletion:

And here is the delete_confirm.pl script:


#!/usr/bin/perl
#--------------------------------------------------------------------------

# delete_confirm.pl

#--------------------------------------------------------------------------
use DBI;

use DBD::mysql;

use CGI qw(:standard);
$query = new CGI;

$id = $query->param("id");
my $Database = "scripting_mysql";
print header;

print <








Scripting MySQL - Address Book - Delete Record Confirmed





Are you sure you want to delete this record?
HTML
# grab the information for this id
	$dbh = ConnectToMySql($Database);
	$query = "select serial, name_first, name_last, address_01, address_02, address_city, address_state, address_postal_code FROM address where serial = '$id'";
    $sth = $dbh->prepare($query);
    $sth->execute();
    $dbh->disconnect;
# list all of the addresses in a table

# be sure that the link for add.html is the correct path according to your system

print "My Addresses - Dashboard";

print "The following information was deleted.
";

print "
";
          while (@data = $sth->fetchrow_array()) {

			$id = $data[0];

			$name_first = $data[1];

			$name_last = $data[2];

			$address_01 = $data[3];

			$address_02 = $data[4];

			$address_city = $data[5];

			$address_state = $data[6];

			$address_postal_code = $data[7];

			}
print <









Name First   $name_first
Name Last   $name_last
Address 1   $address_01
Address 2   $address_02
City   $address_city
State   $address_state
Zip   $address_postal_code




HTML

# leave the above line to the left of the page		
# here we delete the record

	$dbh = ConnectToMySql($Database);
	$query = "delete from address where serial = '$id'";
    $sth = $dbh->prepare($query);
    $sth->execute();
	$dbh->disconnect;
exit;
# more on what I am doing with the accessSM file may be found at:

# http://scriptingmysql.wordpress.com/2011/07/27/connecting-to-mysql-with-perl/
#----------------------------------------------------------------------

sub ConnectToMySql {

#----------------------------------------------------------------------
   my ($db) = @_;
   open(PW, "<..\/accessSM") || die "Can't access login credentials";

   my $db= ;

   my $host= ;

   my $userid= ;

   my $passwd= ;
   chomp($db);

   chomp($host);

   chomp($userid);

   chomp($passwd);
# or, you may comment out the above nine lines - starting with open(PW...

# and just put in your database connection information below

# and un-comment these lines

# but I don't like putting my connection information in a Perl script

#   my $db = "database_name";

#   my $host = "host_name";

#   my $userid = "user_name";

#   my $passwd = "password";
   my $connectionInfo="dbi:mysql:$db;$host";

   close(PW);
   # make connection to database

   my $l_dbh = DBI->connect($connectionInfo,$userid,$passwd);

   return $l_dbh;
}




One alternative to deleting the record would be to include a separate database field called “active”, set the field value to “yes” for all current records (if you have already imported records) and set the default value for this field to “yes” for any new records.  Then, instead of deleting the information, you simply set “active” to “no”.  So when you retrieve your list on the dashboard.pl page, you will only look for records where “active” is equal to “yes”.  You will still have the “deleted” information available to you in the database.  You could even duplicate the dashboard.pl script to have a “delete” page that displays the records that have been deleted.
To use the “active” option, you will need to make the following changes:
You would need to add this line in your create table SQL statement:  (I would insert the line after the “serial” field in your create table statement)


  `active` varchar(3) NOT NULL DEFAULT 'yes',




Then, in your dashboard.pl script, you would simply change the $query to:


$query = "select serial, name_first, name_last, address_01, address_02, address_city, address_state, address_postal_code FROM address where active = 'yes' order by serial";




For your add.pl and add_entry.pl scripts. you don’t need to do anything as the “active” field has a default value of “yes” for new records.
For your edit.pl and edit_save.pl scripts, you don’t need to do anything as you aren’t changing the “active” field.
And in your delete_confirm.pl script, instead of deleting the record, simply change the $query = “delete from address where serial = ‘$id’”; to:


$query = "update address set active = 'no' where serial = '$id'";




With these scripts, you should be able to install everything and have it running in a few minutes.  And, if my lawyer friend gets tired of spending all day at the courthouse, he could always learn a new skill.
 

—————————————–






Tony Darnell is a Principal Sales Consultant for MySQL, a division of Oracle, Inc.  MySQL is the world’s most popular open-source database program.


Tony may be reached at info [at] ScriptingMySQL.com and on LinkedIn.




         


Checking on the Progress of Large DML Commands in MySQL Using Perl – Part Two
Tony Darnell — Fri, 04 Nov 2011 11:01:33 +0000
Part Two of Two:  Checking on database activity when running a large DML (Data Manipulation Language) statement – such as INSERT, DELETE, UPDATE or SELECT.
Part Two:  Monitoring the activity via Perl and SHOW ENGINE INNODB STATUS.  (part of the InnoDB Monitors)

In part one, I showed you how to use a Perl script to insert a million rows of dummy data into a table.  I needed a large database in order to test a Perl script that I would use to monitor the activity when running a large DML statement.
The original reason for creating both of these scripts was to find a quick way to see if a large DML statement was actually being executed.  A customer was performing some modifications on tables with tens of millions of rows, and they wanted to know if they were making any progress.  Since the customer was using the InnoDB storage engine, I thought of a way that you could check on the progress – but only given the fact that nothing else was happening in the database (more on this reason later).
With InnoDB, you can issue the SHOW ENGINE INNODB STATUS and you will get a wealth of information.  I am not going to list any of it here as it would consume too much space.  However, towards the bottom of the output, you will notice under “ROW OPERATIONS” a line that contains the words “Number of rows inserted…”:



--------------

ROW OPERATIONS

--------------

0 queries inside InnoDB, 0 queries in queue

1 read views open inside InnoDB

Main thread id 2960240640, state: waiting for server activity

Number of rows inserted 1202598, updated 97249, deleted 806, read 56448551

0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s




The above line gives you a quick snapshot of how many rows have been inserted, updated, deleted or selected (read).  So, by invoking the SHOW ENGINE INNODB STATUS command, you can figure what database activity is occurring. But, if you invoke this command while the database is being used for other purposes, it will be difficult to figure out your progress on any of the four values shown.

So, all I needed to do was to write a quick Perl script to monitor the changes to that particular line of output, and then I could figure out if a certain statement was indeed being executed.  My original thinking was that I needed a database with a lot of records in it.  So, I decide to first create a dummy table and then shove a million records into it.  But I figured out that by creating a database with a million rows, I could simply just monitor this activity versus issuing a command on the database once it had been built.
I created a quick Perl script that issues the SHOW ENGINE INNODB STATUS command every X number of seconds.  It then looks for the output total for whatever I want to track – either inserts, updates, deletes or selects (reads).  And yes, you could just do this manually, but if you have a command that you want to run overnight, you could execute this script and at least see what happened when you return in the morning.

One caveat – SHOW ENGINE INNODB STATUS does not correspond to a particular point in time, so it may not be consistent – as to ensure consistency would require a global lock which would consume too many resources.  But it is good enough for what we want to do here.

The script will check the status, and then depending upon what variable (inserts, updates, deletes, selects) you want to search for, it will show you the total for that variable as it increases.  I didn’t take time to add a lot of information about how long it would take to complete the original command, as this would be just a wild guess, given the fact that other things could be happening with the database.  You will need to change the variable that you want to look for, how many rows you are expecting to change, how many seconds you want it to refresh, and your database information:


#!/usr/bin/perl -w
# DBI is the standard database interface for Perl

# DBD is the Perl module that we use to connect to the MySQL database

use DBI;

use DBD::mysql;

# use CGI for forms

#use CGI qw(:standard);
$start = '';

$begin = '';

$now = 1;
# How many rows are being changed?

# (whole numbers only)

$rows = 200000;

# how long do you want it to sleep? (seconds)

# you can modify this to be a longer time interval than 60 seconds

# which is advisable, as the SHOW ENGINE query will consume resources

$sleep_time = 60;

# what action do you want to monitor?

# based upon this output:  Number of rows inserted xxxxx, updated xxxxx, deleted xxxxx, read xxxxx

# (use the word to the right of the equal sign)

# inserts = inserted

# update = updates

# deletes = deleted

# selects = read

$action = "inserted";

$Database = "scripting_mysql";

     $dbh = ConnectToMySql($Database);

     $query = "SHOW ENGINE INNODB STATUS";

     $sth = $dbh->prepare($query);

     $sth->execute();
$data = $sth->fetchrow_array();
@all_rows = split("\\n",$data);
foreach (@all_rows) {
     if ($_ =~ "Number of rows")
     {

          #print "Line $_ \n";

     

          @total = split (" ", $_);

          

          # viewing this output from SHOW ENGINE INNODB STATUS

          # Number of rows inserted 912268, updated 96931, deleted 806, read 52052215

          if ($action =~ "inserted") { $column = 4 }

          if ($action =~ "updated") { $column = 6 }

          if ($action =~ "deleted") { $column = 8 }

          if ($action =~ "read") { $column = 10 }
               if ($_ =~ "Number of rows ")

               {

                    $total[$column] =~ s/,//;

                    $begin = $total[$column];

               }               

     }

}
print "Execute your SQL statement, and then press enter/return: ";

$start = <>;
print "Beginning with $begin $action:\n";
while ($now < $rows) {
$print_date_time = &get_date_time;
     $dbh = ConnectToMySql($Database);

     $query = "SHOW ENGINE INNODB STATUS";

     $sth = $dbh->prepare($query);

     $sth->execute();
     $data = $sth->fetchrow_array();
     @all_rows = split("\n",$data);
     foreach (@all_rows) {
               @total = split (" ", $_);

          

          if ($action =~ "inserted") { $column = 4 }

          if ($action =~ "updated") { $column = 6 }

          if ($action =~ "deleted") { $column = 8 }

          if ($action =~ "read") { $column = 10 }

          

               if ($_ =~ "Number of rows ")

               {

                    $total[$column] =~ s/,//;

                    $now = $total[$column];

                    $now = $now - $begin;

                    $progress = $now / $rows;

                    $progress = $progress * 100;

                    print "$print_date_time - $progress\% changed - $now out of $rows.\n";

               }

}
# you can modify this to be a longer time interval than 60 seconds

# which is advisable, as the SHOW ENGINE query will consume resources

sleep $sleep_time;
}
     $sth->finish;

     $dbh->disconnect;
# from Connecting to MySQL with Perl

#----------------------------------------------------------------------

sub ConnectToMySql {

#----------------------------------------------------------------------
     my ($db) = @_;
     open(ACCESS_INFO, "<..\/accessBLOG") || die "Can't access login credentials";

     my $database = ;

     my $host = ;

     my $userid = ;

     my $passwd = ;

# the chomp() function will remove any newline character from the end of a string

chomp ($database, $host, $userid, $passwd);

#print "
$database $host $userid $passwd 
";
     my $connectionInfo="dbi:mysql:$database;$host";

     close(ACCESS_INFO);
     # make connection to database

     my $l_dbh = DBI->connect($connectionInfo,$userid,$passwd);

     return $l_dbh;
}
# ----------------------------------------------------------------------------------

sub get_date_time {

# ----------------------------------------------------------------------------------
     my ($sec,$min,$hour,$mday,$mon,$year) = localtime time;
     $year = $year + 1900;

     $mon = $mon + 1;
     # add a zero if the value is less than 10
     if ($sec < 10) { $sec = "0$sec"; }

     if ($min < 10) { $min = "0$min"; }

     if ($hour < 10) { $hour = "0$hour"; }

     if ($mday < 10) { $mday = "0$mday"; }

     if ($mon < 10) { $mon = "0$mon"; }

     if ($year < 10) { $year = "0$year"; }
     $DateTime = "$year-$mon-$mday $hour:$min:$sec";

     return $DateTime
}

#----------------------------------------------------------------------




Here is the output from the Perl script.  For this example, I was assuming that I would be inserting 200,000 rows.


sh-3.2# /usr/bin/perl monitor.pl

Execute your SQL statement, and then press enter/return:

Beginning with 1196128 inserted:

2011-11-01 17:48:06 - 0.029% changed - 58 out of 200000.

2011-11-01 17:48:16 - 0.464% changed - 928 out of 200000.

2011-11-01 17:48:26 - 0.8645% changed - 1729 out of 200000.

2011-11-01 17:48:36 - 1.259% changed - 2518 out of 200000.

2011-11-01 17:48:46 - 1.628% changed - 3256 out of 200000.

2011-11-01 17:48:56 - 2.0045% changed - 4009 out of 200000.

2011-11-01 17:49:06 - 2.3845% changed - 4769 out of 200000.

2011-11-01 17:49:16 - 2.7685% changed - 5537 out of 200000.

2011-11-01 17:49:26 - 3.1415% changed - 6283 out of 200000.




This script was just a quick hack.  Another way to potentially see if your script is still running is to check out the “TRANSACTIONS” section of the SHOW ENGINE INNODB STATUS output.  In this example below, you can see the insert statement that the Perl script was performing during part one (where I was inserting the 1,000,000 rows).  However, since the database that I was using doesn’t have a lot of activity on it, I was able to see the insert statement multiple times when I resubmitted the SHOW ENGINE INNODB STATUS command.  If you have a database with a lot of transactions running, you might not be able to see your statement in the output.  



------------

TRANSACTIONS

------------

Trx id counter 1818DA1

Purge done for trx's n:o < 1817E3F undo n:o < 0

History list length 3830

LIST OF TRANSACTIONS FOR EACH SESSION:

---TRANSACTION 1818DA0, not started, OS thread id 2961469440 flushing log

mysql tables in use 1, locked 1

MySQL thread id 335232, query id 2818136 192.168.1.2 WebUser query end

insert into test_large (name_first, name_last, address_01, address_02, city, state, zip) values ('1BPnJiuWsyajA4b3SH7OjS4BFJgedK','JPuJ4xh_QEbNokZZGlpcHEHKLk2W__','24IT20mW0moAwWmoYTMOwsv44yRL9mAMoo0mLyLoGoTjaW78O6','6zX2j4PLShQ_IfiOzd0LbTVi8ZaYGk3_6LIafpnFfdFL7kRMTa','mbENuc0kHQz9NLGkn5iy','xR','81941')

---TRANSACTION 0, not started, OS thread id 2960855040

MySQL thread id 326621, query id 2818137 localhost root

SHOW ENGINE INNODB STATUS

---TRANSACTION 1815A3B, not started, OS thread id 2961059840

MySQL thread id 322078, query id 2778540 192.168.1.5 WebUser

---TRANSACTION 1815A32, not started, OS thread id 2960650240

MySQL thread id 319917, query id 2778444 192.168.1.5 WebUser

---TRANSACTION 0, not started, OS thread id 2962083840

MySQL thread id 319845, query id 2771552 192.168.1.5 WebUser

---TRANSACTION 0, not started, OS thread id 2963107840

MySQL thread id 43198, query id 2817803 localhost 127.0.0.1 WebUser




I am sure that there are better ways of monitoring large DML statements, and this will only work for InnoDB tables.  If you try the same command for MyISAM, SHOW ENGINE MyISAM STATUS, the output is blank:



mysql> SHOW ENGINE MyISAM STATUS;

Empty set (0.00 sec)




If you know of a better or different way to do this, please add your thoughts in the comment section below.
 

—————————————–






Tony Darnell is a Principal Sales Consultant for MySQL, a division of Oracle, Inc.  MySQL is the world’s most popular open-source database program.


Tony may be reached at info [at] ScriptingMySQL.com and on LinkedIn.




         


Checking on the Progress of Large DML Commands in MySQL Using Perl – Part One
Tony Darnell — Wed, 02 Nov 2011 11:45:56 +0000
Part One of Two:  Checking on database activity when running a large DML (Data Manipulation Language) statement – such as INSERT, DELETE, UPDATE or SELECT.
Part One:  Inserting a million rows into a database.
A friend of mine had asked a question – “Is there any way you can track how far you have advanced in a slow-moving ALTER or OPTIMIZE statement?”.  A customer was performing some modifications on a database with tens of millions of rows, and they wanted to be able to see if the command was making any progress.
Since the customer was using the InnoDB storage engine, I thought of a way that you could check on the progress – but only given the fact that nothing else (major) was happening in the database (more on this reason later).
With InnoDB, you can issue the SHOW ENGINE INNODB STATUS (part of the InnoDB Monitors) and you will get a wealth of information.  I am not going to list any of the output from the command here as it would consume too much space.  However, towards the bottom of the output from the command, you will notice under “ROW OPERATIONS” a line that contains the words “Number of rows inserted…”:



--------------

ROW OPERATIONS

--------------

0 queries inside InnoDB, 0 queries in queue

1 read views open inside InnoDB

Main thread id 2960240640, state: waiting for server activity

Number of rows inserted 1202598, updated 97249, deleted 806, read 56448551

0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s




The above line starting with “Number of rows…” gives you a quick snapshot of how many rows have been inserted, updated, deleted or selected (read) up to that point in time.  So, by invoking the SHOW ENGINE INNODB STATUS command, you can figure what database activity has occurred since the last time you issued the command. But, if you invoke this command while the database is being used for other purposes, it will be difficult to figure out your progress on any of the four values shown, as the stats aren’t just for your activity.

So, all I needed to do was to write a quick Perl script to monitor that particular line of output, and then I could figure out if a certain statement was indeed being executed, and I could monitor the progress (assuming nothing else was happening on the database).  The problem that I had was that I didn’t have a database with a lot of records in it.  But, I did have a database server that didn’t have much activity.  So, in part one, I will show you how to create a dummy table and then shove a million records into it.  In part two (my next blog entry), I will show you the Perl script to monitor the changes to the output from the SHOW ENGINE INNODB STATUS command. (And yes, you can just run the SHOW ENGINE INNODB STATUS command manually over and over again, but what fun would that be?)

First I created a dummy table named “test_large”:


CREATE TABLE `test_large` (

  `id` int(9) NOT NULL AUTO_INCREMENT,

  `name_first` varchar(30) NOT NULL,

  `name_last` varchar(30) NOT NULL,

  `address_01` varchar(50) NOT NULL,

  `address_02` varchar(50) NOT NULL,

  `city` varchar(20) NOT NULL,

  `state` varchar(2) NOT NULL,

  `zip` varchar(5) NOT NULL,

  PRIMARY KEY (`id`)

) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=latin1




Next, I created a Perl script that would create dummy/fake data and insert it into the table.  This script uses a subroutine named “ConnectToMySql” that I covered in an earlier post – Connecting to MySQL with Perl.  It also uses a subroutine to create fake alpha/numeric data (generate_random_string) and one that creates fake numeric data (generate_random_numbers).  A third subroutine (get_date_time) is used to display the date and time (I use this subroutine when I want to insert a date and time into a MySQL database).
The script is fairly easy to use.  You only need to modify the $total_rows variable to the number of rows that you want to insert, as well as the information about your particular database.  I didn’t include anything in the script to show you the progress of this script, but you could certainly add that if you wish.


#!/usr/bin/perl -w
# DBI is the standard database interface for Perl

# DBD is the Perl module that we use to connect to the MySQL database

use DBI;

use DBD::mysql;
# how many total dummy rows of data do you need?

# (must be an integer)

$total_rows = 10000000;

# make sure that $total_rows is an integer

if ($total_rows =~ /\D/) {

     print "Total rows is not an integer = value = $total_rows.\n";

     exit;

}
# keep count of how many rows we have inserted

$total_completed = 0;
# your database name

$Database = "scripting_mysql";

# get the current system date and time

$print_date_time = &get_date_time;
# print the starting time (optional)

print "Starting $print_date_time\n";
# keep looping until we reach the total number of $total_rows

while ($total_completed < $total_rows) {
     # create dummy data

     $fake_data_01 = &generate_random_string(30);

     $fake_data_02 = &generate_random_string(30);

     $fake_data_03 = &generate_random_string(50);

     $fake_data_04 = &generate_random_string(50);

     $fake_data_05 = &generate_random_string(20);

     $fake_data_06 = &generate_random_string(2);

     $fake_data_07 = &generate_random_numbers(5);
     $dbh = ConnectToMySql($Database);
     $query = "insert into test_large (name_first, name_last, address_01, address_02, city, state, zip) values (?,?,?,?,?,?,?)";

     

     $sth = $dbh->prepare($query);
     $sth->execute($fake_data_01, $fake_data_02, $fake_data_03, $fake_data_04, $fake_data_05, $fake_data_06, $fake_data_07);

     $sth->finish;

          

     $total_completed++;

}
     $dbh->disconnect;
$print_date_time = &get_date_time;
# print our finishing time

print "Finishing $print_date_time\n";

exit;
# ----------------------------------------------------------------------------------

# subroutines

# ----------------------------------------------------------------------------------
# ----------------------------------------------------------------------------------

sub get_date_time {

# ----------------------------------------------------------------------------------
     my ($sec,$min,$hour,$mday,$mon,$year) = localtime time;
     $year = $year + 1900;

     $mon = $mon + 1;
     # add a zero if the value is less than 10
     if ($sec < 10) { $sec = "0$sec"; }

     if ($min < 10) { $min = "0$min"; }

     if ($hour < 10) { $hour = "0$hour"; }

     if ($mday < 10) { $mday = "0$mday"; }

     if ($mon < 10) { $mon = "0$mon"; }

     if ($year < 10) { $year = "0$year"; }
     $DateTime = "$year-$mon-$mday $hour:$min:$sec";

     return $DateTime
}

#----------------------------------------------------------------------
# from Connecting to MySQL with Perl

#----------------------------------------------------------------------

sub ConnectToMySql {

#----------------------------------------------------------------------
     my ($db) = @_;
     open(ACCESS_INFO, "<..\/accessBLOG") || die "Can't access login credentials";

     my $database = ;

     my $host = ;

     my $userid = ;

     my $passwd = ;

# the chomp() function will remove any newline character from the end of a string

chomp ($database, $host, $userid, $passwd);

#print "
$database $host $userid $passwd 
";
     my $connectionInfo="dbi:mysql:$database;$host";

     close(ACCESS_INFO);
     # make connection to database

     my $l_dbh = DBI->connect($connectionInfo,$userid,$passwd);

     return $l_dbh;
}

#----------------------------------------------------------------------
#----------------------------------------------------------------------

sub generate_random_string {

#----------------------------------------------------------------------
     my $length_of_randomstring=shift;# the length of

                # the random string to generate
     my @chars=('a'..'z','A'..'Z','0'..'9','_');

     my $random_string;

     foreach (1..$length_of_randomstring)

     {

          # rand @chars will generate a random

          # number between 0 and scalar @chars

          $random_string.=$chars[rand @chars];

     }

     return $random_string;

}

#----------------------------------------------------------------------
#----------------------------------------------------------------------

sub generate_random_numbers {

#----------------------------------------------------------------------
     my $length_of_randomstring=shift;# the length of

                # the random string to generate
     my @chars=('0'..'9');

     my $random_string;

     foreach (1..$length_of_randomstring)

     {

          # rand @chars will generate a random

          # number between 0 and scalar @chars

          $random_string.=$chars[rand @chars];

     }

     return $random_string;

}

#----------------------------------------------------------------------




This script took about three hours to insert a million rows into the new database.  And since this was a new table, every so often I would just check to see how many rows were in the table with a simple select command:


mysql> select count(*) from test_large;

+----------+

| count(*) |

+----------+

|    98187 |

+----------+

1 row in set (0.00 sec)




In my next post, I will show you the Perl script that I used to monitor the SHOW ENGINE INNODB STATUS results.

 

—————————————–






Tony Darnell is a Principal Sales Consultant for MySQL, a division of Oracle, Inc.  MySQL is the world’s most popular open-source database program.


Tony may be reached at info [at] ScriptingMySQL.com and on LinkedIn.




         


Using MySQL, Perl and jQuery to Auto-Populate a Form Field on a Web Page
Tony Darnell — Fri, 28 Oct 2011 11:10:58 +0000
If you have ever built a form on a web page, you might have used a drop-down menu to display the choices available for a particular field.  With a drop-down menu, you restrict the choices a user may select so that the user doesn’t enter invalid data (among other reasons).  If a user misspells an entry, then a subsequent search for that value would not produce a found result.
A friend of mine who runs an online forum sent me an email about a problem he was having.  He was trying to modify an existing registration web page using jQuery to auto-populate the state names, and then pass the state abbreviation back to his MySQL database.  Believe it or not, he was actually having problems with people knowing their own state abbreviation.  He had searched and found an example of what he wanted to do, but he couldn’t get it to work.  So, I took the example  that he found and figured out what he was doing wrong.
I had first suggested that he just hard-code the list of states and their abbreviations in the HTML code.  But, he said that if he figured out how to use jQuery with his state abbreviation problem, he would also use it on other parts of his web page where he had dynamic data.  With drop-down menus, it is fairly easy to create a list of static data to use in the HTML form.  But, in his case, what happens if the data is dynamic, or if there are too many items to list in a drop-down menu?  If you have ever had to include your country in a form (and you live in the US), it is always a pain to have to scroll down to the bottom of the long list to find “United States”.
An easy way to create a dynamic list (or even a long static list) is to use jQuery to access your data from a MySQL database, and to build an auto-populated field in your form.  When the user starts typing their entry, jQuery will search the MySQL database for a list of “like” matches, and then return a list of all of the matching results for that field.  
For this example, we will be using jQuery, Perl and a MySQL database to auto-populate a “state” field in a form.  Once a user has typed three characters in the state field box, a list of matching states will appear in a drop-down menu.  It will also return the state abbreviation and state id (code), which can then be saved in the MySQL database after the form has been submitted.  In this picture from our example, the user has typed “new”, and returned all states that match that string:

First, let’s start by creating the MySQL database that we will use.  Here is the SQL code to create the database, which is named “states”.  The table will contain the state name, the state abbreviation, and an ID number.


CREATE TABLE `states` (

  `id` int(5) NOT NULL AUTO_INCREMENT,

  `state` varchar(50) NOT NULL,

  `abbrev` char(2) NOT NULL,

  PRIMARY KEY (`id`)

) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=latin1




Next, you will want to import values into the database.  You could do this by importing a .csv file containing this information into MySQL, but to make it quicker for this example, here is the SQL:


insert into states (state, abbrev) values ('Alabama', 'AL');

insert into states (state, abbrev) values ('Alaska', 'AK');

insert into states (state, abbrev) values ('Arizona', 'AZ');

insert into states (state, abbrev) values ('Arkansas', 'AR');

insert into states (state, abbrev) values ('California', 'CA');

insert into states (state, abbrev) values ('Colorado', 'CO');

insert into states (state, abbrev) values ('Connecticut', 'CT');

insert into states (state, abbrev) values ('Delaware', 'DE');

insert into states (state, abbrev) values ('District of Columbia', 'DC');

insert into states (state, abbrev) values ('Florida', 'FL');

insert into states (state, abbrev) values ('Georgia', 'GA');

insert into states (state, abbrev) values ('Hawaii', 'HI');

insert into states (state, abbrev) values ('Idaho', 'ID');

insert into states (state, abbrev) values ('Illinois', 'IL');

insert into states (state, abbrev) values ('Indiana', 'IN');

insert into states (state, abbrev) values ('Iowa', 'IA');

insert into states (state, abbrev) values ('Kansas', 'KS');

insert into states (state, abbrev) values ('Kentucky', 'KY');

insert into states (state, abbrev) values ('Louisiana', 'LA');

insert into states (state, abbrev) values ('Maine', 'ME');

insert into states (state, abbrev) values ('Maryland', 'MD');

insert into states (state, abbrev) values ('Massachusetts', 'MA');

insert into states (state, abbrev) values ('Michigan', 'MI');

insert into states (state, abbrev) values ('Minnesota', 'MN');

insert into states (state, abbrev) values ('Mississippi', 'MS');

insert into states (state, abbrev) values ('Missouri', 'MO');

insert into states (state, abbrev) values ('Montana', 'MT');

insert into states (state, abbrev) values ('Nebraska', 'NE');

insert into states (state, abbrev) values ('Nevada', 'NV');

insert into states (state, abbrev) values ('New Hampshire', 'NH');

insert into states (state, abbrev) values ('New Jersey', 'NJ');

insert into states (state, abbrev) values ('New Mexico', 'NM');

insert into states (state, abbrev) values ('New York', 'NY');

insert into states (state, abbrev) values ('North Carolina', 'NC');

insert into states (state, abbrev) values ('North Dakota', 'ND');

insert into states (state, abbrev) values ('Ohio', 'OH');

insert into states (state, abbrev) values ('Oklahoma', 'OK');

insert into states (state, abbrev) values ('Oregon', 'OR');

insert into states (state, abbrev) values ('Pennsylvania', 'PA');

insert into states (state, abbrev) values ('Rhode Island', 'RI');

insert into states (state, abbrev) values ('South Carolina', 'SC');

insert into states (state, abbrev) values ('South Dakota', 'SD');

insert into states (state, abbrev) values ('Tennessee', 'TN');

insert into states (state, abbrev) values ('Texas', 'TX');

insert into states (state, abbrev) values ('Utah', 'UT');

insert into states (state, abbrev) values ('Vermont', 'VT');

insert into states (state, abbrev) values ('Virginia', 'VA');

insert into states (state, abbrev) values ('Washington', 'WA');

insert into states (state, abbrev) values ('West Virginia', 'WV');

insert into states (state, abbrev) values ('Wisconsin', 'WI');

insert into states (state, abbrev) values ('Wyoming', 'WY');

insert into states (state, abbrev) values ('American Samoa');

insert into states (state, abbrev) values ('Guam', 'GU');

insert into states (state, abbrev) values ('Northern Mariana Islands', 'MP');

insert into states (state, abbrev) values ('Puerto Rico', 'PR');




Now that we have the database, we will need to create the Perl script.  Here is the Perl script, and you will need to change the database variables to match your system:


#!/usr/bin/perl -w

use CGI;

use DBI;

use DBD::mysql;

use JSON;
# print the http header specifically for json

print "Content-type: application/json; charset=iso-8859-1\n\n"; 
# your database variables

my $database = "scripting_mysql";

my $host = "192.168.1.2";

my $port = "3306";

my $tablename = "states";

my $user = "root";

my $pass = "root_password";

my $cgi = CGI->new();

my $term = $cgi->param('term');
# mysql connection information

$dsn = "dbi:mysql:$database:$host:$port";
# open the database connection

$connect = DBI->connect($dsn, $user, $pass) || die print "Can't connect - error...";
# prepare the query

$query = $connect->prepare(qq{select id, state AS value, abbrev FROM states where state like ?;});
# execute the query

$query->execute('%'.$term.'%');
# obtain the results

while ( my $row = $query->fetchrow_hashref ){

    push @query_output, $row;

}
# close the database connection

$connect->disconnect();
# print the json output to be returned to the HTML page

print JSON::to_json(\@query_output);




For this example, the Perl script should be named states.pl.  When executed from the jQuery script in the web page, it will perform a search on the database where the search criteria is in a “like” statement, with a percent sign on both sides of the search term, like this:


# execute the query

$query->execute('%'.$term.'%');




You might want to change this query if you have a long data set where entering a few letters would bring up a result that isn’t what the user would probably want.  Here is an example of the same form where we are asking the user to enter their state name, and they enter “min” for Minnesota, but it also brings up Wyoming:

If you want to only search for the beginning of the word, remove the first percent sign, like this:


# execute the query

$query->execute($term.'%');




This would remove Wyoming from being populated as a choice:

Lastly, here is a simple web page that uses jQuery.  We are using the jQuery hosted by Google, and we are using one of their css templates.  You may delete their css template and use your own.  Also, you will need to change the path of your Perl script on this line:


source: "http://192.168.1.2/cgi-bin/mysql/jquery/state.pl"




And here is the HTML:





















State: 
 














In this example, once the user starts typing past three characters, a drop-down menu will appear with choices that match the letters being typed.  You can decrease/increase the total number of characters that the user will need to type before it queries the database.  This is done via this line in the jQuery code:


minLength: 3,




Every time that the person types a character past the initial three characters, the jQuery script will query the MySQL database.  This needs to be taken into consideration if you will have a lot of traffic on the pages where you are using jQuery.
The Perl script returns three values in json output – the id, state and abbreviation fields.  The values of these fields are place in the text boxes by the autocomplete jQuery function.
In this example, the abbreviation for the state field is returned and placed in the “abbrev” field in the form.  The value for the state id is also returned, but since the id field is a hidden field in the form, you won’t see the output/value – but it will be stored in the form.
If you want to see the value of the state id, simply change your web form to look like this:











State: 
 












You can now see the third field which will contain the state id:

When you enter “min” and then click on Minnesota, you will see the state ID (24) in the third field:

You could also use this to auto-populate city and state fields after someone enters their zip code.  Or you could even use this to auto-populate the shipping costs for an online order (I might tackle this one in a future post).  Hopefully you will be able to copy and paste this example and have a working jQuery form web page example in just a few minutes.
 

—————————————–






Tony Darnell is a Principal Sales Consultant for MySQL, a division of Oracle, Inc.  MySQL is the world’s most popular open-source database program.


Tony may be reached at info [at] ScriptingMySQL.com and on LinkedIn.




         


Using Perl, Cookies and MySQL to Record Web Site Visitor’s Data
Tony Darnell — Fri, 21 Oct 2011 18:26:29 +0000
I have built several web sites and applications in the past, and I have to admit that I am a data junkie.  If there is user data or any other data to be captured when someone uses my application or visits a web page, then I like to capture it (I can always delete it later).  Of course, you can use a third-party Javascript plug-in for web site analytics (which I also use), but I like to gather information myself.
Most of the pages that I have written use Perl – even for the home page when possible (yes, I am “old school”, and my lack of using modern Perl code is evident).  Each time a user clicks to go to another page, I like to capture as much information as possible about their visit.
Perl, PHP and other scripting languages have ways for you to capture the information about your web page visitor.  For Perl, this simple script will tell you a lot of information the visitor and your server.  In order for this script to work, you have to call it from a web browser.




#!/usr/bin/perl -w

use strict;

use CGI qw(:standard);

use CGI::Carp qw(warningsToBrowser fatalsToBrowser);
print header;

print start_html("Environment");
foreach my $key (sort(keys(%ENV))) {

    print "$key = $ENV{$key}
\n";

}
print end_html;





I took the above script and placed it in my webserver’s cgi-bin directory, made it executable, and opened it in my browser.  I added a “?test=1234567″ at the end of the script so that you could see the value of the REQUEST_URI variable.
Here is the web output:




DOCUMENT_ROOT = /Library/WebServer/Documents

GATEWAY_INTERFACE = CGI/1.1

HTTP_ACCEPT = text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

HTTP_ACCEPT_CHARSET = ISO-8859-1,utf-8;q=0.7,*;q=0.7

HTTP_ACCEPT_ENCODING = gzip, deflate

HTTP_ACCEPT_LANGUAGE = en-us,en;q=0.5

HTTP_CONNECTION = keep-alive

HTTP_COOKIE =

HTTP_HOST = 192.168.1.2

HTTP_USER_AGENT = Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:7.0.1) Gecko/20100101 Firefox/7.0.1

PATH = /usr/bin:/bin:/usr/sbin:/sbin

QUERY_STRING = test=1234567

REMOTE_ADDR = 192.168.1.5

REMOTE_PORT = 60398

REQUEST_METHOD = GET

REQUEST_URI = /cgi-bin/mysql/perl_env.pl?test=1234567

SCRIPT_FILENAME = /Library/WebServer/cgi-bin/mysql/perl_env.pl

SCRIPT_NAME = /cgi-bin/mysql/perl_env.pl

SERVER_ADDR = 192.168.1.2

SERVER_ADMIN = email_address@scriptingmysql.com

SERVER_NAME = 192.168.1.2

SERVER_PORT = 80

SERVER_PROTOCOL = HTTP/1.1

SERVER_SIGNATURE =

SERVER_SOFTWARE = Apache/2.2.17 (Unix) mod_ssl/2.2.17 OpenSSL/0.9.7l DAV/2 PHP/5.2.15





This is a lot of information, and you probably don't need or want most of it.  You will want to select information that you can actually or potentially use.  For example, if you want to capture their computer info, HTTP referral and remote IP address, in your Perl script. you would create variables in your Perl script like this:




$HTTP_USER_AGENT = $ENV{'HTTP_USER_AGENT'};

$REMOTE_ADDR = $ENV{'REMOTE_ADDR'};

$HTTP_REFERER = $ENV{'HTTP_REFERER'};




You can then insert this information into your MySQL database, along with an "action" of what they are doing on the page.  I create an activity table to record what the user is doing on various web pages.  If they are visiting your dashboard page, you might set their "activity" to "Dashboard Visit".  I also include a date and time stamp.  A sample activity table might look like this:




CREATE TABLE `activity` (

  `Customer_Serial_Number` varchar(15) NOT NULL,

  `Action` varchar(50) NOT NULL,

  `Date_Time` datetime NOT NULL,

  `Info` varchar(50) NOT NULL,

  `Notes` varchar(100) NOT NULL,

  `HTTP_USER_AGENT` varchar(150) NOT NULL

  `REMOTE_ADDR` varchar(15) NOT NULL

  `HTTP_REFERER` varchar(300) NOT NULL

) ENGINE=InnoDB DEFAULT CHARSET=utf8




If the web site requires a user name and password to login, I also use cookies - which is usually how I retrieve the Customer_Serial_Number and/or any other personal information that they have provided to me.  Or, if they fill out a "contact us" form, you can save their name and information in a cookie to be retrieved when they visit your web site later.  Just be sure to set the cookie expiration date to a date in the future, or to have the cookie never expire.
Here is a quick way to create a cookie in Perl.  In this example, we are creating two cookies, Customer Serial Number and Customer Email, both of which we have retrieved from MySQL after they logged into the site:




   # ===========

   # set cookie for Customer Serial Number

   # ===========

   $customer_serial_number = CGI::Cookie->new(-name => "CustSN",

      -value => $Customer_Serial_Number,

      -expires   => "+15m",

      -domain    => ".mydomain.com",

      -path      => "/"

   );
   # ===========

   # set cookie for Customer Email

   # ===========

   $Email = CGI::Cookie->new(-name => "CustE",

      -value => "$Customer_Email_Address",

      -expires   => "+1y",

      -domain    => ".mydomain.com",

      -path      => "/"

   );
# create the cookie for Customer Serial Number and Customer Email

$q = new CGI;

print $q->header(-cookie => [$customer_serial_number, $Email] );






Here are the definitions for the cookie variables (information from http://perldoc.perl.org/CGI/Cookie.html.




CSN - the name of the cookie.  You will use this name to retrieve the cookie later.

value - the value of the cookie - the value can be a scalar, an array reference, or a hash reference.

expires - how long you want the cookie to be available.  Here are some settings:

   - Cookie expires 10 seconds from now +10s

   - Cookie expires 10 minutes from now +10m

   - Cookie expires 10 hours from now +10h

   - Cookie expires 10 days from now +10d

   - Cookie expires 10 months from now +10M

   - Cookie expires 10 years from now +10y

   - to make it expire immediately, set it to a negative number, such as: -10M

domain - points to a domain name or to a fully qualified host name. If not specified, the cookie will be returned only to the web server that created it.

path - points to a partial URL on the current server. The cookie will be returned to all URLs beginning with the specified path. If not specified, it defaults to '/', which returns the cookie to all pages at your site.




There are a couple of other variables that you can use, so refer to the Perl cookie page for more information.
Deciding which information that you want to keep is up to you.  Then, you can simply create a MySQL table to store this information.  When a user registers on your site, you can store their email address in a cookie that never expires, so the next time they login, you can auto-populate the login form with their email address.
You can also redirect mobile users to a web page that is specific to their screen size.  For example, when I connect to this script with my iPhone, I can tell that the connection is from an iPhone via the HTTP_USER_AGENT:




HTTP_USER_AGENT = Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_2_10 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8E600 Safari/6533.18.5





However, my iPhone's remote IP address is almost useless:




REMOTE_ADDR = 10.1.10.10





So, your Perl script can then detect that this user is on an iPhone, and then it can redirect them with a meta refresh tag to a version of the web page specifically designed for the iPhone screen resolution.
One web site that I designed required a user to login to get past the home screen.  Once they logged in, I would grab their serial number from the MySQL database, and then create a cookie with the serial number value.  With each page they visited, I would send a few bits of information to my MySQL visitor database.
Here is part of the Perl script that I would use to record customer information:  (information on inserting data into MySQL via Perl may be found in an earlier post).




#!/usr/bin/perl -w

use CGI qw(:standard);

use CGI::Carp qw(warningsToBrowser fatalsToBrowser);
# ----------------------------------------------------------------------------------

# grab the environment variables

# ----------------------------------------------------------------------------------
$HTTP_USER_AGENT = $ENV{'HTTP_USER_AGENT'};

$REMOTE_ADDR = $ENV{'REMOTE_ADDR'};
# ----------------------------------------------------------------------------------

# fetch the cookie for the Customer Serial Number

# ----------------------------------------------------------------------------------
%cookies = CGI::Cookie->fetch;

if ($cookies{CustSN}) {

        $Customer_Serial_Name = $cookies{CustSN}->value;

}
# ----------------------------------------------------------------------------------

# grab date and time

# ----------------------------------------------------------------------------------
my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime time;
$year = $year + 1900;

$mon = $mon + 1;
# add a zero if the value is less than 10
if ($sec < 10) { $sec = "0$sec"; }

if ($min < 10) { $min = "0$min"; }

if ($hour < 10) { $hour = "0$hour"; }

if ($mday < 10) { $mday = "0$mday"; }

if ($mon < 10) { $mon = "0$mon"; }

if ($year < 10) { $year = "0$year"; }
$DateTime = "$year-$mon-$mday $hour:$min:$sec";
# ---------------------------------------------------------------------------------

# insert customer activity into database

# ---------------------------------------------------------------------------------
	$Action = "Dashboard Visit";
	$dbh = ConnectToMySql($Database);
	$query = "insert into Activity (Customer_Serial_Number, Action, Date_Time, HTTP_USER_AGENT, REMOTE_ADDR) values (?,?,?,?,?)";
	$sth = $dbh->prepare($query);
	$sth->execute('$Customer_Serial_Number', '$Action', '$DateTime', '$HTTP_USER_AGENT', '$REMOTE_ADDR');
# ---------------------------------------------------------------------------------

# ---------------------------------------------------------------------------------






Once you have enough customer data, you can figure out the paths that your customers are taking when visiting your site.  Do they login and go straight to the forum page?  Or if you have a web store, do they go straight to the clearance page?  Or are they logging in and then doing nothing?  If a customer never visits a certain page, what can you do to drive them to visit that page?  Third-party tools can provide good overall web site analytical information, but nothing beats being able to see one particular customer's activity on your own.
 

-----------------------------------------






Tony Darnell is a Principal Sales Consultant for MySQL, a division of Oracle, Inc.  MySQL is the world's most popular open-source database program.


Tony may be reached at info [at] ScriptingMySQL.com and on LinkedIn.




         


Using Java to Connect to MySQL Enterprise Monitor with Plugin for Connector/J
Tony Darnell — Fri, 14 Oct 2011 19:15:00 +0000
If you use MySQL for any number of databases, and if you are not using the MySQL Enterprise Monitor (MEM) to manage your databases, then you are missing out on a great DBA tool.
From http://dev.mysql.com/doc/mysql-monitor/2.3/en/mem-introduction.html:
“MySQL Enterprise Monitor is a companion product for MySQL Server that monitors your MySQL instances, notifies you of potential issues and problems, and advises you how to fix the issues. MySQL Enterprise Monitor can monitor all kinds of configurations, from a single MySQL instance that is important to your business, all the way up to a huge farm of database server machines powering a busy web site. “
The query analyzer function of the Enterprise Monitor helps you to analyze/tune your queries.  It can tell you which queries are being run the most often, which ones are taking the longest to run, it can even help you find queries that are returning a zero result set, and other query “violators”.
But the caveat is that in order to use the query analyzer function, you have to redirect your MySQL traffic through a proxy server (configured during the MEM installation).  Using the proxy server is an additional step that might be acceptible for testing on development servers, but it might not be a great idea in a production environment.
However, if you are using Java for your database connections, you can bypass the proxy server and send your query information directly to the MEM Service Manager.  This is accomplished with the MySQL Enterprise Plugin for Connector/J.
From http://dev.mysql.com/doc/mysql-monitor/2.3/en/mem-qanal-using-cj.html:
“The MySQL Enterprise Plugin for Connector/J lets you use the Query Analyzer to monitor MySQL queries from any application using the Connector/J JDBC driver. As described in Section 3.2, “Query Analyzer User Interface“, the Query Analyzer can help you locate queries that are inefficient or slow. Tuning such queries helps to shorten load times for web pages, and improves overall system responsiveness and scalability.”
As stated, the information about the queries is sent directly to the MySQL Enterprise Service Manager. Once you install the MySQL Enterprise Plugin for Connector/J and tweak your Java database connection string, query analysis becomes available for your Java applications.  And, the Enterprise Monitor can show you the location of the “offensive” Java code.  Here is an example, with the code location highlighted in blue:

There are a few things that you need to do before you can use this plugin.  The prerequisites are:
 - MySQL Connector/J version 5.1.12 or later.

 - JDK-1.5.0 or later.

 - MySQL Enterprise Service Manager version 2.2 or later.

 - The MySQL instance that the application uses must be monitored by a MySQL Enterprise monitoring agent. The mysql.inventory table must be accessible to the configured user for the Connector/J connection to get the UUID of the monitored server.

 - Apache Commons logging in the CLASSPATH of the application being analyzed. If you are not already using Commons Logging, modify the application’s CLASSPATH as described in the following section to point to the JAR file bundled with the MySQL Enterprise Monitor product.
I don’t write Java code, but I wanted to be able to test this functionality.  I found an example script on MySQL’s web site, and so I was able to modify it to send a test query to my installation of MEM.
I did run into one problem – I wasn’t setting my CLASSPATH variable correctly.  To solve this problem, I edited my .profile file for the root user, and added the following JAR file locations to my CLASSPATH:  (the colors are added for clarity)





export set CLASSPATH=/Applications/mysql/enterprise/monitor/apache-tomcat/webapps/ROOT/WEB-INF/lib/mysql-connector-java-commercial-5.1.17-bin.jar:/Applications/mysql/enterprise/monitor/apache-tomcat/webapps/ROOT/WEB-INF/lib/c-java-mysql-enterprise-plugin-1.0.0.61.jar:/Applications/mysql/enterprise/monitor/apache-tomcat/webapps/ROOT/WEB-INF/lib/required/commons-logging-1.1.1.jar:





I am running MySQL on a Mac Pro Server, so you will need to modify the file paths in your CLASSPATH to match your system.
After you have completed the prerequisites, you are ready to see if you can connect to the Service Manager with a query via Java.  Here is a sample script that you can use (you will need to modify it your system parameters):





import java.io.IOException;

import java.sql.*;
// Notice, do not import com.mysql.jdbc.*

// or you will have problems!
public class LoadDriver {

   public static void main(String[] args) {
         try {

            // The newInstance() call is a work around for some

            // broken Java implementations
            Class.forName("com.mysql.jdbc.Driver").newInstance();
            Connection conn = DriverManager.getConnection("jdbc:mysql://192.168.1.2/database_name?"+

               "user=root&password=root_password"+

               "&statementInterceptors=com.mysql.etools.jdbc.StatementPerformanceCounters"+

               "&serviceManagerUrl=http://192.168.1.2:18080/"+

               "&serviceManagerUser=agent"+

               "&serviceManagerPassword=agent_password");
            //Get a Statement object

               Statement stmt = conn.createStatement();

               ResultSet rs = stmt.executeQuery("SELECT sleep(120)");

               System.out.println("Closing connection");

      
               conn.close();

               } catch (Exception ex) {

               // handle the error

               ex.printStackTrace();

         }

      }

}





You will need to compile the script before running it.  Here are the results from compiling and then executing the script:  (since you are doing a SELECT SLEEP(120), the script will “sleep” for 120 seconds before finishing.)





macserver01:java root# javac LoadDriver.java

macserver01:java root# java LoadDriver

Fri Oct 14 14:08:57 EDT 2011 DEBUG: Using direct result set size methods.

Fri Oct 14 14:08:57 EDT 2011 DEBUG: Using batch-aware prepared sql extractor.

Fri Oct 14 14:08:57 EDT 2011 INFO: MySQL Enterprise Plugin for Connector/J version 1.0.0.61 started for MySQL instance 53e71576-026c-4798-961b-0025efc37db2.

Fri Oct 14 14:08:57 EDT 2011 DEBUG: GET http://192.168.1.2:18080/v2/rest/instance/mysql/StatementAnalysisSupport/53e71576-026c-4798-961b-0025efc37db2

HTTP/1.1 200 OK

Server: Apache-Coyote/1.1

Content-Type: application/json

Transfer-Encoding: chunked

Date: Fri, 14 Oct 2011 18:08:57 GMT
{

    "name": "53e71576-026c-4798-961b-0025efc37db2",

    "parent": "/instance/mysql/Server/53e71576-026c-4798-961b-0025efc37db2",

    "values":     {

        "auto_explain_min_exec_time_ms": 1,

        "capture_examples": "true",

        "capture_explain": "true",

        "enabled": "true",

        "frequency": null

    }

}

Fri Oct 14 14:08:57 EDT 2011 DEBUG: Configuration is: com.mysql.etools.jdbc.StatementAnalysisConfiguration@a14e84[captureExamples=true,captureExplains=true,enabled=true,explainPlanThresholdMicros=1000,usingDefaults=false]

Fri Oct 14 14:08:57 EDT 2011 DEBUG: GET http://192.168.1.2:18080/v2/rest/instance/mysql/StatementAnalysisSupport/53e71576-026c-4798-961b-0025efc37db2

HTTP/1.1 200 OK

Server: Apache-Coyote/1.1

Content-Type: application/json

Transfer-Encoding: chunked

Date: Fri, 14 Oct 2011 18:08:57 GMT

{

    "name": "53e71576-026c-4798-961b-0025efc37db2",

    "parent": "/instance/mysql/Server/53e71576-026c-4798-961b-0025efc37db2",

    "values":     {

        "auto_explain_min_exec_time_ms": 1,

        "capture_examples": "true",

        "capture_explain": "true",

        "enabled": "true",

        "frequency": null

    }

}

Fri Oct 14 14:08:57 EDT 2011 DEBUG: Configuration is: com.mysql.etools.jdbc.StatementAnalysisConfiguration@a14e84[captureExamples=true,captureExplains=true,enabled=true,explainPlanThresholdMicros=1000,usingDefaults=false]




And here are the results as show in the Query Analyzer tab of MySQL Enterprise Monitor:

By clicking on the SELECT sleep(?) link, I can then take a look at the query information:


For more complex queries, this tool really provides a wealth of information to help you tune your queries and increase the performance of your MySQL databases.  MySQL Enterprise Monitor has a wealth of great features.  For more information, see http://dev.mysql.com/doc/mysql-monitor/2.3/en/index.html.
 

—————————————–






Tony Darnell is a Principal Sales Consultant for MySQL, a division of Oracle, Inc.  MySQL is the world’s most popular open-source database program.


Tony may be reached at info [at] ScriptingMySQL.com and on LinkedIn.




         


Steve Jobs, you will be missed. Greatly.
Tony Darnell — Thu, 06 Oct 2011 12:00:24 +0000
I remember my first computer.  It was a TI-99/4A.  I bought it back in 1982 (I think), and it cost around $300.  The entire computer fit inside what looked like a really thick keyboard.  It had a slot on the right for cartridges, and I had a cassette tape drive that I used for backing up the BASIC computer programs that I wrote.  I thought that it was a great computer at the time, but I really didn’t have anything to compare it to.  The games were the best part of the computer (really the only fun part – my BASIC skills were lacking).  And even though the games were fairly lame by even 1982 standards, but they were still plenty of fun to play.
I also remember when I saw a Macintosh for the first time.  I had followed Apple for some time, but I had never had the opportunity to actually see a Macintosh.  I think it was in 1987 or 1988, when I was a student at the University of Georgia.  I believe it was a Mac SE, and it had one megabyte of RAM and a twenty-megabyte hard drive.  Up to that point, I had only played around with MS-DOS, and that was on a computer at work.  I was amazed at the graphical interface.  The mouse was definitely cool.  I had to have one, but at that time, they were a couple thousand dollars (at least), and I was a “poor college student”.  I spent a lot of time at the computer lab, and I waited.
In 1989, I landed my first job out of college, and one of the first things on my “to-buy” list was a Mac.  I just didn’t WANT a Mac – I NEEDED a Mac.  So, with a loan from the bank (yes, we took out a loan), my brother and I bought two Mac SE’s, and an HP Deskwriter dot-matrix printer that cost $1,195 (we had to share the printer).  I remember opening the box and putting the computer on my desk – and I was thrilled and amazed.   With my 2400-baud modem, I could connect to the Unix computers at work.  Life was grand. The first major improvement that I made was to upgrade the RAM to four megabytes.  I think the upgrade was a whopping $400.  And then, about a year later, the hard drive died (one of the only problems that I have ever had with a Mac).  So, I upgraded to a forty-megabyte drive for $500.  I remember thinking that I will wouldn’t need any more additional hard drive space for a really long time.  Besides, I had several hundred 3.5 inch floppies.  I later upgraded to a faster processor and bought an internal video card that let me use a huge monitor at the same time (I think it was a 19″ monitor – but it was a grayscale monitor).  But having two monitors was cool – very cool – even though a color monitor still eluded me.
And then, about every year or two, I would upgrade to a newer model.  I had an SE/30, LC, IIsi, Quadra 840 AV (which I used to edit my first videos), a Power Mac G3, G4 and then several iMac’s.  I remember staying up all night playing SIM City on the LC and playing on Prodigy.  I had other computers in the house – a Windows machine and a couple of Unix boxes.  But I was all-Mac.  I remember my first web cam, which I set up to record people driving my the house.  The motion-detection software would record a few grayscale photos when cars would drive by.  I found out when the trash was picked up and when the mail was delivered.  Technology was amazing.
I followed all of the Apple news.  Even before I bought my first Mac, I remember when Steve was booted from Apple.  I winced as I remember watching Scully and Amelio take Apple to the brink of bankruptcy.  When I traveled to the San Francisco area in the mid-1990′s on a business trip, I took some time to drive to Cupertino to the Apple Campus.  While I would have loved to have been an Apple employee back then, moving to California was pretty much out of the question.  But, I wanted to at least state that I had applied for a job there – which I did.  I wasn’t hired, but at least I tried, at least I did that.  And I remember when Steve Jobs returned.  Mac lovers now had hope.  The love affair was rekindled – and it had the opportunity to be much, much stronger than before.  Steve would fix it. He had to fix it.  And he did. 
When I started a company a few years ago, the office was filled with Intel iMac’s and a Mac Pro Server (running MySQL of course!). My wife has a MacBook Pro, my kids have Mac Mini’s, and I just recently purchased a MacBook Pro.  We even have an iMac in the kitchen.  My Mom has an iMac and my Dad has my Mom’s old G4 (when she upgrades, he gets the hand-me-downs).  My sister recently “upgraded” from Windows to an iMac.  I gave my niece a MacBook when she started graphics arts school.  (My brother is the only current hold-out, but he is on the way to converting back to the Mac).  I think you get my point – I really love Macintosh computers and I enjoy “converting people” to the cause.
Needless to say, my life has been touched in some way throughout the past 20-plus years by the Macintosh.  I have done everything from creating proposals, creating and listening to music, to editing my youngest son’s birth video (yes, it is rated G) – all on a Macintosh.  It is even where I store my tens of thousands of digital photos.  I have designed web sites, created software products and ran a business on Macintosh computers. My wife produced a video for my 40th birthday party on her Mac.  And, I can’t forget my first iPod, my first iPod Touch, and my iPhones – versions 3 and 4 (I am holding out for the iPad 3).  Heck, I even still use the old AppleWorks application every once in a while.
Even though the Mac isn’t and has never been the most dominant “PC” in the marketplace – if you ask anyone that uses a Mac if they would ever use anything else, the vast majority of them would answer with a strong and resounding “no” (or even HELL NO).  Even when Apple stock was trading at less than five bucks per share, and it seemed like it was going out of business, I remember telling friends that Apple would pull through (heck – it HAD to pull through).  I couldn’t imagine having to use a Windows machine.
And so, I owe a lot of my memories over the past twenty years to Apple Computer, and of course, to Steve Jobs.  Steve wouldn’t just manufacture a computer, he would design a work of art (albeit a more expensive work of art).  And the operating system was, well, extremely easy to use.  Having an Apple computer is tantamount to having a love affair – a comment that I usually don’t hear from users of that other operating system.  Steve’s vision for what a computer should do for the user and how the user interacts with the computer was pure genius.  And being an ex-Unix admin, once the Mac OS switched to a Unix-based OS, I was even more smitten.
When I had heard the Steve Jobs had passed, I was sad.  Well, maybe a bit more than sad.  I was never really a big fan of Steve Jobs as a person, mainly because I really didn’t know a lot about him personally – I just loved his products, his vision.  When he would announce a new product, you could tell that he was really, really proud of what he had achieved.  He wasn’t just up on stage hawking widgets, he was introducing something truly historic.  Something that would cause other companies to scurry and to try and play catch-up (they rarely succeeded).  I can only hope that the team that he has placed at Apple will continue his legacy of producing innovation at the same level of genius that he has cultivated over the years.  If not, then I will also be sad for my two kids, as they won’t get to experience the joy of using a Macintosh as I have.  Steve Jobs, you will be missed.  Greatly.
 

—————————————–






Tony Darnell is a Principal Sales Consultant for MySQL, a division of Oracle, Inc.  MySQL is the world’s most popular open-source database program.


Tony may be reached at info [at] ScriptingMySQL.com and on LinkedIn.




         


Sending and Managing Email Accounts in a MySQL Database using Perl
Tony Darnell — Fri, 30 Sep 2011 16:19:04 +0000
A friend of mine has a small newsletter that he sends via email once a month.  Initially, he was just blind-copying everyone on his list, but as the list grew, it was a pain for him to keep manually editing his BCC list.  Also, some people weren’t receiving the newsletter as they had email filters that would block emails that didn’t have their name in the “To” field.  And, he needed a way to allow people to opt-out of his newsletter without having to contact him directly.  His solution was that he wanted to use a MySQL database to send the email and manage his list, so that he could easily bypass the subscribers that had opted out. 
We started by creating a MySQL table with just a few fields – ID, email_address and news_and_events.  The “news_and_events” field would be set to “yes” for those that wanted to receive the newsletter, and could be set to “no” if they wanted to opt-out.  Here is the database:



CREATE TABLE `email_address_test` (

  `id` int(5) NOT NULL AUTO_INCREMENT,

  `email_address` varchar(50) NOT NULL,

  `news_and_events` varchar(3) NOT NULL,

  PRIMARY KEY (`id`)

) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=latin1



After he had imported all of the email addresses into the database (including setting the news_and_events column value to “yes”), we were ready to go.  We are using a Perl script to connect to the MySQL database and to retrieve all of the email addresses that have a value of “news_and_events = yes”.  We can then send each one an email with their email address in the “To” field.  We also create a log file to store the results.  To manage new subscribers, he modified his web site so that new email addresses would also be stored in the database.  An earlier post shows you how to insert information into a MySQL database via Perl.
This script uses a text file to store the database connectivity information.  (See this earlier post for instructions on how to use this text file).
This script also includes a “remove me” link (using a Perl script named remove_me.pl) in the email to allow people to opt-out of receiving future emails.  I include the remove_me.pl script below the main script.
Here is the main script to send email.  There are comments in the script to help you figure out how to modify the script for your use.



#!/usr/bin/perl -w use strict;

use DBI;

my $database;
# create a log file to store the results

open LOG, ">filename.txt" or die "Cannot create output file: $!";
my $id;

my $CurrentLine = 0;
     # connect to database
     $dbh = ConnectToMySql($database);
     $query = "select email_address, id from email_address_test where news_and_events = \'yes\'";
     $sth = $dbh->prepare($query);
     $sth->execute();
   while (@data = $sth->fetchrow_array()) {

       $email_address = $data[0];

       $id = $data[1];
     # count the number of emails that we send

     $CurrentLine++;

     

# edit this line to match the location of your sendmail program

$sendmail = "/usr/sbin/sendmail -t";
# you will need to modify this information

$reply_to = "From: Scripting MySQL Blog \n";

$from = "From: Scripting MySQL Blog \n";

$subject = "Subject: Test email from MySQL\n";

$to = "To: $email_address\n";
# this is the body of the email - along with the unsubscribe script - remove_me.pl

$content = "This is a test email.
" .
          "Remove me.<\/a>";

     

     open(SENDMAIL, "|$sendmail") or die "Cannot open $sendmail: $!";

     print SENDMAIL $reply_to;

     print SENDMAIL $from;

     print SENDMAIL $subject;

     print SENDMAIL $to;

# use this line if you want to send a plain text email

#     print SENDMAIL "Content-type: text/plain\n\n";

     print SENDMAIL "Content-type: text/html\n\n";

     print SENDMAIL $content;

     close(SENDMAIL);

     

     # comment these line if you don't want output to your monitor

     print "$CurrentLine Email sent to $email_address - $subject";

     # comment this line if you don't want output to your log file

     print LOG "$CurrentLine Email sent to $email_address - $subject";

     # sleep X seconds after each email is sent

     sleep 1;

     

}
print "\nFinished...$CurrentLine emails were sent.\n";

print LOG "\nFinished...$CurrentLine emails were sent.\n";
close(LOG);
# for more information on this sub-routine and the use of the file accessBLOG, visit

# http://scriptingmysql.wordpress.com/2011/07/27/connecting-to-mysql-with-perl/
#----------------------------------------------------------------------

sub ConnectToMySql {

#----------------------------------------------------------------------
   open(ACCESS_INFO, "<..\/accessBLOG") || die "Can't access login credentials";

     $database = ;

     $host = ;

     $userid = ;

     $passwd = ;
     # the chomp() function will remove any newline character from the end of a string

     chomp ($database, $host, $userid, $passwd);
   my $connectionInfo="dbi:mysql:$database;$host";

   close(ACCESS_INFO);
   # make connection to database

   my $l_dbh = DBI->connect($connectionInfo,$userid,$passwd);

   return $l_dbh;
}



And here is the remove_me.pl script.  I don’t provide any error-checking to make sure that the news_and_events value is equal to “yes”, so you will need to edit the script if you want this functionality.



#! /usr/bin/perl

# DBI is the standard database interface for Perl

# DBD is the Perl module that we use to connect to the MySQL database

use DBI;

use DBD::mysql;

# use CGI for forms

use CGI qw(:standard);
my $database;
$query = new CGI;
$id = $query->param("id");

$email_address = $query->param("email_address");
print header;
# connect to the database and update news_and_events to "no"
     $dbh = ConnectToMySql($database);
     $query = "update email_address_test set news_and_events = 'no' where id = '$id'";

          

     $sth = $dbh->prepare($query);
     $sth->execute();

     

     $sth->finish;
     $dbh->disconnect;
print "Your email address $email_address has been removed.";
#----------------------------------------------------------------------

sub ConnectToMySql {

#----------------------------------------------------------------------
     open(ACCESS_INFO, "<..\/accessBLOG") || die "Can't access login credentials";

     my $database = ;

     my $host = ;

     my $userid = ;

     my $passwd = ;
# the chomp() function will remove any newline character from the end of a string

chomp ($database, $host, $userid, $passwd);

#print "
$database $host $userid $passwd 
";
     my $connectionInfo="dbi:mysql:$database;$host";

     close(ACCESS_INFO);
     chomp($userid);

     chomp($passwd);

     # make connection to database

     my $l_dbh = DBI->connect($connectionInfo,$userid,$passwd);

     return $l_dbh;
}
exit;


 

—————————————–






Tony Darnell is a Principal Sales Consultant for MySQL, a division of Oracle, Inc.  MySQL is the world’s most popular open-source database program.


Tony may be reached at info [at] ScriptingMySQL.com and on LinkedIn.