The many connected worlds of Internet of Things

The Internet of things seems to have caught the fancy of every one from the everyday consumer to the government. It is exciting indeed to think what we could accomplish by having devices talk to each other over the internet. The excitement is growing rapidly and smart devices are being released with such ferocity that one wonders that when  billions of devices are connected in the next few years will there be order or chaos.

From tracking pets to  doctors monitoring their patient’s health to cars being controlled via smart phones. From power plants to smart grids to smart cities.

Even though the concept of a connected world or the Internet of Things is being accepted with fervor, intuition dictates that we do this with a grain of salt. 

Every example that we see of a connected world actually is a very small connected world within a large number of connected worlds. A network of connected worlds so to speak.

Consider the home automation system. It is one connected home leading to a network of many homes in the same network. The network is then a collection of home installs for a given company. This means that a city could have multiple company networks like the cable companies have today. 

Some IoT installs use WIFI while some use ZigBee while some Bluetooth. Each minor connected world tends to behave independently of the other. Some for business reasons while some for convenience of implementation.

History has shown that many of these small worlds,  will gravitate towards a few technological survivors . Others will fall by the wayside only to have gratified their creators that they were part of this revolutionary development. 

But in all this there remains the problem of interoperability.  It is critical that we start planning on how to make sure that there is a common standard so that all the small interconnected worlds  can communicate with each other without sacrificing their innovational privacies.

I do realize that there are many IoT standards in the fray competing to become the de-facto standard. But if history of standards is any indication this is going to be an uphill task.

As a developer this is very worrisome. The so called connected world is destined to becoming the fragmented world. Very soon and very quickly.

Worth reading is a post by David Evans in Tech Crunch titled “We Need To Get The Internet Of Things Right

Array_intersect and set theory

Sometime back a colleague asked me when would one use a function like array_intersect(). At that time other than asking him to read up on set theory I filed the question away. If you think about it when was the last time you thought about set theory while programming:-)

This week a simple problem was presented to me. A table of users has 4322 users and a new list had 1698 users. The new list was the latest list with valid users and the users table had to be synced up to the new list. I realized that this was a classic set theory problem and broke it down in to three parts:

  1. If a user in the user table is also in the new list – do nothing
  2. If a user in the user table is not in the new list then delete it from the user table
  3. If a user in the new list is not in the user table, add it to the user table

Using a Venn diagram this can be shown as follows:

sets-img1

In this we have two sets, A and B where:
A = New list of users (latest list of users)
B = User table (existing users)

The common area where the “users in the new list” are also in the “user table” is called an “intersection” in set theory and can be shown as follows (satisfies item-1 above):

C = A ∩ B

Users in the user table but not in the new list can be shown using set difference like so (satisfies item-2 above):

D = B – C

Users in the new list but not in the user table can be shown again using difference like so (satisfies item-3 above):

E = A – C

If we were to use an example:

New list: A = {‘John’, ‘Mary’, ‘David’, ‘Jack’, ‘Jason’}

User table: B = {‘Dexter’, ‘William’, ‘Jack’, ‘John’, ‘Christine’}

Users in user table and new list: C = A ∩ B = {‘Jack’, ‘John’}.
Users in the user table but not in the new list: D = B – C = {‘Dexter’, ‘William’, ‘Christine’}.
Users in the new list but not in the user table: E = A – C = {‘Mary’, ‘David’, ‘Jason’};

If we use Venn diagrams:

sets-img2

sets-img3

Using PHP we can write this as:

$A = array('John', 'Mary', 'David', 'Jack', 'Jason');
$B = array('Dexter', 'William', 'Jack', 'John', 'Christine');
$C = array_intersect($A, $B);

$D = array_diff($B, $C);
$E = array_diff($A, $C);

//display the results
echo 'New list A: ', print_r($A, true);
echo 'User table B: ', print_r($B, true);

echo 'Users in user table and new list(Item 1) – do nothing C: ', print_r($C, true);
echo 'Users in the user table but not in the new list(Item 2) – delete from user table D: ', print_r($D, true);

echo 'Users in the new list but not in the user table(Item 3) – add to user table E: ', print_r($E, true);

Jayeshs-Mac-mini:test jwadhwani$ php test.php
New list A: Array
(
[0] => John
[1] => Mary
[2] => David
[3] => Jack
[4] => Jason
)
User table B: Array
(
[0] => Dexter
[1] => William
[2] => Jack
[3] => John
[4] => Christine
)
Users in user table and new list (Item 1) – do nothing C: Array
(
[0] => John
[3] => Jack
)
Users in the user table but not in the new list (Item 2) – delete from user table D: Array
(
[0] => Dexter
[1] => William
[4] => Christine
)
Users in the new list but not in the user table (Item 3) – add to user table E: Array
(
[1] => Mary
[2] => David
[4] => Jason
)

Once you have your arrays you can iterate over them and perform any operations such as deleting records from the database.

There are other ways to solving this problem but none that are elegant come to mind. How would you solve such a problem? I look forward to your comments.

Happy Programming!

-Jayesh

Google image charts in the PHP MVC pudding!

The other day I had to implement Google Image charts using a MVC framework. I could not find any information easily so came up with this idea which I wanted to share with you. I am sure that there are others who have also thought about this.

So in essence, I had a controller, where the action would:

1. Get the charting information from the model
2. Prepare the post packet for the call to Google charts api
3. Display the chart image in the view.

In this case my view contains in addition to other markup an image tag to receive the chart image:

<img src="/myController/myAction/" width="xxx" height="yyy">

The source attribute of an image is any URL and the only requirement is that the return data be of type ‘image’. In this case the URL follows the standard Controller/Action format.You can add parameters to this URL in case you wanted to something special in your action.

My action in the controller is something like this:

public function myAction(){
        //data from your model
        //for simplicity I have created an array for demo
	$rows = array(array('Count' => 2, 'Date' => '08/09/2010'), 
                array('Count' => 2, 'Date' => '08/09/2010'), 
                array('Count' => 2, 'Date' => '08/09/2010'), 
                array('Count' => 2, 'Date' => '08/09/2010'),
	$chd = 't:';
	$chxl = '0:|';
	$chxp = '0,';
	$xoff = 35;
	foreach($rows as $row){
		$chd .= $row['Count'] . ',';
		$chxl .= urlencode($row['Date']) . '|';
		$chxp .= $xoff . ',';
		$xoff = $xoff * 2;
	}
	$chd = rtrim($chd, ',');
	$chxl = rtrim($chxl, '|');
	$chxp = rtrim($chxp, ',');
	
	  // Add data, chart type, chart size, and scale to params.
	  $chart = array(
	    'cht' => 'bvg',
	    'chs' => '600x500',
	    'chxr' => '1,0,7',
	    'chbh' => '50',
	    'chds' => '0,7',
	    'chxp' => $chxp,
	    'chxt' => 'x,y',
	    'chxs' => '2,000000,12',
	    'chxl' => $chxl,
	    'chtt' => 'Movies VS. Date',
	    'chd' => $chd);
		
		$p = '';
		foreach($chart as $k => $v){
			$p .= "&{$k}={$v}";
		}
		
	    //make the call to Google
            $url = 'https://chart.googleapis.com/chart?chid=' . md5(uniqid(rand(), true));
	    
	    $ch = curl_init();
	    // set URL and other appropriate options
	    curl_setopt($ch, CURLOPT_URL, $url);
	    curl_setopt($ch, CURLOPT_HEADER, 0);
	    curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30);
	    curl_setopt($ch, CURLOPT_POST, 1);
	    curl_setopt($ch, CURLOPT_POSTFIELDS, $p);	
	    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
	    curl_setopt($ch, CURLOPT_VERBOSE, 1);
	    curl_setopt($ch, CURLOPT_TIMEOUT, 180);   
	
	    // grab URL and pass it to the browser
	    $response = curl_exec($ch);
	    $errorNo = curl_errno($ch);
	
	    if ($errorNo !== 0) {
	        $info = curl_getinfo($ch);
	        curl_close($ch);
	        error_log ("ErrorNo: {$errorNo} Curl Info: " . print_r($info, true));
	    }
	
	    curl_close($ch);
	
	   echo $response;

So, when the view loads , the src attribute makes the call and an image is returned. Nice and simple.

But then what happens if in this same view I need to select different type of charts?

In that case my view is now something like this:

<script>
	$(document).ready(function(){
		$('#chart_type_select').change(function(){
			var type = $(this).val();			
			$('#chart_draw').attr('src', '/myController/myAction?type=' + type);
		})	
	});
</script>


<select id='chart_type_select'>
  <option value =''>Select Chart</option>
  <option value ='1'>Chart Type 1</option>
  <option value ='2'>Chart Type 2</option>
</select>

<img id="chart_draw" width="xxx" height="yyy">

All I do in this case is switch the src attribute of the image tag which now also has a get parameter, ‘type’. You can now acquire this parameter in your action and process a different chart! No page refreshes!

Whenever we do not want page refreshes the usual temptation is of course to use an AJAX call. However if you do that you will run into cross domain issues and using AJAX with large binary payloads is not recommended.

I hope this idea of using an image tag instead of an AJAX call helps you.

As always I look forward to your comments.

Happy programming!

-Jayesh

Apple push notification, PHP and XML diff

In a recent project I had to send a push notification to my iOS app by differencing two XML files.

The XML file basically contains a list of incidents and looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<incidents>
    <incident>
        <description><![CDATA[Server crashed.]]></description>
        <id>88</id>
        <time>1/4/2012 5:31 PM</time>
    </incident>
    <incident>
        <description><![CDATA[Server stolen.]]></description>
        <id>87</id>
        <time>1/3/2012 4:30 PM</time>
    </incident>
    <incident>
        <description><![CDATA[Server on fire.]]></description>
        <id>86</id>
        <time>1/2/2012 3:29 PM</time>
    </incident>
    <incident>
        <description><![CDATA[Server damaged.]]></description>
        <id>85</id>
        <time>1/2/2012 2:28 PM</time>
    </incident>
    <incident>
        <description><![CDATA[Server misplaced.]]></description>
        <id>84</id>
        <time>1/1/2012 1:27 PM</time>
    </incident>
</incidents>

Please note that for the purposes of this post I have pared down the structure and content of the file. Also, the XML file was sent to me from a parent application server and I had no control over its generation.

For the push notification I had to inform the app how many incidents had changed and new ones added. This implied that I had to do an XML diff on the current XML file and the previous one.

I could recurse over the two files but then that would be too cumbersome. It would be coupled with the structure of the xml file and I am against tight coupling. So, I thought of comparing hashes of the old and new incidents.

My strategy was to use three arrays, one which would contain the previous hashes for reference, second which would store the incident hashes from the current XML and the third which would store the counts for changed, new and deleted and un-changed.

Here is the code:

define('INCIDENT_HASHES_FILENAME', 'incident_hashes.txt');
define('XML_FILE', 'test.xml');

$incidentsPrev = array(); //will contain previous hashes if any 
$incidentHashes = array(); //will contain new hashes
$incidentStatus = array(); //will contain final status id => status - 'N' = New 'C' = Changed, 'D' = Deleted 'NC' => No Change

//check if a reference file exists.
//if a file exists but returns corrupt information
//start anew 
if(file_exists(INCIDENT_HASHES_FILENAME)){
	$incidentsPrev = unserialize(trim(file_get_contents(INCIDENT_HASHES_FILENAME)));
	if(!is_array($incidentsPrev)){
		$incidentsPrev = array();
	}
}

//debug
echo 'Previous Hashes: ', '<pre>', print_r($incidentsPrev, true), '</pre>';

//contents of the new xml file
$xml = file_get_contents(XML_FILE);
$xmlIterator = new SimpleXMLIterator($xml);

//iterate and get hashes
foreach($xmlIterator as $incident){
	//get hash for the incident
	$hash = md5($incident->asXML());
	$id = (int)$incident[0]->id;
	$incidentHashes[$id] = $hash;
}

//compare hashes
foreach($incidentHashes as $id => $hash){
	//check if the incident exists in the hash array
	if(array_key_exists($id, $incidentsPrev) === true){
		if($incidentsPrev[$id] === $hash){ //no change
			$incidentStatus[$id] = 'NC';	
		}else{
			$incidentStatus[$id] = 'C'; //changed
		}
		unset($incidentsPrev[$id]); //we are done with this one
	}else{
		$incidentStatus[$id] = 'N';//new one
	}
	//at this point what $incidentsPrev contains are incidents
	//which are deleted
	foreach($incidentsPrev as $k => $v){ //all deletes
		$incidentStatus[$k] = 'D';
	}
}

//save away the results for next time
file_put_contents(INCIDENT_HASHES_FILENAME, serialize($incidentHashes));

//results
$new = count(array_keys($incidentStatus, 'N'));
$changed = count(array_keys($incidentStatus, 'C'));
$deleted = count(array_keys($incidentStatus, 'D'));
$noChange = count(array_keys($incidentStatus, 'NC'));

echo '<p>', 'New: ', $new, ' Changed: ', $changed, ' Deleted: ', $deleted, ' No Change: ', $noChange, '</p>';

echo 'Current Hashes: ', '<pre>', print_r($incidentHashes, true), '</pre>';
echo 'Status: ', '<pre>', print_r($incidentStatus, true), '</pre>';

First off I check if I have a file with previous hashes. First time around I will not have any previous hashes so all incidents will be labeled as new. This is what my output looks like:

Previous Hashes: Array
(
)
New: 5 Changed: 0 Deleted: 0 No Change: 0

Current Hashes: Array
(
    [88] => e99e791d7413872d2b10b2774ae75e32
    [87] => 68e218269cc5a548d86272b4dee27015
    [86] => 71cc7681549a1590748ceef9034069b0
    [85] => 1d71598d3142a946e9d115276a4baee8
    [84] => eb2869de9544a89515604da930522298
)
Status: Array
(
    [88] => N
    [87] => N
    [86] => N
    [85] => N
    [84] => N
)
</pre>

Now I will change one incident, delete another and add a new one. The XML file now looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<incidents>
	<incident>
        <description><![CDATA[Rat ate server **NEW**.]]></description>
        <id>89</id>
        <time>1/5/2012 6:32 PM</time>
    </incident>
    <incident>
        <description><![CDATA[Server crashed.]]></description>
        <id>88</id>
        <time>1/4/2012 5:31 PM</time>
    </incident>
    <incident>
        <description><![CDATA[Server stolen again. **CHANGED**]]></description>
        <id>87</id>
        <time>1/3/2012 4:30 PM</time>
    </incident>
    <incident>
        <description><![CDATA[Server on fire.]]></description>
        <id>86</id>
        <time>1/2/2012 3:29 PM</time>
    </incident>
    <incident>
        <description><![CDATA[Server misplaced.]]></description>
        <id>84</id>
        <time>1/1/2012 1:27 PM</time>
    </incident>
</incidents>

Now if I run the program again this is what I get:

Previous Hashes: Array
(
    [88] => e99e791d7413872d2b10b2774ae75e32
    [87] => 68e218269cc5a548d86272b4dee27015
    [86] => 71cc7681549a1590748ceef9034069b0
    [85] => 1d71598d3142a946e9d115276a4baee8
    [84] => eb2869de9544a89515604da930522298
)

New: 1 Changed: 1 Deleted: 1 No Change: 3

Current Hashes: Array
(
    [89] => 98b372c37454d5e0209109cc3393274a
    [88] => e99e791d7413872d2b10b2774ae75e32
    [87] => 9f72a06f5f193ec350300754419444dc
    [86] => 71cc7681549a1590748ceef9034069b0
    [84] => eb2869de9544a89515604da930522298
)


Status: Array
(
    [89] => N
    [88] => NC
    [87] => C
    [86] => NC
    [85] => D
    [84] => NC
)

The code is pretty straightforward. Here are the significant steps:

0. Get hold of previous hashes if available.
1. Iterate over the XML and get all the current hashes.
2. Now compare the two.
3. If not found, must be new.
4. If found and hashes do not match then data has changed, otherwise unchanged.
5. The remaining hashes indicate that those incidents have been removed.

Possibilities for something like this are many. For example I could send the status array as a JSON package and inform the user what has changed, new or deleted with icons or change in background color.

Also note that you could use array_intersect_assoc for getting unchanged incidents or array_diff_assoc for new incidents. I will leave the rest to your imagination.

I hope this helps you and I look forward to your comments.

Happy Coding!

Jayesh

Implementation of a Tree Structure in PHP – II

I received an excellent comment by ‘Chris Rogers’ on my earlier post Implementation of a Tree Structure in PHP. He commented to the fact that the iterator does not work if the child was created prior to the parent. Excellent catch!

To that end I have modified the class, specifically the iterator’s ‘createNode’ method which takes care of this fact. The final code can be found on Github: tree-php


/**
 * JTreeRecursiveIterator
 *
 * To use a recursive iterator you have to extend of the RecursiveIteratorIterator
 * As an example I have built an unordered list
 * For detailed information on please see RecursiveIteratorIterator
 * http://us.php.net/manual/en/class.recursiveiteratoriterator.php
 *
 * @package   JTree
 * @author Jayesh Wadhwani
 * @copyright Jayesh Wadhwani
 * @license  GNU GENERAL PUBLIC LICENSE 3.0
 * @version 1.0 2011
 */
class JTreeRecursiveIterator extends RecursiveIteratorIterator {
   /**
   * @var _jTree the JTree object
   */
	private $_jTree;
   /**
   * @var _str string with ul/li string
   */
	private $_str;

	/**
	 * JTreeRecursiveIterator::__construct()
	 *
	 * @param mixed $jt - the tree object
	 * @param mixed $iterator - the tree iterator
	 * @param mixed $mode
	 * @param integer $flags
	 * @return
	 */
	public function __construct(JTree $jt, $iterator, $mode = RecursiveIteratorIterator::LEAVES_ONLY, $flags = 0) {

		parent::__construct($iterator, $mode, $flags);
		$this->_jTree = $jt;
		$this->_str = "<ul>\n";
	}

	/**
	 * JTreeRecursiveIterator::endChildren()
	 * Called when end recursing one level.(See manual)
	 * @return void
	 */
	public function endChildren() {
		parent::endChildren();
		$this->_str .= "</ul></li>\n";
	}

	/**
	 * JTreeRecursiveIterator::callHasChildren()
	 * Called for each element to test whether it has children. (See Manual)
    *
	 * @return mixed
	 */
	public function callHasChildren() {
		$ret = parent::callHasChildren();
		$value = $this->current()->getValue();

		if($ret === true) {
			$this->_str .= "<li>{$value}<ul>\n";
		} else {
			$this->_str .= "<li>{$value}</li>\n";
		}
		return $ret;
	}

	/**
	 * JTreeRecursiveIterator::__destruct()
	 * On destruction end the list and display.
	 * @return void
	 */
	public function __destruct() {
		$this->_str .= "</ul>\n";
      echo $this->_str;
	}

}

Now if you were to re-arrange the array like so where I have made ‘Fires'(id=5) as a child of Hurricanes(id=9)

$categories = array();
$categories[] = array('id' => 1, 'weather_condition' => 'weather', 'parent_id' => 9999);
$categories[] = array('id' => 2, 'weather_condition' => 'Earthquakes', 'parent_id' => 1);
$categories[] = array('id' => 3, 'weather_condition' => 'Major', 'parent_id' => 2);
$categories[] = array('id' => 4, 'weather_condition' => 'Minor', 'parent_id' => 2);
$categories[] = array('id' => 5, 'weather_condition' => 'Fires', 'parent_id' => 9);
$categories[] = array('id' => 6, 'weather_condition' => 'Rain', 'parent_id' => 1);
$categories[] = array('id' => 7, 'weather_condition' => 'Flooding', 'parent_id' => 6);
$categories[] = array('id' => 8, 'weather_condition' => 'Washout', 'parent_id' => 6);
$categories[] = array('id' => 9, 'weather_condition' => 'Hurricanes', 'parent_id' => 1);

And run it

//create a new tree object
$jt = new JTree();

//iterate building the tree
foreach($categories as $category) {
   $uid = $jt->createNode($category['weather_condition'],$category['id'], $category['parent_id']);
}

//update: removed third variable. Use defaults 
$it = new JTreeRecursiveIterator($jt, new JTreeIterator($jt->getTree()));

//iterate to create the ul list
foreach($it as $k => $v) {}

Note that I have removed addChild. It is included in createNode.

The result being:

  • weather
    • Earthquakes
      • Major
      • Minor
    • Rain
      • Flooding
      • Washout
    • Hurricanes
      • Fires

Many thanks to Chris for pointing this out.

Happy computing!

AJAX in jQuery leaks memory, really, is that true?

Difficult to believe, but yes AJAX in jQuery version 1.4.2 does leak. I had a nasty suspicion about this and a reported bug 6242 confirmed it. Not being comfortable with taking the recommended fix at face value I decided to look into this myself.

Before I continue I want to encourage you to read my previous posts XMLHttpRequest Leak in IE 7/8 and xmlhttprequest-leak-in-ie-78-forgot-the-abort-thing to fully understand my reasoning in this post.

To test the severity of the leak I wrote a simple test.

test-jquery-ajax-leak6.html

<html>
<head>
   <title>jQuery Ajax Leak Demo</title>
   <script src="jquery-1.4.2.js"></script>
<script>
$(document).ready(function(){

   //no caching of calls to for better accuracy
   $.ajaxSetup({cache: false});
   
   var interval;
   var i = 1000; //number of calls
   
   $('#button1').click(function(){
     interval = setInterval(makeLeak, 50); 
   });
   
   
   function makeLeak(){
      $.get('test6.php', function(){
          if(--i === 0){
            //all calls done. Cleanup
               clearInterval(interval);
               interval = null;
               alert('All Done');
            }
      });
   };

});
 
</script>
  
</head>

<body>
   <button id="button1" >Fire</button>
</body>
</html>

Nothing fancy here. 1000 calls at interval of 50ms. Also, the test6.php contains just a dummy echo:

test6.php

<?php echo ''; ?>

If I run the html file in sIEve I get this:

Only making the AJAX calls and doing zero data processing the memory consumption increased from 15,600 bytes to 41,428 bytes! More than 2.5 times. For 10,000 iterations the memory went up from 15,680 to 257,208 bytes. We have a leak!

To see what was going on I dissected the jQuery AJAX code. For sake of clarity I have removed code not relevant to this discussion. The pared down code from ‘jquery-1.4.2.js’ looks like this:

ajax: function( origSettings ) {

		var requestDone = false;

		// Create the request object
		var xhr = s.xhr();

		if ( !xhr ) {
			return;
		}

		// Open the socket
		// Passing null username, generates a login popup on Opera (#2865)
		if ( s.username ) {
			xhr.open(type, s.url, s.async, s.username, s.password);
		} else {
			xhr.open(type, s.url, s.async);
		}

		// Wait for a response to come back
		var onreadystatechange = xhr.onreadystatechange = function( isTimeout ) {
			// The request was aborted
			if ( !xhr || xhr.readyState === 0 || isTimeout === "abort" ) {
				//this code removed

                        requestDone = true;
				if ( xhr ) {
					xhr.onreadystatechange = jQuery.noop;
				}

			// The transfer is complete and the data is available, or the request timed out
			} else if ( !requestDone && xhr && (xhr.readyState === 4 || isTimeout === "timeout") ) {
				requestDone = true;
				xhr.onreadystatechange = jQuery.noop;
                                
                                //fire success callback
				success();

                                //fire complete callback
                                complete();

                                //more code removed here

				if ( isTimeout === "timeout" ) {
					xhr.abort();
				}

				// Stop memory leaks
				if ( s.async ) {
					xhr = null;
				}
			}
		};

		// Override the abort handler, if we can (IE doesn't allow it, but that's OK)
		// Opera doesn't fire onreadystatechange at all on abort
		try {
			var oldAbort = xhr.abort;
			xhr.abort = function() {
				if ( xhr ) {
					oldAbort.call( xhr );
				}

				onreadystatechange( "abort" );
			};
		} catch(e) { }

      	// Send the data
		try {
			xhr.send( type === "POST" || type === "PUT" || type === "DELETE" ? s.data : null );
		} catch(e) {
			jQuery.handleError(s, xhr, null, e);
			// Fire the complete handlers
			complete();
		}

		// return XMLHttpRequest to allow aborting the request etc.
		return xhr;
	}


Stepping through the function:

1. An instance of the ‘XMLHttpRequest’ object is created and put in variable named ‘xhr’.
2. the ‘open’ method is executed in preparation for ‘send’.
3. A handler is defined for the callback ‘xhr.onreadystatechange’ .
4. The ‘abort’ method is over-ridden.
5. The ‘send’ request is made.
6. ‘xhr’ the reference to the ‘XMLHttpRequest’ object instance is returned.

When the call returns it fires the ‘xhr.onreadystatechange’ handler. The following sequence of event takes place:

1. Assuming that the call is complete(readyState == 4) the else part of the if is executed.
2. To prevent leaks the handler is cleaned up like so: xhr.onreadystatechange = jQuery.noop;. (jQuery.noop is a jQuery no-operation function and is defined as noop: function() {}(~Line 520))
3. The success callback handler is called.
4. The complete callback handler is called.
5. If there is a timeout call is aborted.
6. Finally to avoid leaks xhr is set to null. (See my previous post to see why)

So, if the ‘xhr.onreadystatechange’ callback handler is being cleaned up and ‘xhr’ is being set to null why is there a leak? It is because the ‘abort’ method is over-ridden but not cleaned up. To do that we need to replace(around line 5220):

// Stop memory leaks
if ( s.async ) {
xhr = null;
}

with

// Stop memory leaks
if ( s.async ) {
xhr.abort = jQuery.noop;
xhr = null;
}

Let’s run the test again:

As you can see the consumption has stabilized around 18K. Also note how the memory is being released(-green) as the number of calls progress.

To my satisfaction I also found the official fix

Happy computing!

XMLHttpRequest Leak in IE 7/8, forgot the abort thing!

After I was done writing my previous post XMLHttpRequest Leak in IE 7/8 I suddenly remembered that I had missed the ‘abort’ method of ‘XMLHttpRequest’.

It means what it says:-) It aborts the current call.

Many a times I want to do some housekeeping either before or after calling the ‘abort’ method. To make the housekeeping transparent I will usually end up over-riding ‘abort’ before exposing it. Many libraries do it including jQuery. Consider the following code:

<html>
<head>
	<title>xmlHttpRequest Leak, Abort Override Demo</title>
<script>

function makeAjaxCall(){
    var xhr = new XMLHttpRequest();
    xhr.open('GET', 'test1.php', true);
    xhr.onreadystatechange = function (event){
      if(xhr.readyState == 4 && xhr.status == 200){
         //clean up to avoid leaks
         xhr.onreadystatechange = new Function;
         xhr = null;
      }
   };
   
   //override abort here
   var oldAbort = xhr.abort;
   xhr.abort = function() {
   	if ( xhr ) {
   		oldAbort.call( xhr );
   	}
      //some housekeeping code
      //-----
      //-----
   	
   };
	
   xhr.send(null);
   return xhr;
} 


var xhr = makeAjaxCall();

//To abort I would call
xhr.abort();
  
</script>
  
</head>

<body>
</body>
</html>

Note that I have over-ridden the abort method. Nothing fancy or tricky here.

Just to make sure everything is OK I am going to create a test to check for any leaks. Here is the test case you are probably familiar with by now:

leak-test4.html

<html>
<head>
	<title>xmlHttpRequest Leak, Abort Override Demo</title>

<script>

var interval;
var i = 1000;
function fire(){
  interval = setInterval(makeLeak, 50); 
} 

function makeLeak(){
    var xhr = new XMLHttpRequest();
    xhr.open('GET', 'test1.php?i='+i, true);
    xhr.onreadystatechange = function (event){
      if(xhr.readyState == 4 && xhr.status == 200){
         //cleanup to avoid leaks
         xhr.onreadystatechange = new Function;
         xhr = null;
            if(--i === 0){
               clearInterval(interval);
               interval = null;
               alert('All Done');
            }
      }
   };
   
   
   var oldAbort = xhr.abort;
   xhr.abort = function() {
   	if ( xhr ) {
   		oldAbort.call( xhr );
   	}
      //housekeeping code
      //----
      //------
   	
   };
	
   xhr.send(null);
   
   return xhr;
} 


  
</script>
  
</head>


<body>
   <button id = "button1" onclick="fire(false);">Fire</button>
</body>
</html>

Running this in sIEve produces the following:

IE Leak Abort
IE Leak Over-riding Abort

It seems there is a leak. I confirmed that I had cleaned up by putting the following code in the ‘xhr.onreadystatechange’ callback handler. What went wrong now?

xhr.onreadystatechange = new Function;
xhr = null;

Well I did not do a thorough job of cleaning up!. It seems I forgot to use my own advice. Like I mentioned in my previous post, you need to clean up functions you over-ride. I need to clean up ‘abort’ by adding the following:
xhr.abort= new Function;

My ‘xhr.onreadystatechange’ handler will now look like this:

xhr.onreadystatechange = function (event){
      if(xhr.readyState == 4 && xhr.status == 200){
         //cleanup to avoid leaks
         xhr.onreadystatechange = new Function;
         xhr.abort = new Function; //clean up abort override
         xhr = null;
            if(--i === 0){
               clearInterval(interval);
               interval = null;
               alert('All Done');
            }
      }
   };

Having done that, this is what I get from sIEve

IE Leak Abort Fig 2
IE Leak Over-riding Abort Fig 2

Notice that the leak has stopped.

Remember, always cleanup after yourself.

Happy computing!

XMLHttpRequest Leak in IE 7/8

Recently I came across a leak in a legacy IE 7/8 application. Memory was building up proportionally with the increasing frequency of AJAX calls. Further research brought me to this page Internet Explorer: memory leak in XMLHttpRequest (on-page) where to my surprise I found the reason for the leaks.

Sergey Ilinsky, the author of the page notes:

“Bug: The instance of XMLHttpRequest doesn’t get garbage collected in case you have a reference to the instance or to an other COM object (for example: DOM Node etc.) in its onreadystatechange handler, thus producing runtime memory leaks. “

He has also proposed a fix. I will talk about that in a moment.

To measure the leak here is a simple test page, test-leak1.html.

test-leak1.html

<html>
<head>
	<title>xmlHttpRequest Leak Demo I</title>

<script>

var interval;
var i = 1000;
function fire(){
  interval = setInterval(makeLeak, 50);
} 

function makeLeak(){
    var xhr = new XMLHttpRequest();
    //open ajax call. 'i' is dummy character to avoid caching of requests
    xhr.open('GET', 'test1.php?i='+i, true);
    xhr.onreadystatechange = function (event){
      if(xhr.readyState == 4 && xhr.status == 200){
            //all iterations done...stop timer and cleanup
            if(--i === 0){
               clearInterval(interval);
               interval = null;
               alert('All Done');
            }
      }
   };
   xhr.send(null);

}
</script>
</head>

<body>
   <button id = "button1" onclick="fire();">Fire</button>
</body>
</html>

The setup is extremely simple where after the ‘Fire’ button is clicked 1000 AJAX calls are made at 50ms intervals.

The test1.php contains on an echo statement. The intent being that I don’t want to do any data processing.

<?php
echo '';
?>

Here are the results when you run ‘test-leak1.html’ in ‘sIEve’:

IE Leak Test Fig 1
IE Leak Test Fig 1

Note that memory consumption grows from 13748 bytes to 31764 bytes. More than doubled and we haven’t even processed any data or manipulated DOM elements. You can very well imagine the damage this can do in an intensive AJAX application.

So, what is causing the leak? Let’s dissect the code to understand:

  1. 1. ‘makeLeak’ function is called.
  2. 2. An instance of ‘XMLHttpRequest’ is created and its reference put in the variable ‘xhr’.
  3. 3. The ‘open’ method is called to prepare the send request. Note the request is not sent yet.
  4. 4. ‘xhr.onreadystatechange’ callback is defined
  5. 5. The request is now sent using the ‘send’ method.
  6. 6. ‘makeLeak’ function ends and goes out of scope.

Form the above we can draw the following logical observations:

  1. 1. ‘makeLeak’ is the outer function.
  2. 2. ‘xhr.onreadystatechange’ is the inner function.
  3. 3. The inner function(‘xhr.onreadystatechange’) has access to the ‘xhr’ variable defined in its outer scope. This is true even though the outer function has gone out of scope! Think of ‘xhr’ as a static variable.
  4. 4. This implies that the outer function(‘makeLeak’) closes over the inner function ‘xhr.onreadystatechange’. Hence ‘makeLeak’ is a ‘closure’.

When the AJAX call returns:

  1. 1. ‘xhr.onreadystatechange’ fires.
  2. 2. A check is made if the AJAX call is successful(xhr.readyState == 4 && xhr.status == 200).
  3. 3. If it is not done it returns. Note that the outer variable ‘xhr’ still holds a valid reference to the ‘XMLHttpRequest’ object.
  4. 4. If it is successful(xhr.readyState == 4 && xhr.status == 200) then as far as IE is concerned the AJAX call is done.
  5. 5. When the garbage collector comes around it finds the inner function still referencing ‘xhr’ in the outer function’s scope. It leaves it alone.
  6. 6. So every time ‘makeLeak’ is called, a new instance of XMLHttpRequest is created and orphaned.

But before we start fixing this leak let’s confirm that XMLHttpRequest is still alive in well even after everything is over. Consider the following code:

test-leak3.html

<html>
<head>
<title>xmlHttpRequest Leak Demo I</title>

<script>
var success = false;

function makeLeak(){
   console.log('In outer function scope');
    var xhr = new XMLHttpRequest();
    xhr.open('GET', 'test2.php', true);
    xhr.onreadystatechange = function (event){
      if(xhr.readyState == 4 && xhr.status == 200){
         console.log('In inner function scope(AJAX Success). Response text: ' + xhr.responseText);
         success = true;
      }
   };
   xhr.send(null);

   //return reference to xhr
   var getXhr = function(){
      return xhr;
   }
   //return the reference to the function getXhr
   return getXhr;
}   

//call function makeLeak. starts the AJAX process and returns a reference
//to the method getXhr. In other words g() = makeLeak.getXhr()
var g = makeLeak();

//set up a timer to check the state of 'xhr' after the AJAX call completes.
var interval = setInterval(checkXhr, 1000);

function checkXhr(){
   if(success === true){
      //get the value of xhr;
      var xhr = g();
      if(xhr){
         console.log('xhr.responseText: ' + xhr.responseText);
      }else{
         console.log('xhr.responseText: ' + xhr);
      }
      //cleanup
      clearInterval(interval);
      interval = null;
   }
}
</script>
</head>

<body>
</body>
</html>

and test2.php contains:

<?php
echo 'Hello';
?>

The intent is to check the state of the ‘XMLHttpRequest’ object after the AJAX call is complete.

The ‘makeLeak’ function defines two inner functions. One is the callback ‘xhr.onreadystatechange’ and the other is ‘getXhr’. ‘getXhr’ only returns the handle to the ‘XMLHttpRequest’ object.

I have also defined a new variable ‘success’ . Once the AJAX call is successful it is set to ‘true’. The interval handler checks for the ‘success’ flag and if true tries to get the handle to the XMLHttpRequest object(xhr in makeLeaK). Once it has that it displays ‘responseText’.

Let’s check it out. After running the file the console windows shows the following:
Console [20]=
In outer function scope

Console [21]=
In inner function scope(AJAX Success). Response text: Hello

Console [22]=
xhr.responseText: Hello

Notice that even though ‘makeLeak’ has gone out of scope its internals are preserved. The bottom line is that the garbage collector will not cleanup an object if it determines that there is a reference to the object. So we need to help out. Please consider the following code in test-leak2.html.

test-leak2.html

<html>
<head>
	<title>xmlHttpRequest Leak Demo I</title>

<script>

var interval;
var i = 1000;
function fire(){
  interval = setInterval(makeLeak, 50);
} 

function makeLeak(){
    var xhr = new XMLHttpRequest();
    //open ajax call. 'i' is dummy character to avoid caching of requests
    xhr.open('GET', 'test1.php?i='+i, true);
    xhr.onreadystatechange = function (event){
      if(xhr.readyState == 4 && xhr.status == 200){
            //make object null, help the garbage collector to clean up
            xhr.onreadystatechange = new Function; //empty function
            xhr = null;
            //all iterations done...stop timer and cleanup
            if(--i === 0){
               clearInterval(interval);
               interval = null;
               alert('All Done');
            }
      }
   };
   xhr.send(null);

}
</script>
</head>

<body>
   <button id = "button1" onclick="fire();">Fire</button>
</body>
</html>

The only difference between this file and ‘test-leak1.html’ is these two lines in the ‘xhr.onreadystatechange’ handler:

xhr.onreadystatechange = new Function; //empty function
xhr = null;

Only xhr = null would also work but it is always a good practice to cleanup all that you have defined. In this case we defined the callback so replace it with an empty function.

Now if I run test-leak2.html in sIEve here is what I get:

IE Leak Fig 2
IE Leak Fig 2

There is virtually no memory buildup and it seems to have stabilized around 15,400 bytes.

I hope reading this post was as educational for you as it was for me writing it.

Happy computing!

jQuery Document Ready…the sequel

My previous post jQuery Document Ready, the saga continues… got me thinking. Why not write a jQuery plugin which simulates a ready handler tied to an AJAX success handler. Let me explain.

Consider test1.html, view1.html and test1.js from the previous post:
test1.html

<html>
    <head>
        <title>Document Ready Demo</title>

        <script src = "jquery-1.4.2.js"></script>
        <script src = "test1.js"></script>

        <script>
            $(document).ready(function () {
                //make sure there is no caching to avoid confusion
                $.ajaxSetup({cache: false});

                $('#fetchView1').click(function () {
                    $.get('view1.html', function (data) {
                        $('#viewText1').html(data);
                    });
                });
            });
        </script>
    </head>

    <body>
        <button id = "fetchView1">Fetch View1</button>

        <div id = "viewText1">
            View1 content goes here
        </div>
    </body>
</html>

view1.html

<div>
   <input type="text" id="inputView1" value="Click button to fill with 1111" size="50" />
   <button id="buttonView1">Fill Input with number 1111</button>
</div>

test1.js

(function($) {
   $(document).ready(function(event) {

   console.log('test1.js document.ready executed');

   $('#buttonView1').click(function() {
        console.log('buttonView1 view clicked');
        $('#inputView1').val('1111');
   });
   console.log('buttonView1 button bound now');

//possibily more processing code and bindings

   });
})(jQuery);

Like I discussed, the intent of having the document.ready is that somehow it attaches itself to view1.html. Well it attaches itself to test1.html and because of this programmers resort to using live or liverquery. Again, nothing wrong with these tools but the penalty of freedom is performance overhead.

Ideally it would be nice if we could execute the JS code on AJAX success, rather automatically. Taking a cue and inspiration from the livequery plugin this is what I came up with:

jquery.jready.js

/*!
 * jQuery plugin jReady v1.0
 * https://phptouch.com/
 *
 * Copyright 2011, Jayesh Wadhwani
 * Dual licensed under the MIT or GPL Version 2 licenses.
 *
 * Released under the MIT, BSD, and GPL Licenses.
 *
 * Date: 31-July-2011
 */
(function($) {
    $.fn.jready = function(jreadyId, fn) {
        var j;
        //make a jready object
        j = new $.jready(this, jreadyId, fn);
        return this;
    };
    
    //jready function. Objects are made of this
    $.jready = function(context, jreadyId, fn) {
//        this.selector = selector;
        this.jreadyId = jreadyId;
        this.context = context || document;
        this.fn = fn;
        this.id = $.jready.fns.push(this) - 1;

        return this;
    };
    
    //function prototype
    $.jready.prototype = {
        //execute attached functions
        execute: function() { 
         var that = this; 
         this.context.each(function(){
            that.fn.apply(this);   
         })
         
        }
    };

    $.extend($.jready, {
        fns: [], //jready objects collection
        //jready handler
        execute: function(event, xhr, settings) {
            //check if there is a jready id present in the url
            var match = /.+?jready=(\w+)[\?\&]?/ig.exec(settings.url);

            if (match !== null) {
                var jreadyId = match[1] || null;
                //if there is a jready toke/id present then 
                //check if in the collection and execute the handler
                //in the object
                if (jreadyId) {
                    $.each($.jready.fns, function(index, fn) {
                        if (fn.jreadyId == jreadyId) {
                            fn.execute();
                            return false;
                        };
                    });
                }
            }
        },
        //bind the global ajax success
        bind: function() { $(document).ajaxSuccess($.jready.execute); }
    });
    // //bind the global ajax success when dom is ready
    $(function() { $.jready.bind(); });
})(jQuery);

I will explain the internals of the plugin in a minute.

Using this plugin test1.js now contains:

(function($) {
   $(document).jready('view1', function(event) {
      console.log('test1.js-view1 jready executed');    
      
      $('#buttonView1').click(function(){
         console.log('buttonView1 view clicked');
         $('#inputView1').val('1111');
      });
      console.log('buttonView1 button bound now');
      //event bidings
      
      //processing code
      
   });
})(jQuery);

Finally test1.html now contains:

<html>
    <head>
        <title>Document Ready Demo</title>

        <script src = "jquery-1.4.2.js"></script>
        <script src = "jquery.jready.js"></script>
        <script src = "test1.js"></script>

        <script>
            $(document).ready(function () {
                //make sure there is no caching to avoid confusion
                $.ajaxSetup({cache: false});

                $('#fetchView1').click(function () {
                    $.get('view1.html?jready=view1', function (data) {
                        $('#viewText1').html(data);
                    });
                });
            });
        </script>
    </head>

    <body>
        <button id = "fetchView1">Fetch View1</button>

        <div id = "viewText1">
            View1 content goes here
        </div>
    </body>
</html>

If I now run test1.html, no binding takes place. It only takes place when the AJAX call completes. Let me explain how that happens.

1. In the URL I have attached an identifying token, ‘view1.html?jready=view1. This token can be any alpha-numeric characters.

2. In test1.js I have replaced $(document).ready(function(event) with $(document).jready(‘view1’, function(event) . Note that the “event type” here is identical to the identifying token in the URL

3. When $(document).jready(‘view1’, function(event) is executed, all the plugin does is to store away the handler function in a function object which in turn is stored in an array.

4. Also note that the global handler ajaxSuccess at the end of the plugin code. The function attached to this event checks for a valid identifying token. If a valid token is detected then it iterates over all function objects looking for a match. If a match is found then it executes the jready handler.

I still have to work on the plugin, particularly in the area of error handling. But for now it should be enough to get you going.

Elementary I believe:-)

Happy Computing!

jQuery Document Ready, the saga continues…

Further to my earlier post jQuery Document Ready…use it wisely! one reader pointed out a different kind of architecture:

The html, test1.html:

<html>
    <head>
        <title>Document Ready Demo</title>

        <script src = "jquery-1.4.2.js"></script>
        <script src = "test1.js"></script>

        <script>
            $(document).ready(function () {
                //make sure there is no caching to avoid confusion
                $.ajaxSetup({cache: false});
                
                $('#fetchView1').click(function () {
                    $.get('view1.html', function (data) {
                        $('#viewText1').html(data);
                    });
                });
            });
        </script>
    </head>

    <body>
        <button id = "fetchView1">Fetch View1</button>

        <div id = "viewText1">
            View1 content goes here
        </div>
    </body>
</html>

When the button is clicked, an AJAX call is made. This call fetches the contents of view1.html and puts it in the div with id viewText1. Now let’s look at the view:

view1.html:

<div>
   <input type="text" id="inputView1" value="Click button to fill with 1111" size="50" />
   <button id="buttonView1">Fill Input with number 1111</button>
</div>


A very simple view file indeed. Just an input text field and a button. The objective being that when this button is clicked the event handler will fill the input field with '1111'. So where is the event binding and handler? That is in the included test1.js.

The included test1.js looks like this:

(function($) { 
   $(document).ready(function(event) {

   console.log('test1.js document.ready executed');

   $('#buttonView1').click(function() {
        console.log('buttonView1 view clicked');
        $('#inputView1').val('1111');
   });
   console.log('buttonView1 button bound now');

//possibily more processing code and bindings

   });
    
})(jQuery);

Note here that the button is attached to the click event handler which fills the input with '1111'. There is also a document.ready binding and its handler which will fire when the document is good and ready. The question is which document. The answer in a minute:-)

When I run test.html1 in a browser the console displays the following:

Console [1]= test1.js document.ready executed

Console [2]= buttonView1 button bound now

Now if I were to click on the button titled "Fetch View1" the AJAX call fires and the browser output should look something like this:

If you now click the button you probably expect to see the input box filled with '1111'. But when you do, nothing happens. However the console statement above does indicate that the button was bound!

The problem is that the binding in test1.js took place when the document.ready fired which was when test1.html was loaded. At this time view1 did not exist and obviously neither did the input field and the button. So the binding in test1.js was ineffective and of no use. Imagine 55 JS included files with an average of 5 binding in each and each file with a document.ready handler. Every handler firing when the outer document is loaded. Waste of resources!

The binding should actually take place after view1.html is loaded. Not fully understanding this concept programmers resort to using live event binding or the livequery plugin. Mind you, both these are superb tools but if used without proper understanding your application will suffer from severe performance issues.

Two simple solutions come to mind.

1. Put the JS code in view1.html wrapped in document ready.

<script>
 $(document).ready(function(event) {

   console.log('test1.js document.ready executed');

   $('#buttonView1').click(function() {
        console.log('buttonView1 view clicked');
        $('#inputView1').val('1111');
   });
   console.log('buttonView1 button bound now');


//possibily more processing code and bindings

   });
</script>
<div>
   <input type="text" id="inputView1" value="Click button to fill with 1111" size="50" />
   <button id="buttonView1">Fill Input with number 1111</button>
</div>

Remove the test1.js include from test1.html:

<html>
    <head>
        <title>Document Ready Demo</title>

        <script src = "jquery-1.4.2.js"></script>

        <script>
            $(document).ready(function () {
                //make sure there is no caching to avoid confusion
                $.ajaxSetup({cache: false});

                $('#fetchView1').click(function () {
                    $.get('view1.html', function (data) {
                        $('#viewText1').html(data);
                    });
                });
            });
        </script>
    </head>

    <body>
        <button id = "fetchView1">Fetch View1</button>

        <div id = "viewText1">
            View1 content goes here
        </div>
    </body>
</html>

If you run this, you will see that the bindings work fine and the input field is filled with '1111' when the button is clicked. Reason being that jQuery considers view1.html as a document and fires the document.ready event when the view1.html DOM has loaded. Remember this will only work if you are using jQuery methods such as html, append etc. It will not work if you use core dom methods/properties such as innerHTML.

2. Make test1.js a function call.

You could also change test1.js as a function and call it after the AJAX call completes like so:

test1.js will now contain:

function test1(){ 
   console.log('test1.js function executed');

   $('#buttonView1').click(function() {
        console.log('buttonView1 view clicked');
        $('#inputView1').val('1111');
   });
   console.log('buttonView1 button bound now');

//possibily more processing code and bindings
}

test1.html will now become:

<html>
    <head>
        <title>Document Ready Demo</title>

        <script src = "jquery-1.4.2.js"></script>
        <script src = "test1.js"></script>

        <script>
            $(document).ready(function () {
               $.ajaxSetup({cache: false});
               
                $('#fetchView1').click(function () {
                    $.get('view1.html', function (data) {
                        $('#viewText1').html(data);
                        test1();
                    });
                });
            });
        </script>
    </head>

    <body>
        <button id = "fetchView1">Fetch View1</button>

        <div id = "viewText1">
            View1 content goes here
        </div>
    </body>
</html>

and finally view1.html will go back to its original form:

<div>
   <input type="text" id="inputView1" value="Click button to fill with 1111" size="50" />
   <button id="buttonView1">Fill Input with number 1111</button>
</div>

If you now run test1.html it will work as expected. Note the function call test1.js() after the AJAX call completes.

This is only the beginning. I am sure you have thought of other ways of doing this. My main purpose however of writing this post is to emphasize not to use document.ready, live bindings or livequery before fully understanding as to what is involved. For everything there is a purpose and a matching tool. Be prudent and pragmatic.

Happy programming!