Ashita.org

The following post draws significantly from a post by Jan Odvarko at http://www.softwareishard.com/blog/firebug/nsitraceablechannel-intercept-http-traffic/ but goes a bit further. There are also some sections which were inspired by Firebug, but are heavily modified.

• What you need to know

  Before I get into the code, understand that one of the most important things in this process to understand is that your extension’s listener is just one in a chain. It is the responsibility of every listener in the chain to pass on the information. Failure to do this has some amusing consequences…. like nothing loading in the browser.

  Just to make it really clear — Don’t drop the ball. (Edit: And while you can edit the data in the stream — don’t do it unless you have a really good reason.)

• Convenience methods and aliases

  A lot of the Firefox internals are accessed using Components.classes and Components.interfaces. While the verbosity makes it clear, it can at times be overly repetitive and, honestly, can take a long time to write out. A fairly common shorthand is in use, Cc and Ci with a few other less common shorthands like CCIN (for creating instances of a class based on a class name and an interface name) and CCSV (similarly creating a service based on a class name and interface name)

  
  if (typeof Cc == "undefined") {
  	var Cc = Components.classes;
  }
  if (typeof Ci == "undefined") {
          var Ci = Components.interfaces;
  }
  if (typeof CCIN == "undefined") {
  	function CCIN(cName, ifaceName){
  		return Cc[cName].createInstance(Ci[ifaceName]);
  	}
  }
  if (typeof CCSV == "undefined") {
  	function CCSV(cName, ifaceName){
  		if (Cc[cName])
  			// if fbs fails to load, the error can be _CC[cName] has no properties
  			return Cc[cName].getService(Ci[ifaceName]);
  		else
  			dumpError("CCSV fails for cName:" + cName);
  	};
  }

  • What’s with all of the typeof checks?

    Firebug, gotta love it, but it declares the same things using const. Inside of an if() block, a const is still seen and conflicts, even when the if condition evaluates to false. The code above is essentially a workaround to satisfy both possibilities. If the user has firebug installed, then carry on; if the user doesn’t have firebug installed, declare those shorthands

• The constructor

  function TracingListener() {
  }

  Above is a (very) simple constructor function for us to create objects from. The methods and properties on the prototype are below. Note that while I could have changed the structure to accommodate better data-hiding, the method below reduces the number of new functions created by making them all declared only once on the prototype. Functions in the constructor are recreated every time the constructor is called with new yourConstructor() whereas functions on the prototype are shared by all instances.

• The prototype definition

  • Basic properties

    TracingListener.prototype =
    {
        originalListener: null,
        receivedData: null,   //will be an array for incoming data.
    

    The first part of the prototype definition is setting up some basic properties. Note that both are assigned null. These properties will exist on all instances of TracingListener, and thus not be undefined if/when checking. In the case of receivedData, do not be tempted to make it an array here. Remember that methods and properties on the prototype are shared by all instances of the same type — and we don’t want all instances to share the same array for data.

    Also worth note is that receivedData is a good candidate for data-hiding and declaring it local to the constructor… but scope and visibility limitations would mean the functions requiring access to it would either need to be in the constructor as well, or have accessor and mutator methods for it. If you’re making a Singleton or a small number of instances, declaring functions in the constructor is no big deal, but this listener will be instantiated hundreds or thousands of times and it’s important to keep the duplication to a minimum.

  • Methods on the prototype

    • Interface Requirements

          //For the listener this is step 1.
          onStartRequest: function(request, context) {
          	this.receivedData = []; //initialize the array
      
      	//Pass on the onStartRequest call to the next listener in the chain -- VERY IMPORTANT
      	this.originalListener.onStartRequest(request, context);
          },

      onStartRequest is the first thing called when the actual request processing begins. This is also the best opportunity to initialize the array on this listener.

          //This is step 2. This gets called every time additional data is available
          onDataAvailable: function(request, context, inputStream, offset, count)
          {
             var binaryInputStream = CCIN("@mozilla.org/binaryinputstream;1",
                                       "nsIBinaryInputStream");
              binaryInputStream.setInputStream(inputStream);
      
              var storageStream = CCIN("@mozilla.org/storagestream;1",
                                       "nsIStorageStream");
              //8192 is the segment size in bytes, count is the maximum size of the stream in bytes
              storageStream.init(8192, count, null); 
      
      	var binaryOutputStream = CCIN("@mozilla.org/binaryoutputstream;1",
                                       "nsIBinaryOutputStream");
              binaryOutputStream.setOutputStream(storageStream.getOutputStream(0));
      
              // Copy received data as they come.
              var data = binaryInputStream.readBytes(count);
      
              this.receivedData.push(data);
      
              binaryOutputStream.writeBytes(data, count);
      
              //Pass it on down the chain
              this.originalListener.onDataAvailable(request,
                                                context,
                                                storageStream.newInputStream(0),
                                                offset,
                                                count);
          },

      onDataAvailable essentially copies the data from the binaryInputStream to our receivedData array and to the storageStream (via the binaryOutputStream). Then we pass a new InputStream from our storageStream onto the next listener in the chain.

          onStopRequest: function(request, context, statusCode)
          {
      	try
      	{
                      //QueryInterface into HttpChannel to access originalURI and requestMethod properties
      		request.QueryInterface(Ci.nsIHttpChannel);
      
                      //this is specific to the PirateQuesting Add-on, but is left here as an example of how to modify behaviour based on the requested URL
      		if (request.originalURI
                          && piratequesting.baseURL == request.originalURI.prePath
                          && request.originalURI.path.indexOf("/index.php?ajax=") == 0)
      		{
      
      			var data = null;
      			if (request.requestMethod.toLowerCase() == "post")
      			{
      				var postText = this.readPostTextFromRequest(request, context);
      				if (postText)
      					data = ((String)(postText)).parseQuery();
      
      			}
      
                              //Combine the response into a single string
      			var responseSource = this.receivedData.join('');
      
      			//fix leading spaces bug
      			//(FM occasionally adds spaces to the beginning of their ajax responses...
                              //which breaks the XML)
      			responseSource = responseSource.replace(/^\s+(\S[\s\S]+)/, "$1");
      
                              //gets the date from the response headers on the request.
                              //For PirateQuesting this was preferred over the date on the user's machine
      			var date = Date.parse(request.getResponseHeader("Date"));
      
                              //Again a PQ specific function call, but left as an example.
                              //This just passes a string URL, the text of the response,
                              //the date, and the data in the POST request (if applicable)
      			piratequesting.ProcessRawResponse(request.originalURI.spec,
                                                     responseSource,
                                                     date,
                                                     data);
      		}
      	}
      	catch (e)
      	{
      		//standard function to dump a formatted version of the error to console
      		dumpError(e);
      	}
      	//Pass it on down the chain
      	this.originalListener.onStopRequest(request,
                                               context,
                                               statusCode);
          },

      The onStopRequest above has a few tricky parts. The first is the QueryInterface to nsIHttpChannel – this is critical to getting the info needed. The second tricky part is to get the posted variables. To do so, you need to check that the requestMethod was indeed post, and then we call readPostTextFromRequest which I’ll introduce in a bit. The last tricky bit is getting the Date header from the response. Date.parse() plays nicely with those (assuming the server response conforms)

          QueryInterface: function (aIID) {
              if (aIID.equals(Ci.nsIStreamListener) ||
                  aIID.equals(Ci.nsISupports)) {
                  return this;
              }
              throw Components.results.NS_NOINTERFACE;
          },

      This is pretty standard for anything fulfilling an interface contract for Firefox (or other mozilla-based browsers). QueryInterface is part of the nsISupports interface and is the only part which is scriptable. All interfaces are derived from nsISupports, so it has to be there.

    • Utility methods

      The following methods are required by our TracingListener but are not part of the interface contract. (It would also have been possible to define them globally or within a pseudo-namespace.)

          readPostTextFromRequest : function(request, context) {
              try
              {
      	        var is = request.QueryInterface(Ci.nsIUploadChannel).uploadStream;
      	        if (is)
      	        {
      	            var ss = is.QueryInterface(Ci.nsISeekableStream);
      	            var prevOffset;
      	            if (ss)
      	            {
      	                prevOffset = ss.tell();
      	                ss.seek(Ci.nsISeekableStream.NS_SEEK_SET, 0);
      	            }
      
      	            // Read data from the stream..
      		    var charset = "UTF-8";
      		    var text = this.readFromStream(is, charset, true);
      
      	            if (ss && prevOffset == 0)
      	                ss.seek(Ci.nsISeekableStream.NS_SEEK_SET, 0);
      
      	            return text;
      	        }
      		else {
      			dump("Failed to Query Interface for upload stream.\n");
      		}
      	    }
      	    catch(exc)
      	    {
      			dumpError(exc);
      	    }
      
      	    return null;
      	},

      I will readily admit that readPostTextFromRequest is mostly taken from Firebug, though there are a few changes. Basically, we have to do the same thing as before and QueryInterface into the appropriate interface. In this case we need nsIUploadChannel to get access to uploadStream. And then we QueryInterface the uploadStream into a nsISeekableStream (noticing a pattern, yet? QueryInterface is your best friend.. and worst enemy.). After that we store the original offset in the stream in prevOffset, and then seek to the beginning of the stream. Then we read the data and, if the stream was at position 0 originally, we seek to the beginning again.

      	readFromStream : function(stream, charset, noClose)	{
      
      	    var sis = CCSV("@mozilla.org/binaryinputstream;1",
                                  "nsIBinaryInputStream");
      	    sis.setInputStream(stream);
      
      	    var segments = [];
      	    for (var count = stream.available(); count; count = stream.available())
      	        segments.push(sis.readBytes(count));
      
      	    if (!noClose)
      	        sis.close();
      
      	    var text = segments.join("");
      	    return text;
      	}
      
      }

      readFromStream is also largely from Firebug with a few modifications. It is however remarkably similar to what is done in onDataAvailable and onStopRequest. Basically, we get a BinaryInputStream to work with the stream given. Then we loop through the segments of the stream (size provided by available()) and add them to an array. When finished with that, we join the segments and return the text.

      httpRequestObserver = {
      
      	observe: function(request, aTopic, aData){
      		if (typeof Cc == "undefined") {
      			var Cc = Components.classes;
      		}
      		if (typeof Ci == "undefined") {
      			var Ci = Components.interfaces;
      		}
      	    	if (aTopic == "http-on-examine-response") {
      	    		request.QueryInterface(Ci.nsIHttpChannel);
      
      			if (request.originalURI
                                  && piratequesting.baseURL == request.originalURI.prePath
                                  && request.originalURI.path.indexOf("/index.php?ajax=") == 0) {
      				var newListener = new TracingListener();
          				request.QueryInterface(Ci.nsITraceableChannel);
          				newListener.originalListener = request.setNewListener(newListener);
      			}
      		}
      	},
      
      	QueryInterface: function(aIID){
      		if (typeof Cc == "undefined") {
      			var Cc = Components.classes;
      		}
      		if (typeof Ci == "undefined") {
      			var Ci = Components.interfaces;
      		}
      		if (aIID.equals(Ci.nsIObserver) ||
      		aIID.equals(Ci.nsISupports)) {
      			return this;
      		}
      
      		throw Components.results.NS_NOINTERFACE;
      
      	},
      };

      This part is fairly straightforward. The object httpRequestObserver has to fulfill the contract for the nsIObserver interface — which only has two methods: observe and QueryInterface.

• Observer registration

  Finally, we need to register the observer:

  var observerService = Cc["@mozilla.org/observer-service;1"]
      .getService(Ci.nsIObserverService);
  
  observerService.addObserver(httpRequestObserver,
      "http-on-examine-response", false);

  Now the observerService will call the observe method on httpRequestObserver whenever it notifies observers with the http-on-examine-response topic.

  When you want to unregister the observer, use:

  observerService.removeObserver(httpRequestObserver,
      "http-on-examine-response");

  As you can see, getting the text and post variables from an http request is non-trivial.

Note, though, that this code does not check the context to determine whether the http request is for a browser window, or from a browser window so depending on the complexity of your situation, you may want to do that as well. Perhaps, I’ll add that in another post.

(See Firebug license here. Special thanks to the Firebug team and to Jon Odvarko for providing so much useful material. The interface docs at oxymoronical are a great resource. The Mozilla Developer Center also deserves special credit for great documentation. )

Update (Jan 17, 2010): Corrected a small bug in onStopRequest (Thanks Broady!). See below for details.

Update (April 22, 2010): Corrected a bug which doesn’t occur if Firebug is installed (Thanks Harini!). See below for details.

Tags: AJAX, Firefox Extension, JavaScript, XmlHttpRequest

When making Firefox extensions with modular components, it’s nice to be able to include overlays based on preferences or some other criteria at loadtime. One of the problems that quickly comes up is that sequential loadOverlay calls will fail. This bug is documented here. The solution described there, and elsewhere, is to use chained observers and a custom queue. Strangely, I never came across an implementation example of such an observer/queue combination. So… with that stunning introduction, I give you the system used in PirateQuesting 2.

The first part here is an overlay registry. Essentially, the queue portion of the solution.

piratequesting.overlayRegistry = function() {
	var overlays = [];
	var index = 0;

	function conflicts(tabid, tabpanelid, overlayFile) {
		for (var i = 0, len = overlays.length; i<len;++i)
			if ((tabid == overlays[i].getTabId() && tabid != null )  || (tabpanelid == overlays[i].getTabPanelId() && tabpanelid != null) || overlayFile == overlays[i].getOverlayFile())
				return true;
		return false;
	}

	function overlay(tabid, tabpanelid, overlayFile) {
		var tabid = tabid;
		var tabpanelid = tabpanelid;
		var overlayFile = overlayFile;
		var added = false;

		return {
			toString : function() {
				return "tabid: " + tabid + "\ntabpanelid: " + tabpanelid
						+ "\noverlayFile: " + overlayFile;
			},
			getTabId : function() {
				return tabid;
			},
			getTabPanelId : function() {
				return tabpanelid;
			},
			getOverlayFile : function() {
				return overlayFile;
			},
			getAdded : function() {
				return added;
			},
			setAdded : function(val) {
				added = !!val; // ensure boolean
			}
		}

	}

	return {
		addOverlay : function(tabid, tabpanelid, stringFile) {
			if (!conflicts(tabid, tabpanelid, stringFile))
				overlays.push(new overlay(tabid, tabpanelid, stringFile));
			else
				dump("\nOverlay conflict occurred on: " + tabid + ", " + tabpanelid + ", " + stringFile);
		},
		getOverlayByIndex : function(index) {
			return overlays[index];
		},
		getOverlayByTabId : function(tabid) {
			var ol = overlays.length;
			for (var i = 0; i < ol; i++) {
				if (overlays[i].getTabId() == tabid)
					return overlays[i];
			}
			return false;
		},
		count : function() {
			return overlays.length;
		},
		reset : function() {
			index = 0;
		},
		next : function() {
			if (index < overlays.length) {
				return overlays[index++];
			} else
				return null;
		},
		progress : function() {
			return Math.ceil(index * 100 / overlays.length);
		},
		resetAll : function() {
			this.reset();
			var nex;
			while (nex = this.next()) {
				nex.setAdded(false);
			}
			this.reset();
		}

	}
}();

Now, that is a somewhat specialized case. The tabid and tabpanelid are arguably unnecessary but have been included to prevent two modules having the same tab ids, and, more importantly, to be able to refer to the overlay by a known value, in this case the tabid.

A somewhat stripped down version might look like:

var overlayRegistry = function() {
	var overlays = [];
	var index = 0;

	function conflicts(overlayFile) {
		for (var i = 0, len = overlays.length; i<len;++i)
			if (overlayFile == overlays[i].getOverlayFile())
				return true;
		return false;
	}

	function overlay(overlayFile) {
		var overlayFile = overlayFile;
		var added = false;

		return {
			toString : function() {
				return "\noverlayFile: " + overlayFile;
			},
			getOverlayFile : function() {
				return overlayFile;
			},
			getAdded : function() {
				return added;
			},
			setAdded : function(val) {
				added = !!val; // ensure boolean
			}
		}

	}

	return {
		addOverlay : function(stringFile) {
			if (!conflicts(stringFile))
				overlays.push(new overlay(stringFile));
			else
				dump("\nOverlay conflict occurred on: " + stringFile);
		},
		getOverlayByIndex : function(index) {
			return overlays[index];
		},
		count : function() {
			return overlays.length;
		},
		reset : function() {
			index = 0;
		},
		next : function() {
			if (index < overlays.length) {
				return overlays[index++];
			} else
				return null;
		},
		progress : function() {
			return Math.ceil(index * 100 / overlays.length);
		},
		resetAll : function() {
			this.reset();
			var nex;
			while (nex = this.next()) {
				nex.setAdded(false);
			}
			this.reset();
		}

	}
}();

The overlay registry is simply an iterator-style queue. This makes walking through the items very easy for the observer (shown next) which doesn’t (and shouldn’t) really have any idea of the state of the queue. The queue, overlayRegistry, makes a number of methods available for getting basic info about the queue (size, progress, etc) for use in progress bars or the like. It also provides ways of resetting the queue. Obviously, since it’s designed to be an iterator, there are ways of getting the next item and advancing the queue.

The second part of the solution is using a chained observer. Again, the piratequesting implementation is:

function overlayObserver()
{
  this.register();
}
overlayObserver.prototype = {
  observe: function(subject, topic, data) {

  	function cleanUp() {
		sidebar.contentDocument.getElementById("pqmain_deck").selectedIndex="1";

		var mod_boxes = sidebar.contentDocument.getElementsByTagName("tabbox");
		for (var i=0,len=mod_boxes.length;i<len;++i) {
			if (hasClassName(mod_boxes[i],"moduleBox")) {
				mod_boxes[i].selectedIndex = 0;
			}
		}
  	}

  	if (topic == "xul-overlay-merged") {
		try {
			var nex = piratequesting.overlayRegistry.next();
			if (nex) {
  				sidebar.contentDocument.getElementById("pqloadprogress").value = piratequesting.overlayRegistry.progress();
  				try {
  					sidebar.contentDocument.loadOverlay(nex.getOverlayFile(),this);
	  			} catch (error) {
  					cleanUp();
  					dump("Failed to load: " + nex.getOverlayFile() + "\nReported Error " + getErrorString(error));
	  			}
  			} else {
  				cleanUp();
	  		}
  		} catch (error) { alert(getErrorString(error)); }
  	}
  },
  register: function() {
    var observerService = Components.classes["@mozilla.org/observer-service;1"]
                          .getService(Components.interfaces.nsIObserverService);
    observerService.addObserver(this, "xul-overlay-merged", false);
  },
  unregister: function() {
    var observerService = Components.classes["@mozilla.org/observer-service;1"]
                            .getService(Components.interfaces.nsIObserverService);
    observerService.removeObserver(this, "xul-overlay-merged");
  }
}

And a stripped down version would look something like this:

function overlayObserver()
{
  this.register();
}
overlayObserver.prototype = {
  observe: function(subject, topic, data) {

  	if (topic == "xul-overlay-merged") {
		try {
			var nex = overlayRegistry.next();
			if (nex) {
  				try {
  					document.loadOverlay(nex.getOverlayFile(),this);
	  			} catch (error) {
  					dump("Failed to load: " + nex.getOverlayFile() + "\nReported Error " + getErrorString(error));
	  			}
  			} else {
  				cleanUp();
	  		}
  		} catch (error) { alert(getErrorString(error)); }
  	}
  },
  register: function() {
    var observerService = Components.classes["@mozilla.org/observer-service;1"]
                          .getService(Components.interfaces.nsIObserverService);
    observerService.addObserver(this, "xul-overlay-merged", false);
  },
  unregister: function() {
    var observerService = Components.classes["@mozilla.org/observer-service;1"]
                            .getService(Components.interfaces.nsIObserverService);
    observerService.removeObserver(this, "xul-overlay-merged");
  }
}

This observer chain is started with:

overlayObserver.observe(null,"xul-overlay-merged", null);

The observer code is also fairly simple but relies on some things that were not terribly well documented. The key thing to know about loadOverlay is that it raises an xul-overlay-merged observer notification. here, but you’ll notice what’s missing. This entry, however, has that piece of info. If you’ve never used observers before, this gives a good rundown.

Sooo…. What happens? how does it work? Basically, at some point during the initialization of the sidebar (not important in this discussion), the observer chain is started and will loop as follows:

1. receive notification of the last loadOverlay finishing
2. get the next item in the iterator
3. if the “next item” is null, stop
4. load the overlay file specified by said item

And you’re done. One Note I would make is that if there are multiple extensions making use of this at the same time, you’re very likely to have problems when both extensions receive the notifications and both start loading their next items. As firefox still lacks a built-in queue for overlay loading, we’re stuck hoping that nobody else will use this at the same time.

Note: You may notice that a lot of the piratequesting code makes use of a function getErrorString(). This is a very siple function that puts all of the error info I want into a string. dump() is a function available in firefox for dumping text to the console. I am also currently working on an error logging system and will cover all of these issues in more detail when that is finished. For the time being, ignore how I handle the errors but for obvious reasons, you will want to have some kind of error handling in place.

Edit 2009/04/04:
It has since occurred to me that by using additional information, that is, subject, from the observer notification, I can reduce the chances of conflict by ensuring it only fires on the completion of it’s own loadOverlay calls.

This is going to be a pretty short tutorial today to help explain how to pass information to an window when opening it. The following is taken from PirateQuesting.

First, here is the function for entering the code when PirateQuest asks the user for verification.

function enterCode(url,imgsrc, func) {
	var params = {in:imgsrc, out:null};
	window.openDialog("chrome://piratequesting/content/codeDialog.xul", "",
		"chrome, dialog, modal, resizable=no, status=no,
		height=250, width=400", params).focus();
	if (params.out) {
		piratequesting.Code.submit(url,params.out,imgsrc,func);
	}
	else {
	    // User clicked cancel. stop here
	}
}

So, as we can see in the first line the params variable stores a hash. A has is useful here as it allows us to easily pass more than one variable in without goign to the work of creating an object. There are actually much more significant differences between a hash and an object but, for this tutorial, know that it stores values in name:value pairs separated by commas and all of it enclosed by curly braces. The last element must not be followed by a comma.

Next, when we use openDialog we pass params into the dialog.

After the user has clicked OK, the value of params.out is checked. The condition will be true unless the value is still null or by some strange miracle taken on a value like ‘false’.

Now, let’s look at the code behind the dialog itself

function codeDialogOnLoad() {

    // Use the arguments passed to us by the caller
    document.getElementById("codeImage").setAttribute('src',
            window.arguments[0].in);
}

// Called once if and only if the user clicks OK
function onOK() {
    window.arguments[0].out = document.getElementById("codeValue").value;
    return true;
}

Ok, so what do we have here? well, when the dialog first loads we call codeDialogOnLoad which then sets the image source on the dialog based on the value passed in params.in. Note that it is now referred to as window.arguments[0].in.

When the user presses OK, the value of an input box, codeValue, is assigned to params.out (a.k.a. window.arguments[0].out).

Last thing to look at is the codeDialog.xul

<?xml version="1.0"?>
<?xml-stylesheet href="chrome://global/skin/global.css" type="text/css"?>
<dialog
  xmlns="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul"
  id="codeDialog"
  title="Enter the Code"
  ondialogaccept="return onOK();"
  buttonlabelaccept="Submit"
  onload="codeDialogOnLoad();"
  persist="screenX screenY width height">

	<script type="application/x-javascript" src="chrome://piratequesting/content/codeDialog.js"/>
	<vbox>
		<label value="Enter the code shown below" />
		<image id="codeImage" />
		<textbox width="50" id="codeValue" />
	</vbox>
</dialog>

As you can see, codeDialog.xul is very simple and contains only three elements inside a vbox. This is really one of the simplest examples you could use and was chosen to illustrate how to simply and easily pass information into and out of a dialog

Tags: CSS, DOM, Firefox, Firefox Extension, JavaScript

Pirate Questing is a firefox addon thatI’ve been devloping for a while. It’s available at AMO here

I’ll mostly be using this space to document interesting solutions I come across.

Tags: Firefox, JavaScript, XUL