Oct 11 2009

Howto: XHR Listening by a Firefox Addon

Category: Firefox,JavaScript,Pirate Questing — Jon @ 4:00 am

The following post draws significantly from a post by Jan Odvarko at http://www.softwareishard.com/blog/firebug/nsitraceablechannel-intercept-http-traffic/ but goes a bit further. There are also some sections which were inspired by Firebug, but are heavily modified.

  • What you need to know

    Before I get into the code, understand that one of the most important things in this process to understand is that your extension’s listener is just one in a chain. It is the responsibility of every listener in the chain to pass on the information. Failure to do this has some amusing consequences…. like nothing loading in the browser.

    Just to make it really clear — Don’t drop the ball. (Edit: And while you can edit the data in the stream — don’t do it unless you have a really good reason.)

  • Convenience methods and aliases

    A lot of the Firefox internals are accessed using Components.classes and Components.interfaces. While the verbosity makes it clear, it can at times be overly repetitive and, honestly, can take a long time to write out. A fairly common shorthand is in use, Cc and Ci with a few other less common shorthands like CCIN (for creating instances of a class based on a class name and an interface name) and CCSV (similarly creating a service based on a class name and interface name)

    
if (typeof Cc == "undefined") {
	var Cc = Components.classes;
}
if (typeof Ci == "undefined") {
        var Ci = Components.interfaces;
}
if (typeof CCIN == "undefined") {
	function CCIN(cName, ifaceName){
		return Cc[cName].createInstance(Ci[ifaceName]);
	}
}
if (typeof CCSV == "undefined") {
	function CCSV(cName, ifaceName){
		if (Cc[cName])
			// if fbs fails to load, the error can be _CC[cName] has no properties
			return Cc[cName].getService(Ci[ifaceName]); 
		else
			dumpError("CCSV fails for cName:" + cName);
	};
}

    • What’s with all of the typeof checks?

      Firebug, gotta love it, but it declares the same things using const. Inside of an if() block, a const is still seen and conflicts, even when the if condition evaluates to false. The code above is essentially a workaround to satisfy both possibilities. If the user has firebug installed, then carry on; if the user doesn’t have firebug installed, declare those shorthands

  • The constructor

    function TracingListener() {
}

    Above is a (very) simple constructor function for us to create objects from. The methods and properties on the prototype are below. Note that while I could have changed the structure to accommodate better data-hiding, the method below reduces the number of new functions created by making them all declared only once on the prototype. Functions in the constructor are recreated every time the constructor is called with new yourConstructor() whereas functions on the prototype are shared by all instances.

  • The prototype definition

    • Basic properties

      TracingListener.prototype =
{
    originalListener: null,
    receivedData: null,   //will be an array for incoming data.

      The first part of the prototype definition is setting up some basic properties. Note that both are assigned null. These properties will exist on all instances of TracingListener, and thus not be undefined if/when checking. In the case of receivedData, do not be tempted to make it an array here. Remember that methods and properties on the prototype are shared by all instances of the same type — and we don’t want all instances to share the same array for data.

      Also worth note is that receivedData is a good candidate for data-hiding and declaring it local to the constructor… but scope and visibility limitations would mean the functions requiring access to it would either need to be in the constructor as well, or have accessor and mutator methods for it. If you’re making a Singleton or a small number of instances, declaring functions in the constructor is no big deal, but this listener will be instantiated hundreds or thousands of times and it’s important to keep the duplication to a minimum.

    • Methods on the prototype

      • Interface Requirements
            //For the listener this is step 1.
    onStartRequest: function(request, context) {
    	this.receivedData = []; //initialize the array

	//Pass on the onStartRequest call to the next listener in the chain -- VERY IMPORTANT
	this.originalListener.onStartRequest(request, context); 
    },

        onStartRequest is the first thing called when the actual request processing begins. This is also the best opportunity to initialize the array on this listener.

            //This is step 2. This gets called every time additional data is available
    onDataAvailable: function(request, context, inputStream, offset, count)
    {
       var binaryInputStream = CCIN("@mozilla.org/binaryinputstream;1", 
                                 "nsIBinaryInputStream");
        binaryInputStream.setInputStream(inputStream);

        var storageStream = CCIN("@mozilla.org/storagestream;1", 
                                 "nsIStorageStream");
        //8192 is the segment size in bytes, count is the maximum size of the stream in bytes
        storageStream.init(8192, count, null); 

	var binaryOutputStream = CCIN("@mozilla.org/binaryoutputstream;1",  
                                 "nsIBinaryOutputStream");
        binaryOutputStream.setOutputStream(storageStream.getOutputStream(0));

        // Copy received data as they come.
        var data = binaryInputStream.readBytes(count);

        this.receivedData.push(data);

        binaryOutputStream.writeBytes(data, count);

        //Pass it on down the chain
        this.originalListener.onDataAvailable(request, 
                                          context,
                                          storageStream.newInputStream(0), 
                                          offset, 
                                          count); 
    },

        onDataAvailable essentially copies the data from the binaryInputStream to our receivedData array and to the storageStream (via the binaryOutputStream). Then we pass a new InputStream from our storageStream onto the next listener in the chain.

            onStopRequest: function(request, context, statusCode)
    {
	try
	{
                //QueryInterface into HttpChannel to access originalURI and requestMethod properties
		request.QueryInterface(Ci.nsIHttpChannel);

                //this is specific to the PirateQuesting Add-on, but is left here as an example of how to modify behaviour based on the requested URL
		if (request.originalURI 
                    && piratequesting.baseURL == request.originalURI.prePath 
                    && request.originalURI.path.indexOf("/index.php?ajax=") == 0)
		{

			var data = null;
			if (request.requestMethod.toLowerCase() == "post")
			{
				var postText = this.readPostTextFromRequest(request, context);
				if (postText)
					data = ((String)(postText)).parseQuery();

			}

                        //Combine the response into a single string
			var responseSource = this.receivedData.join('');

			//fix leading spaces bug 
			//(FM occasionally adds spaces to the beginning of their ajax responses... 
                        //which breaks the XML)
			responseSource = responseSource.replace(/^\s+(\S[\s\S]+)/, "$1");

                        //gets the date from the response headers on the request.
                        //For PirateQuesting this was preferred over the date on the user's machine
			var date = Date.parse(request.getResponseHeader("Date"));

                        //Again a PQ specific function call, but left as an example. 
                        //This just passes a string URL, the text of the response,
                        //the date, and the data in the POST request (if applicable)
			piratequesting.ProcessRawResponse(request.originalURI.spec, 
                                               responseSource, 
                                               date, 
                                               data);
		}
	}
	catch (e)
	{
		//standard function to dump a formatted version of the error to console
		dumpError(e); 
	}
	//Pass it on down the chain
	this.originalListener.onStopRequest(request, 
                                         context, 
                                         statusCode); 
    },

        The onStopRequest above has a few tricky parts. The first is the QueryInterface to nsIHttpChannel – this is critical to getting the info needed. The second tricky part is to get the posted variables. To do so, you need to check that the requestMethod was indeed post, and then we call readPostTextFromRequest which I’ll introduce in a bit. The last tricky bit is getting the Date header from the response. Date.parse() plays nicely with those (assuming the server response conforms)

            QueryInterface: function (aIID) {
        if (aIID.equals(Ci.nsIStreamListener) ||
            aIID.equals(Ci.nsISupports)) {
            return this;
        }
        throw Components.results.NS_NOINTERFACE;
    },

        This is pretty standard for anything fulfilling an interface contract for Firefox (or other mozilla-based browsers). QueryInterface is part of the nsISupports interface and is the only part which is scriptable. All interfaces are derived from nsISupports, so it has to be there.

      • Utility methods

        The following methods are required by our TracingListener but are not part of the interface contract. (It would also have been possible to define them globally or within a pseudo-namespace.)

            readPostTextFromRequest : function(request, context) {
        try
        {
	        var is = request.QueryInterface(Ci.nsIUploadChannel).uploadStream;
	        if (is)
	        {
	            var ss = is.QueryInterface(Ci.nsISeekableStream);
	            var prevOffset;
	            if (ss)
	            {
	                prevOffset = ss.tell();
	                ss.seek(Ci.nsISeekableStream.NS_SEEK_SET, 0);
	            }

	            // Read data from the stream..
		    var charset = "UTF-8";
		    var text = this.readFromStream(is, charset, true);

	            if (ss && prevOffset == 0)
	                ss.seek(Ci.nsISeekableStream.NS_SEEK_SET, 0);

	            return text;
	        }
		else {
			dump("Failed to Query Interface for upload stream.\n");
		}
	    }
	    catch(exc)
	    {
			dumpError(exc);
	    }

	    return null;
	},

        I will readily admit that readPostTextFromRequest is mostly taken from Firebug, though there are a few changes. Basically, we have to do the same thing as before and QueryInterface into the appropriate interface. In this case we need nsIUploadChannel to get access to uploadStream. And then we QueryInterface the uploadStream into a nsISeekableStream (noticing a pattern, yet? QueryInterface is your best friend.. and worst enemy.). After that we store the original offset in the stream in prevOffset, and then seek to the beginning of the stream. Then we read the data and, if the stream was at position 0 originally, we seek to the beginning again.

        	readFromStream : function(stream, charset, noClose)	{

	    var sis = CCSV("@mozilla.org/binaryinputstream;1", 
                            "nsIBinaryInputStream");
	    sis.setInputStream(stream);

	    var segments = [];
	    for (var count = stream.available(); count; count = stream.available())
	        segments.push(sis.readBytes(count));

	    if (!noClose)
	        sis.close();

	    var text = segments.join("");
	    return text;
	}

}

        readFromStream is also largely from Firebug with a few modifications. It is however remarkably similar to what is done in onDataAvailable and onStopRequest. Basically, we get a BinaryInputStream to work with the stream given. Then we loop through the segments of the stream (size provided by available()) and add them to an array. When finished with that, we join the segments and return the text.

        httpRequestObserver = {

	observe: function(request, aTopic, aData){
		if (typeof Cc == "undefined") {
			var Cc = Components.classes;
		}
		if (typeof Ci == "undefined") {
			var Ci = Components.interfaces;
		}
	    	if (aTopic == "http-on-examine-response") {
	    		request.QueryInterface(Ci.nsIHttpChannel);

			if (request.originalURI 
                            && piratequesting.baseURL == request.originalURI.prePath 
                            && request.originalURI.path.indexOf("/index.php?ajax=") == 0) {
				var newListener = new TracingListener();
    				request.QueryInterface(Ci.nsITraceableChannel);
    				newListener.originalListener = request.setNewListener(newListener);
			}
		}
	},

	QueryInterface: function(aIID){
		if (typeof Cc == "undefined") {
			var Cc = Components.classes;
		}
		if (typeof Ci == "undefined") {
			var Ci = Components.interfaces;
		}
		if (aIID.equals(Ci.nsIObserver) ||
		aIID.equals(Ci.nsISupports)) {
			return this;
		}

		throw Components.results.NS_NOINTERFACE;

	},
};

        This part is fairly straightforward. The object httpRequestObserver has to fulfill the contract for the nsIObserver interface — which only has two methods: observe and QueryInterface.

  • Observer registration

    Finally, we need to register the observer:

    var observerService = Cc["@mozilla.org/observer-service;1"]
    .getService(Ci.nsIObserverService);

observerService.addObserver(httpRequestObserver,
    "http-on-examine-response", false);

    Now the observerService will call the observe method on httpRequestObserver whenever it notifies observers with the http-on-examine-response topic.

    When you want to unregister the observer, use:

    observerService.removeObserver(httpRequestObserver,
    "http-on-examine-response");

    As you can see, getting the text and post variables from an http request is non-trivial.

Note, though, that this code does not check the context to determine whether the http request is for a browser window, or from a browser window so depending on the complexity of your situation, you may want to do that as well. Perhaps, I’ll add that in another post.

(See Firebug license here. Special thanks to the Firebug team and to Jon Odvarko for providing so much useful material. The interface docs at oxymoronical are a great resource. The Mozilla Developer Center also deserves special credit for great documentation. )

Update (Jan 17, 2010): Corrected a small bug in onStopRequest (Thanks Broady!). See below for details.

Update (April 22, 2010): Corrected a bug which doesn’t occur if Firebug is installed (Thanks Harini!). See below for details.

Tags: , , ,

44 Responses to “Howto: XHR Listening by a Firefox Addon”

  1. Ashita.org » Getting the source window of a request says:
    October 12th, 2009 3:35 pm

    [...] on presenting how to get the window from which a request originates. Yesterday’s post about XHR Listening by a Firefox Addon gives a good basis to work from so I will assume you’ve read over that and understood it (you [...]

    Reply
  2. Ashita.org » Getting the source window of a request says:
    October 17th, 2009 5:14 pm

    [...] post about XHR Listening by a Firefox Addon gives a good basis to work from so I will assume you’ve read over that and understood it (you [...]

    Reply
  3. anupbasil says:
    October 25th, 2009 1:48 am

    Hi,
    I need to get the “post” request data onRequestStart or onDataAvailble of TracingListener but when i do this it seems as if the httpchannel is closed and i get no response data. It seems to be working fine when i read the request on onRequestStop. Can you help me with this or with an alternate solution?

    Reply
  4. anupbasil says:
    October 25th, 2009 5:46 am

    Sorry about the previous post. It looks like a typo error

    Reply
  5. Jonathan Fingland says:
    October 25th, 2009 8:45 am

    No worries. One good option is to get the post data and save it as a property on the channel when listening to http-on-modify-request. That way you don’t have to worry about requesting it again and again.

    Reply
  6. Broady says:
    January 17th, 2010 9:49 pm

    Thank You for this great article – it was really useful for me. I’m playing with this code for a while and I found a bug. There is:

    var responseSource = this.receivedData.join();

    And in a fact there should be

    var responseSource = this.receivedData.join(”);

    Because the default separator for join is a coma ‘,’ ! So if You experienced problems with big files – that’s probably the bug :)

    Reply
  7. MeatPopsicle says:
    January 30th, 2010 4:02 pm

    Any chance on getting something packaged as an extension? I’ve been trying to get this to work, but I just can’t figure out wtc I’m doing wrong.

    Reply
    • Jonathan Fingland says:
      January 30th, 2010 10:23 pm

      @MeatPopsicle This is currently working in the piratequesting add-on. You can download that from http://pq.ashita.org. Just change the .xpi extenstion to .zip and you’re good to go. Note that the current release doesn’t work with the latest FF (new release soon) but all the code is there to make it work.

      Reply
  8. MeatPopsicle says:
    January 31st, 2010 1:21 pm

    Thanks, been trying to get a datamining app done for another browser game, it uses AJAX calls to pull in market data.

    Reply
  9. Alexi Jordanov says:
    February 28th, 2010 11:13 am

    I tried modifying the response as follows, but it did not work. Any ideas guys?

    var moredata = ”.getBytes();

    var binaryInputStream = CCIN(“@mozilla.org/binaryinputstream;1″,
    “nsIBinaryInputStream”);
    var storageStream = CCIN(“@mozilla.org/storagestream;1″, “nsIStorageStream”);
    var binaryOutputStream = CCIN(“@mozilla.org/binaryoutputstream;1″,
    “nsIBinaryOutputStream”);

    binaryInputStream.setInputStream(inputStream);
    storageStream.init(8192, count + moredata.length, null);
    binaryOutputStream.setOutputStream(storageStream.getOutputStream(0));

    // Copy received data as they come.
    var data = binaryInputStream.readBytes(count);
    this.receivedData.push(data);

    binaryOutputStream.writeBytes(data, count);
    binaryOutputStream.writeBytes(moredata, moredata.length);

    this.originalListener.onDataAvailable(request, context,
    storageStream.newInputStream(0), offset, count + moredata.length);

    Reply
  10. Alexi Jordanov says:
    February 28th, 2010 11:15 am

    The website cut the value of moredata, should look like:
    var moredata = ‘some javascript or HTML code’.getBytes();

    Reply
  11. Alexi Jordanov says:
    February 28th, 2010 11:40 am

    I just did it, had to remove getBytes() :)

    Reply
  12. Mauricio Gaueca F. says:
    March 2nd, 2010 5:49 am

    Thanks Jonathan, this is a great article.
    What is piratequesting object? I’m trying to run this example, but i get an error.

    Best regards.

    Reply
    • Jonathan Fingland says:
      March 2nd, 2010 6:08 am

      @Maurico

      piratequesting is an add-on I made for the browser game at http://www.piratequest.net. You can download the add-on at pq.ashita.org and test it out. Unfortunately I haven’t updated it in a while and it is no longer compatible with the current version of Firefox. If you turn off version compatibility checking it works just fine.

      Reply
  13. Harini says:
    April 22nd, 2010 9:54 pm

    Hi Jonathan,

    Tnx for this article. I got my extension to listen in on XHR request using this code but I have a problem, the code only works if firebug is installed! Without firebug the onStart, onDataAvailable and onStopRequest all get value as ‘undefined’ for their request parameter(i didn’t check for other parameters). But whenever firebug is also installed in the browser the request value gets passed and everything works.

    Can you pls let me how i can fix this?

    tnx
    Harini

    Reply
    • Jonathan Fingland says:
      April 23rd, 2010 1:17 am

      Harini,

      Good catch. I had fixed this bug in PirateQuesting several months back but obviously I’d forgotten to update this blog entry.

      Needed to add:


      if (typeof Cc == "undefined") {
      var Cc = Components.classes;
      }
      if (typeof Ci == "undefined") {
      var Ci = Components.interfaces;
      }

      to the httpRequestObserver methods as they operate in a different scope than TracingListener. Firebug’s const values were still valid though and the oversight was missed in testing.

      Reply
  14. designer mode says:
    June 27th, 2010 10:20 am

    This is exactly what I was looking for!

    Reply
  15. Andy E says:
    June 30th, 2010 12:37 pm

    Hi,

    this is a great site. I have to bookmark this one now as Honza has closed the comments section (for no obvious reason, whatsoever!)
    Well then.

    I wonder why you need a BINARY input stream for all that?
    Supposed I want to use this code as a basis to transform a web page “on the fly” before the browser can get hold of it, e. g. to remove over-eager javascripts or even only a few JS _lines_.

    Supposed too that the stream is nothing but HTML+JS, it would be great to be able to use a DOM function like getElementsByTagName() to get some nodes *before* the browser can do anything with the page.
    AFAIK this is not possible with a binary input stream.

    So am I doomed to use the BIS or can I also use a sort of “character input stream”?

    Reply
  16. Jonathan Fingland says:
    July 1st, 2010 1:42 am

    @Andy,

    What you *could* try doing is not passing things on down the listener chain until the very end. Do your processing on the completed string/page and then pass the modified version on to the other listeners.

    To perform normal DOM methods, you can load the content into a virtual iframe:

    function createDoc (htmlText) {
    _iframe.docShell.allowJavascript = false;
    _iframe.docShell.allowAuth = false;
    _iframe.docShell.allowPlugins = false;
    _iframe.docShell.allowMetaRedirects = false;
    _iframe.docShell.allowSubframes = false;
    _iframe.docShell.allowImages = false;

    var doc = _iframe.contentDocument;
    var strip = /([\s\S]*?)< \/html>/i;
    if (strip.test(htmlText)) {
    doc.getElementsByTagName(“html”)[0].innerHTML = strip.exec(htmlText)[1];
    }

    return doc;
    }

    And then just get the content back out again when you want to pass it on to the next listener

    Reply
  17. Andy E says:
    July 1st, 2010 9:11 pm

    @ Jonathan,

    thanks for your reply, and your sample code. However, you’re one step ahead :)

    I’m still in onDataAvailable():
    var data = binaryInputStream.readBytes(count);

    At first, I’ll need the step how to get the binaryInputStream to something better parseable.

    Reply
  18. Jonathan Fingland says:
    July 3rd, 2010 4:46 am

    @Andy,

    The binary input stream’s readBytes() method is just a string of bytes… you *could* parse it–and manipulate it–but there are too many potential problems.

    Don’t try to parse until the download is complete. Let’s say the stream comes in two parts… You can’t create a parse-able document from that. You’ll have unclosed tags, parts of multi-byte characters, etc.

    By the way… in all honesty, it’s probably better to modify the document after it has loaded. Delaying document loading is definitely unexpected behaviour.

    Reply
  19. Andy E says:
    July 7th, 2010 7:13 am

    @Johnathan,

    well you do have a point here, but … to get rid of over-eager javascripts I MUST modify the document before the browser sees it (if this was what you mean, that is). The JS interpreters embedded in various browsers will always *pre-process* all inline JS, so any changes after loading the full document will get dismissed. (Trust me, I had to go through all this first to realize that my attempts were fruitless :))

    Reply
  20. Divya says:
    March 29th, 2011 11:43 pm

    Likes!! :)
    I have a question. I am trying to show the response in a dialog, before loading it in the browser.
    But if i do this in onStopRequest its too late.. parts of the response have already reached the browser.. and if i do this in onDataAvailable its too early, coz parts of the response may have not yet been recieved..
    What approach should i take?

    Reply
    • Jonathan Fingland says:
      March 30th, 2011 7:12 am

      Divya, One option, I suppose, would be to not pass it on to the next listener in the chain at the end of onDataAvailable. Then, in the onStopRequest, pass all of the accumulated data to the next listener in the onDataAvailable chain. The only trick is that all of the parameters have to be preserved as the other listeners may rely on them.

      For example:

      
TracingListener.prototype =
{
    originalListener: null,
    receivedData: null,
    buffer: null,

    onStartRequest: function(request, context) {
    	this.receivedData = []; //initialize the array
    	this.buffer = []; //initialize the array

	//Pass on the onStartRequest call to the next listener in the chain -- VERY IMPORTANT
	this.originalListener.onStartRequest(request, context);
    },
    onDataAvailable: function(request, context, inputStream, offset, count)
    {
       var binaryInputStream = CCIN("@mozilla.org/binaryinputstream;1",
                                 "nsIBinaryInputStream");
        binaryInputStream.setInputStream(inputStream);

        var storageStream = CCIN("@mozilla.org/storagestream;1",
                                 "nsIStorageStream");
        //8192 is the segment size in bytes, count is the maximum size of the stream in bytes
        storageStream.init(8192, count, null); 

	var binaryOutputStream = CCIN("@mozilla.org/binaryoutputstream;1",
                                 "nsIBinaryOutputStream");
        binaryOutputStream.setOutputStream(storageStream.getOutputStream(0));

        // Copy received data as they come.
        var data = binaryInputStream.readBytes(count);

        this.receivedData.push(data);

        binaryOutputStream.writeBytes(data, count);
        this.buffer.push({request:request, context:context, inputStream:storageStream.newInputStream(0), offset:offset, count:count});
    },

onStopRequest: function(request, context, statusCode)
    {
	try
	{
                //QueryInterface into HttpChannel to access originalURI and requestMethod properties
		request.QueryInterface(Ci.nsIHttpChannel);

                //this is specific to the PirateQuesting Add-on, but is left here as an example of how to modify behaviour based on the requested URL
		if (request.originalURI
                    && piratequesting.baseURL == request.originalURI.prePath
                    && request.originalURI.path.indexOf("/index.php?ajax=") == 0)
		{

			var data = null;
			if (request.requestMethod.toLowerCase() == "post")
			{
				var postText = this.readPostTextFromRequest(request, context);
				if (postText)
					data = ((String)(postText)).parseQuery();

			}

                        //Combine the response into a single string
			var responseSource = this.receivedData.join('');

			//fix leading spaces bug
			//(FM occasionally adds spaces to the beginning of their ajax responses...
                        //which breaks the XML)
			responseSource = responseSource.replace(/^\s+(\S[\s\S]+)/, "$1");

                        //gets the date from the response headers on the request.
                        //For PirateQuesting this was preferred over the date on the user's machine
			var date = Date.parse(request.getResponseHeader("Date"));

                        //Again a PQ specific function call, but left as an example.
                        //This just passes a string URL, the text of the response,
                        //the date, and the data in the POST request (if applicable)
			piratequesting.ProcessRawResponse(request.originalURI.spec,
                                               responseSource,
                                               date,
                                               data);
		}

		//Now that we:re done with the data for our part, we can pass it on.
		var buffItem;
		for (var i=0, len=this.buffer.length;i<len;i++) {
                       buffItem = this.buffer[i];
		        this.originalListener.onDataAvailable(buffItem.request,
                                          buffItem.context,
                                          buffItem.inputStream,
                                          buffItem.offset,
                                          buffItem.count);
		}

	}
	catch (e)
	{
		//standard function to dump a formatted version of the error to console
		dumpError(e);
	}
	//Pass it on down the chain
	this.originalListener.onStopRequest(request,
                                         context,
                                         statusCode);
    },
.
.
.
(etc)

      note the use of a buffer array to store, in order, every onDataAvaliable call so that we can sort of replay them in the onStopRequest. Please note that I haven’t tested this, but I would expect it to work. We:re basically just holding it back and letting it all go at once, this way. That said, depending on the situation, this could result in highly unexpected behaviour for the end user. most people are used to seeing pages come in gradually rather than all at once.

      EDIT: Fixed Type (inputSteam -> inputStream)

      Reply
  21. Divya says:
    March 31st, 2011 4:38 pm

    Thanks dude..
    The code definitely works…
    But my firefox crashes the moment I use this approach..Any ideas as to why this might be happening?

    Reply
  22. Divya says:
    March 31st, 2011 6:00 pm

    I think I spoke too early..
    [Exception... "Component returned failure code: 0x80004003 (NS_ERROR_INVALID_POINTER) [nsIBinaryInputStream.setInputStream]” nsresult: “0×80004003 (NS_ERROR_INVALID_POINTER)”
    Any comments?

    Reply
    • Jonathan Fingland says:
      April 1st, 2011 7:20 am

      Divya, my guess is that it is failing when passing to the next listener. To minimize differences from the original code, I created the new input streams on the onDataAvailable and tried to use them later on in the onStopRequest.

      Below is a rather different approach, but may work better

      
TracingListener.prototype =
{
    originalListener: null,
    receivedData: null,
    buffer: null,

    onStartRequest: function(request, context) {
    	this.receivedData = []; //initialize the array
    	this.buffer = []; //initialize the array

	//Pass on the onStartRequest call to the next listener in the chain -- VERY IMPORTANT
	this.originalListener.onStartRequest(request, context);
    },
    onDataAvailable: function(request, context, inputStream, offset, count)
    {
       var binaryInputStream = CCIN("@mozilla.org/binaryinputstream;1",
                                 "nsIBinaryInputStream");
        binaryInputStream.setInputStream(inputStream);

        // Copy received data as they come.
        var data = binaryInputStream.readBytes(count);

        this.receivedData.push(data);

        this.buffer.push({request:request, context:context, data:data, offset:offset, count:count});
    },

    onStopRequest: function(request, context, statusCode)
    {
	try
	{
                //QueryInterface into HttpChannel to access originalURI and requestMethod properties
		request.QueryInterface(Ci.nsIHttpChannel);

                //this is specific to the PirateQuesting Add-on, but is left here as an example of how to modify behaviour based on the requested URL
		if (request.originalURI
                    && piratequesting.baseURL == request.originalURI.prePath
                    && request.originalURI.path.indexOf("/index.php?ajax=") == 0)
		{

			var data = null;
			if (request.requestMethod.toLowerCase() == "post")
			{
				var postText = this.readPostTextFromRequest(request, context);
				if (postText)
					data = ((String)(postText)).parseQuery();

			}

                        //Combine the response into a single string
			var responseSource = this.receivedData.join('');

			//fix leading spaces bug
			//(FM occasionally adds spaces to the beginning of their ajax responses...
                        //which breaks the XML)
			responseSource = responseSource.replace(/^\s+(\S[\s\S]+)/, "$1");

                        //gets the date from the response headers on the request.
                        //For PirateQuesting this was preferred over the date on the user's machine
			var date = Date.parse(request.getResponseHeader("Date"));

                        //Again a PQ specific function call, but left as an example.
                        //This just passes a string URL, the text of the response,
                        //the date, and the data in the POST request (if applicable)
			piratequesting.ProcessRawResponse(request.originalURI.spec,
                                               responseSource,
                                               date,
                                               data);
		}

		//Now that we:re done with the data for our part, we can pass it on.
		var buffItem;
		for (var i=0, len=this.buffer.length;i<len;i++) {
                       buffItem = this.buffer[i];
                       
                       var storageStream = CCIN("@mozilla.org/storagestream;1",
                                 "nsIStorageStream");
                       //8192 is the segment size in bytes, count is the maximum size of the stream in bytes
                       storageStream.init(8192, buffItem.count, null); 

                      	var binaryOutputStream = CCIN("@mozilla.org/binaryoutputstream;1",
                                 "nsIBinaryOutputStream");
                       binaryOutputStream.setOutputStream(storageStream.getOutputStream(0));

                       
                       binaryOutputStream.writeBytes(buffItem.data, buffItem.count);

                       
		        this.originalListener.onDataAvailable(buffItem.request,
                                          buffItem.context,
                                          storageStream.newInputStream(0),
                                          buffItem.offset,
                                          buffItem.count);
		}

	}
	catch (e)
	{
		//standard function to dump a formatted version of the error to console
		dumpError(e);
	}
	//Pass it on down the chain
	this.originalListener.onStopRequest(request,
                                         context,
                                         statusCode);
    },
.
.
.
(etc)

      What this attempts to do is create new input streams immediately before passing it on to the next listener.

      Reply
      • Sasha says:
        December 18th, 2011 1:50 pm

        Hey Jonathan & all…

        FIRST, GREAT ARTICLE!

        I know that I’m late to the party but I’m trying your suggestion for storing up all the data from the onDataAvailable callbacks and then creating my own calls to originalListener.onDataAvailable and originalListener.onStopRequest.

        In my solution I actually build one large buffer and try to make one call to originalListener.onDataAvailable and one call to originalListener.onStopRequest.

        MY PROBLEM: The browser has the original source (HTML) and not my modified version. I inspect all the variables and know for sure that I modified the text and I know for sure that I’m not passing the onDataAvailable calls to the original listener until the onStopRequest. I don’t see how this is possible.

        I’ve tried things like caching the first request and context but that doesn’t do it.

        ANY IDEAS?

        Here is a code snippet from onStopRequest….

        onStopRequest: function (request, context, statusCode)
        {
        try
        {
        // QueryInterface into HttpChannel to access originalURI and requestMethod properties
        request.QueryInterface(Ci.nsIHttpChannel);

        var storageStream = CCIN(“@mozilla.org/storagestream;1″, “nsIStorageStream”);
        storageStream.init(8192, this.m_nDataSize, null); //8192 is the segment size in bytes, count is the maximum size of the stream in bytes

        var binaryOutputStream = CCIN(“@mozilla.org/binaryoutputstream;1″, “nsIBinaryOutputStream”);
        binaryOutputStream.setOutputStream(storageStream.getOutputStream(0));
        binaryOutputStream.writeBytes(this.m_strData, this.m_nDataSize);

        this.originalListener.onDataAvailable(request,
        context,
        storageStream.newInputStream(0),
        0,
        this.m_nDataSize);
        }
        catch (e)
        {
        dump(e);
        }

        //Pass it on down the chain
        this.originalListener.onStopRequest(request, context, statusCode);
        },

        Reply
        • Jonathan Fingland says:
          December 18th, 2011 5:34 pm

          Hi Sasha,

          The post was originally written around the time of Firefox 3.5, so a lot of things may have changed since then. For example, I wouldn’t be surprised if Firefox handles the case where a listener in the chain drops the ball, or does something unexpected. I haven’t tested the code in question in a while, but did you try the technique I used in the previous comment? (Just to be sure that everything else is working correctly)

          I see why you would want to consolidate the buffer (subtring replacement or, preferably, document creation, modification, and then getting the string representation). I’ll try putting something together using your example. Any chance you could post additional code? What you’re trying to do should work according to all of the documentation (which is sadly out of date).

          Basically, I see a few ways that the browser is not getting the modified html:

          • the browser listener is added after yours (and thus your listener is downstream)
          • your listener is not passing it on correctly and the browser is correcting for this somehow
          • you are inserting something that is not valid html and is then ignored when the browser is parsing and rendering the page
          • the onDataAvailable method isn’t doing what you expect (you said you verified the contents passed on by onStopRequest so this is unlikely, but…)

          There are probably more possibilities but that’s all I’ve got right now. As I said, I’ll try whipping up an example like yours and see if I can figure out what went wrong.

          Reply
        • Sasha says:
          December 19th, 2011 3:17 am

          By the way.

          The code works!

          The problem was just that Firefox must do a new ‘GET’ of the code when you view source. The source is changed correctly in Firebug. :-)

          Reply
  23. Divya says:
    April 6th, 2011 8:11 pm

    I am sorry I did not respond immedialtely..
    This was not working bcoz of a typo..

    this.buffer.push({request:request, context:context, inputSteam:storageStream.newInputStream(0), offset:offset, count:count});

    it should be
    inputStream:storageStream.newInputStream(0)

    an ‘r’ was missing

    Thanks a lot dude..

    Reply
    • Jonathan Fingland says:
      April 6th, 2011 10:23 pm

      Ahh, well I’ve fixed the previous comment now, thanks. I suppose both versions would work, though the first would be preferable.

      Reply
  24. Nickolay Ponomarev says:
    April 22nd, 2012 5:15 pm

    Note that you might need to handle redirects (3xx responses) in a special way: unconditionally installing a listener on each http-on-examine-response may lead to installing two listeners for a single channel. See https://bugzilla.mozilla.org/show_bug.cgi?id=728901#c8

    Reply
    • Jon says:
      April 23rd, 2012 12:56 pm

      Thanks for bringing that to my attention, Nickolay. I’ll do a some testing and see if I can nail down what the behaviour is. You, Boris, and Loic seem to have it pretty well nailed down, but it would be nice to have a workaround that doesn’t rely on the specific status. In my real-world implementation of this, I would have to test for status 200 or 500 as the service provider occasionally spits out a 500 on a seemingly valid page. I can foresee lots of edge cases getting handled unecessairly because all we really want to do is verify that the channel already has a listener.

      Reply
  25. Sam Hall says:
    April 28th, 2012 12:14 pm

    PirateQuesting extension has a lot of stuff in it, what’s safe to prune so I can get started playing with this? Grep tells me overlay.js has the code you listed above, can I simply delete everything else in the “content” directory and start hacking away?

    Excuse me for being lazy, I have similar plans for collecting and displaying various stats from a browser based game. Normally I’d use Greasemonkey, but this game performs most of it’s requests via Flash. I just want to quickly confirm if this technique captures requests from flash components before I delve in too far.

    It would be good to have a library with basic FF extension skeleton specifically for scraping from all requests as they come in to Firefox. If your method is the way to achieve this, I’d be happy to clean up the code and start a github project for it.

    Reply
    • Jon says:
      April 28th, 2012 12:26 pm

      @Sam, Yeah, for the most part everything else can be ignored. It would be easiest to simply comment out the parts for loading the other modules and modify the PQPageLoad function to your situation. That said, I don’t think this will be of much help as the browser isn’t really involved in flash’s “ajax”. The easiest way to see all of the xmlhttprequests the browser is aware of is to use firebug. If firebug doesn’t see it, neither will the technique used in piratequesting.

      I’ve been meaning to set up something like you describe but just haven’t had the time to do it. You’re more than welcome to start a github project for it and use piratequesting code towards that end. Ideally such a system would be released as a module (See https://developer.mozilla.org/en/JavaScript_code_modules/Using at MDN) which would make including it in other projects much easier.

      Reply
      • Sam Hall says:
        May 5th, 2012 9:33 am

        The requests sent by my Flash game are visible in Firebug, and I can see the response which contains some json (previously I used Tamper Data, but it’s method of viewing the response via the cache seemed less reliable). So whatever Firebug is doing is exactly what I need.

        So I’ve just had another crack at the piratequesting addon, but so far no luck. I’ve even started playing PirateQuesting to test it out, but can’t see any side bar. I have cranked up the debug level to 16 in about:config and still can’t see any evidence that it’s running. Though I did get one error in the regular firefox console ‘doc.getElementById(“bankval2″) is null’ in Bank.js, is that a show stopper? I was hoping to get it working then gradually reduce the functionality down to what I need to capture every request sent by the browser.

        Reply
        • Jon says:
          May 5th, 2012 12:11 pm

          Sam, Try using DebugLevel 15 or, if you were intending to include PQ_DEBUG_INSANE, use DebugLevel 31. Note that those debug statements (any most errors) are output to the error/developer console. See https://developer.mozilla.org/en/DOM/window.dump at MDN for more info. (need to change a setting in about:config and add -console to the launch parameters).

          That said, if you don’t see the sidebar at all, chances are it hasn’t been opened. There are a few ways to open it: go to the View menu -> Sidebar -> PirateQuesting, or customize the toolbar and add the PirateQuesting button.

          I can’t promise that it still works with the PirateQuest game. I stopped development on it quite a while back. It was a classic problem of building on someone else’s API: the interface changes and things break. That said, I did just log in and test it out. Seems they’ve made significant changes to how things work.

          Reply
          • Sam Hall says:
            May 7th, 2012 3:48 am

            Thanks for your help, yeah I was looking in the error console. Now I can see the dumps I have enough to get started.

  26. DevelopIt says:
    July 13th, 2012 3:31 am

    Hi Jonathan,
    great detailed description.
    Unfortunatelly I can’t get it running at all. E.g. dumpError is not known in my environment. Do you have an complete (minimal) extension example with the trace listener functionality available, that you could provide to me / post here?

    Reply
    • Jon says:
      July 13th, 2012 8:27 am

      Hi DevelopIt,

      dumpError is a simple utility function I used to format errors when dumping to the console.

      
function getErrorString(e) {
	if (typeof e === "string") 
		return e;
	return "Message: " + e.message + "\nFile: " + e.fileName + "  line: " + e.lineNumber;
}

function dumpError(error) {
	dump("\n" + getErrorString(error) + "\n");
}

      Though in retrospect, testing if the error was a string and otherwise assuming it was an Exception was a poor choice on my part.

      I do have a complete extension that includes a working example, called PirateQuesting. However it is no longer actively maintained and as a result may not work 100%.You may also want to look at the immediately preceeding conversation with Sam Hall as he was in a similar situaion (trying to integrate the code into his own project)

      Reply
  27. DevelopIt says:
    July 17th, 2012 3:33 am

    Hi Jon,

    thanks for your help. Yes I did see Sam Halls comments. I was just wondering, if you have a runable example.
    But now I did get thinks running, at least I had to add this:

    var {Cc,Ci} = require(“chrome”);

    And for my purpose I had to add:

    observerService.addObserver(httpRequestObserver, “http-on-modify-request”, false);

    Thanks again for your great description!

    Reply
  28. billy says:
    September 14th, 2012 6:19 am

    Hi Jon,
    Thanks for your great tutorial.Recently I’m developing a firefox add-on that needs to catch the xhr fo the website,

    When comes to post case, there’re some params,I wonder how can I get this params?

    Reply
    • Jon says:
      September 14th, 2012 6:35 am

      Hi billy,

      in the function onStopRequest(), there are the following lines:

      
			if (request.requestMethod.toLowerCase() == "post")
			{
				var postText = this.readPostTextFromRequest(request, context);
				if (postText)
					data = ((String)(postText)).parseQuery();

			}

      And the relevant method:

      
readPostTextFromRequest : function(request, context) {
        try
        {
	        var is = request.QueryInterface(Ci.nsIUploadChannel).uploadStream;
	        if (is)
	        {
	            var ss = is.QueryInterface(Ci.nsISeekableStream);
	            var prevOffset;
	            if (ss)
	            {
	                prevOffset = ss.tell();
	                ss.seek(Ci.nsISeekableStream.NS_SEEK_SET, 0);
	            }

	            // Read data from the stream..
		    var charset = "UTF-8";
		    var text = this.readFromStream(is, charset, true);

	            if (ss && prevOffset == 0)
	                ss.seek(Ci.nsISeekableStream.NS_SEEK_SET, 0);

	            return text;
	        }
		else {
			dump("Failed to Query Interface for upload stream.\n");
		}
	    }
	    catch(exc)
	    {
			dumpError(exc);
	    }

	    return null;
	},

      The parseQuery() function I used is just a string -> array conversion:

      
String.prototype.parseQuery = function() {
	var qry = this;
	var rex = /[?&]?([^=]+)(?:=([^&#]*))?/g;
	var qmatch, key;
	var paramValues = {};
	// parse querystring storing key/values in the ParamValues associative array
	while (qmatch = rex.exec(qry)) {
		key = decodeURIComponent(qmatch[1]);// get decoded key
		val = decodeURIComponent(qmatch[2]);// get decoded value

		paramValues[key] = val;
	}
	return paramValues;
}

      With those three snippets you should be able to get the POST data from the request.

      Reply

Leave a Reply