To open a new browser window, you can simply use window.open(). However, window.open() returns a Window object for content, not for the browser window itself, so you should get the chrome Window first. The simplest way to do that is to use nsIWindowMediator [developer.mozilla.org].

window.open();
var wm = Components.classes["@mozilla.org/appshell/window-mediator;1"]
                   .getService(Components.interfaces.nsIWindowMediator);
var newWindow = wm.getMostRecentWindow("navigator:browser");
var b = newWindow.getBrowser();