I created this script because I wasn’t happy with the native Roboform importer inside Bitwarden. This fixed multiple problems including:
* Ignoring MatchUrls
* Parent folders weren’t created if they had no items in them (e.x. if I had identities in Financial/Banks, but nothing in /Financial, then it wouldn’t be created /Financial)
* Completely ignoring the extra fields (RfFieldsV2)


This fixes all those problems.


This needs to be ran via php in a command line interface, so `php convert.php`. It requires the [bitwarden cli “bw” command](https://bitwarden.com/help/cli/).
There are 2 optional arguments you can pass:
1) The name of the import file. If not given, `./RfExport.csv` will be used
2) The “bw” session token. If not given as a parameter, it can be set directly inside this file on line #4. You get this token by running `bw unlock` (after logging in with `bw login`).


This script does the following:
* Reads from a csv file exported by Roboform to import into bitwarden
* Runs `bw sync` before processing
* Imported Fields:
  * Name: Becomes the name of the item
  * Url: Becomes the URL for the item
  * MatchUrl: If this is not the same “Url” then it is added as another URL for the item
  * Login: Becomes the login (user) name
  * Pwd: Becomes the password
  * Note: Becomes the note
  * Folder: Item is put in this folder
  * RfFieldsV2 (multiple):
    * Fields that match the login or password fields are marked as “linked fields” to those. If the field names are the following, they are not added as linked fields: *user, username, login, email, userid, user id, user id$, user_email, login_email, user_login, password, passwd, password$, pass, user_password, login_password, pwd, loginpassword*
    * Fields that are different than the login or password are added as appropriate “Custom fields”. Supported types: '', rad, sel, txt, pwd, rck, chk, are
    * Each field has 5 values within it, and 3 of those seem to be different types of names. So as long as they are not duplicates of each other, each name is stored as a seperate field.
* If all fields are blank but “name” and “note” then the item is considered a “Secure Note”
* While reading the csv file for import, errors and warnings are sent to stdout
* After the csv import has complete, it will give a total number of warnings/errors and ask the user if they want to continue
* Creates missing folders (including parents that have no items in them)
* During export to bitwarden:
  * This process can be quite slow since each instance of running “bw” has to do a lot of work
  * Keeps an active count of items processed and the total number of items
  * If duplicates are found (same item name and folder) then the user is prompted for what they want to do. Which is either s=skip or o=overwrite. A capitol of those letters can be given to perform the same action on all subsequent duplicates.
  * Errors are sent to stdout

<?php
global $BW_SESSION;
/** @noinspection SpellCheckingInspection,RedundantSuppression */
$BW_SESSION=($argv[2] ?? 'FILL_ME_IN');

print RunMe($argv)."\n";

//Make sure “bw” is installed, given session is valid, and is synced
function CheckBWStatus(): ?string
{
	if(is_string($StatusCheck=RunBW('status')))
		return $StatusCheck;
	else if(!is_object($StatusCheck) || !isset($StatusCheck->status))
		return 'bw return does not have status';
	else if($StatusCheck->status==='locked')
		return 'bw is locked';
	else if($StatusCheck->status!=='unlocked')
		return 'bw status is invalid';

	$ExpectedMessage='Syncing complete.';
	if(is_string($SyncCheck=RunBW('sync', null, false)))
		return $SyncCheck;
	else if($SyncCheck[0]!==$ExpectedMessage)
		return "Sync expected “${ExpectedMessage}” but got “${SyncCheck[0]}”";

	return null;
}

//Pull the known folders
function PullKnownFolders(): string|array
{
	$FolderIDs=[];
	if(is_string($FolderList=RunBW('list folders')))
		return $FolderList;
	foreach($FolderList as $Folder)
		$FolderIDs[$Folder->name]=$Folder->id;
	unset($FolderIDs['No Folder']);
	return $FolderIDs;
}

//Get the file handle and the column indexes
function GetFileAndColumns(): string|array
{
	//Prepare the import file for reading
	ini_set('default_charset', 'UTF-8');
	$FileName=$argv[1] ?? './RfExport.csv';
	if(!file_exists($FileName))
		return 'File not found: '.$FileName;
	else if(!($f=fopen($FileName, 'r')))
		return 'Error opening file: '.$FileName;

	//Check the header row values
	if(!($HeadRow=fgetcsv($f)))
		return 'Error opening file: '.$FileName;
	else if(!count($HeadRow) || $HeadRow[0]===NULL)
		return 'Missing head row: '.$FileName;
	if(str_starts_with($HeadRow[0], "\xEF\xBB\xBF")) //Remove UTF8 BOM
		$HeadRow[0]=substr($HeadRow[0], 3);
	unset($FileName);

	$ExpectedCols=array_flip(['Url', 'Name', 'MatchUrl', 'Login', 'Pwd', 'Note', 'Folder', 'RfFieldsV2']);
	$ColNums=[];
	foreach($HeadRow as $Index => $Name) {
		if(!isset($ExpectedCols[$Name]))
			if(isset($ColNums[$Name]))
				return 'Duplicate column title: '.$Name;
			else
				return 'Unknown column title: '.$Name;
		$ColNums[$Name]=$Index;
		unset($ExpectedCols[$Name]);
	}
	if(count($ExpectedCols))
		return 'Required columns not found: '.implode(', ', array_keys($ExpectedCols));
	else if($ColNums['RfFieldsV2']!==count($ColNums)-1)
		return 'RfFieldsV2 must be the last column';

	return [$f, $ColNums];
}

//Process the rows
function ProcessRows($f, array $ColNums, array &$FolderIDs): array
{
	$Counts=['Error'=>0, 'Warning'=>0, 'Success'=>0];
	$RowNum=0;
	$FolderNames=[];
	$Items=[];
	while($Line=fgetcsv($f, null, ",", "\"", "")) {
		//Process the row and add result type to counts
		$RowNum++;
		$Result=ProcessRow($Line, $ColNums);
		$Counts[$Result[0]]++;

		//Handle errors and warnings
		if($Result[0]!=='Success') {
				print "$Result[0]: Row #$RowNum $Result[1]\n";
				continue;
		} else if(isset($Result['Warnings']) && count($Result['Warnings'])) {
			print "Warning(s): Row #$RowNum ".implode("\n", $Result['Warnings'])."\n";
			$Counts['Warning']+=count($Result['Warnings']);
		}

		//Add the folder to the list of folders (strip leading slash)
		$FolderName=($Line[$ColNums['Folder']] ?? '');
		$FolderName=substr($FolderName, ($FolderName[0] ?? '')==='/' ? 1 : 0);
		$Result['Data']->folderId=&$FolderIDs[$FolderName];
		$FolderNames[$FolderName]=1;

		//Save the entry
		$Items[]=$Result['Data'];
	}
	fclose($f);
	return [$Counts, $FolderNames, $Items, $RowNum];
}

//Process a single row
function ProcessRow($Line, $ColNums): array
{
	//Skip blank lines
	if(!count($Line) || (count($Line)===1 && $Line[0]===null))
		return ['Warning', 'is blank'];

	//Extract the columns by name
	$C=(object)[];
	foreach($ColNums as $Name => $ColNum)
		$C->$Name=$Line[$ColNum] ?? '';

	//Check for errors and end processing early for notes
	if($C->Name==='')
		return ['Error', 'is missing a name'];
	else if($C->Url==='' && $C->MatchUrl==='' & $C->Login==='' && $C->Pwd==='') {
		return ['Success', 'Data'=>(object)['type'=>2, 'notes'=>$C->Note, 'name'=>$C->Name, 'secureNote'=>['type'=>0]]];
	}

	//Create a login card
	$Ret=(object)[
		'type'=>1,
		'name'=>$C->Name,
		'notes'=>$C->Note===''  ? null : $C->Note,
		'login'=>(object)[
			'uris'=>[(object)['uri'=>$C->Url]],
			'username'=>$C->Login,
			'password'=>$C->Pwd,
		],
	];
	if($C->MatchUrl!==$C->Url)
		$Ret->login->uris[]=(object)['uri'=>$C->MatchUrl];

	//Create error string for mist fields
	$i=0; //Declared up here so it can be used in $FormatError
	$FormatError=function($Format, ...$Args) use ($ColNums, &$i): string {
		return 'RfFieldsV2 #'.($i-$ColNums['RfFieldsV2']+1).' '.sprintf($Format, ...$Args);
	};

	//Handle misc/extra (RfFieldsV2) fields
	$Warnings=[];
	for($i=$ColNums['RfFieldsV2']; $i<count($Line); $i++) {
		//Pull in the parts of the misc field
		if(count($Parts=str_getcsv($Line[$i], ",", "\"", ""))!==5)
			return ['Error', $FormatError('is not in the correct format')];
		$Parts=(object)array_combine(['Name', 'Name3', 'Name2', 'Type', 'Value'], $Parts);
		if(!trim($Parts->Name))
			return ['Error', $FormatError('invalid blank name found')];

		//Figure out which “Names” to process
		$PartNames=[trim($Parts->Name)];
		foreach([$Parts->Name2, $Parts->Name3] as $NewName)
			if($NewName!=='' && !in_array($NewName, $PartNames))
				$PartNames[]=$NewName;

		//Process the different names
		foreach($PartNames as $PartName) {
			//Determined values for the item
			$LinkedID=null;
			$PartValue=$Parts->Value;
			/** @noinspection PhpUnusedLocalVariableInspection */ $Type=null; //Overwritten in all paths

			//Handle duplicate usernames and password fields
			if(($IsLogin=($PartValue===$C->Login)) || $PartValue===$C->Pwd) {
				if($Parts->Type!==($IsLogin ? 'txt' : 'pwd'))
					$Warnings[]=$FormatError('expected type “%s” but got “%s”', $IsLogin ? 'txt' : 'pwd', $Parts->Type);
				$Type=3;
				$PartValue=null;
				$LinkedID=($IsLogin ? 100 : 101);

				//If a common name then do not add the linked field
				/** @noinspection SpellCheckingInspection */
				if(in_array(
					strToLower($PartName),
					$IsLogin ?
						['user', 'username', 'login', 'email', 'userid', 'user id', 'user id$', 'user_email', 'login_email', 'user_login'] :
						['password', 'passwd', 'password$', 'pass', 'user_password', 'login_password', 'pwd', 'loginpassword']
				))
					continue;
			} else {
				//Convert the type
				switch($Parts->Type) {
					case '': //For some reason some text fields have no type given
					case 'rad':
					case 'sel':
					case 'txt': $Type=0; break;
					case 'pwd': $Type=1; break;
					case 'rck': //Radio check?
						$Type=0;
						if($PartName!==$Parts->Name) //Ignore second names for radio checks
							continue 2;
						if(count($RadioParts=explode(':', $PartName))!==2)
							return ['Error', $FormatError('radio name needs 2 parts separated by a colon')];
						$PartName=$RadioParts[0];
						$PartValue=$RadioParts[1];
						break;
					case 'chk':
						$Type=2;
						if($PartValue==='*' || $PartValue==='1')
							$PartValue='true';
						else if($PartValue==='0')
							$PartValue='false';
						else
							return ['Error', $FormatError('invalid value for chk type “%s”', $PartValue)];
						break;
					case 'are': //This seems to be a captcha
						continue 2;
					default:
						return ['Error', $FormatError('invalid field type “%s”', $Parts->Type)];
				}
			}

			//Create the return object
			if(!isset($Ret->fields))
				$Ret->fields=[];
			$Ret->fields[]=(object)[
				'name'=>$PartName,
				'value'=>$PartValue,
				'type'=>$Type,
				'linkedId'=>$LinkedID,
			];
		}
	}

	//Return finished item and warnings
	if(count($Warnings))
		$Warnings[0]="RfFieldsV2 warnings:\n".$Warnings[0];
	return ['Success', 'Data'=>$Ret, 'Warnings'=>$Warnings];
}

//Ask the user if they want to continue
function ConfirmContinue($Counts, $RowNum): ?string
{
	//Ask the user if they want to continue
	printf("Import from spreadsheet is finished. Total: %d; Successful: %d, Errors: %d, Warnings: %d\n", $RowNum, $Counts['Success'], $Counts['Error'], $Counts['Warning']);
	while(1) {
		print 'Do you wish to continue the export to bitwarden? (y/n): ';
		fflush(STDOUT);
		switch(trim(strtolower(fgets(STDIN)))) {
			case 'n':
				return 'Exiting';
			case 'y':
				break 2;
		}
	}

	return null;
}

//Get folder IDs and create parent folders for children
function GetFolders($FolderNames, &$FolderIDs): ?string
{
	foreach(array_keys($FolderNames) as $FolderName) {
		//Skip “No Folder”
		if($FolderName==='') {
			$FolderIDs[$FolderName]='';
			continue;
		}

		//Check each part of the folder tree to make sure it exists
		$FolderParts=explode('/', $FolderName);
		$CurPath='';
		foreach($FolderParts as $Index => $FolderPart) {
			$CurPath.=($Index>0 ? '/' : '').$FolderPart;

			//If folder is already cached then nothing to do
			if(isset($FolderIDs[$CurPath])) {
				continue;
			}

			//Create the folder
			print "Creating folder: $CurPath\n";
			if(is_string($FolderInfo=RunBW('create folder '.base64_encode(json_encode(['name'=>$CurPath])), 'create folder: '.$CurPath)))
				return $FolderInfo;
			else if (!isset($FolderInfo->id))
				return "bw folder create failed for “${CurPath}”: Return did not contain id";
			$FolderIDs[$CurPath]=$FolderInfo->id;
		}
	}
	return null;
}

//Pull the known items
function PullKnownItems($FoldersByID): string|array
{
	$CurItems=[];
	if(is_string($ItemList=RunBW('list items')))
		return $ItemList;
	foreach($ItemList as $Item) {
		$CurItems[($Item->folderId===null ? '' : $FoldersByID[$Item->folderId].'/').$Item->name]=$Item->id;
	}
	return $CurItems;
}

function StoreItems($Items, $CurItems, $FoldersByID): int
{
	print "\n"; //Give an extra newline before starting the processing
	$NumItems=count($Items);
	$NumErrors=0;
	$HandleDuplicatesAction=''; //The "always do this" action
	foreach($Items as $Index => $Item) {
		//Clear the current line and print the processing status
		printf("\rProcessing item #%d/%d", $Index+1, $NumItems);
		fflush(STDOUT);

		//If the item already exists then request what to do
		$FullItemName=($Item->folderId===null ? '' : $FoldersByID[$Item->folderId].'/').$Item->name;
		if(isset($CurItems[$FullItemName])) {
			//If no "always" action has been set in $HandleDuplicatesAction then ask the user what to do
			if(!($Action=$HandleDuplicatesAction)) {
				print "\n";
				while(1) {
					print "A duplicate at “${FullItemName}” was found. Choose your action (s=skip,o=overwrite,S=always skip,O=always overwrite):";
					fflush(STDOUT);
					$Val=trim(fgets(STDIN));
					if(in_array(strtolower($Val), ['s', 'o']))
						break;
				}
				$Action=strtolower($Val);
				if($Val!==$Action) //Upper case sets the "always" state
					$HandleDuplicatesAction=$Action;
			}

			//Skip the item
			if($Action==='s')
				continue;

			//Overwrite the item
			if(is_string($Ret=RunBW('edit item '.$CurItems[$FullItemName].' '.base64_encode(json_encode($Item)), 'edit item: '.$FullItemName))) {
				$NumErrors++;
				printf("\nError on item #%d: %s\n", $Index+1, $Ret);
			}
			continue;
		}

		//Create the item
		if(is_string($Ret=RunBW('create item '.base64_encode(json_encode($Item)), 'create item: '.$FullItemName))) {
			$NumErrors++;
			printf("\nError on item #%d: %s\n", $Index+1, $Ret);
		}
	}
	return $NumErrors;
}

//Run a command through the “bw” application
function RunBW($Command, $Label=null, $DecodeJSON=true): string|object|array
{
	//$Label is set to $Command if not given
	if($Label===null)
		$Label=$Command;

	//Run the command and check the results
	global $BW_SESSION;
	exec(sprintf('BW_SESSION=%s bw %s --pretty 2>&1', $BW_SESSION, $Command), $Output, $ResultCode);
	if($ResultCode===127)
		return 'bw is not installed';
	else if($ResultCode===1)
		return "bw “${Label}” threw an error: \n".implode("\n", $Output);
	else if($ResultCode!==0)
		return "bw “${Label}” returned an invalid status code [$ResultCode]: \n".implode("\n", $Output);
	else if($Output[0]==='mac failed.')
		return 'Invalid session ID';
	else if(!$DecodeJSON)
		return [implode("\n", $Output)];
	else if(($JsonRet=json_decode(implode("\n", $Output)))===null)
		return "bw “${Label}” returned non-json result";

	//Return the json object
	return $JsonRet;
}

function RunMe($argv): string
{
	//Make sure “bw” is installed and given session is valid
	if(is_string($Ret=CheckBWStatus()))
		return $Ret;

	//Pull the known folders
	if(is_string($FolderIDs=PullKnownFolders()))
		return $FolderIDs;

	//Get the file handle and the column indexes
	if(is_string($Ret=GetFileAndColumns()))
		return $Ret;
	[$f, $ColNums]=$Ret;
	unset($argv);

	//Process the rows and ask the user if they want to continue
	[$Counts, $FolderNames, $Items, $RowNum]=ProcessRows($f, $ColNums, $FolderIDs);
	if(is_string($Ret=ConfirmContinue($Counts, $RowNum)))
		return $Ret;
	unset($Counts, $RowNum, $f, $ColNums);

	//Get folder IDs and create parent folders for children
	if(is_string($Ret=GetFolders($FolderNames, $FolderIDs)))
		return $Ret;
	unset($FolderNames);

	//Pull the known items
	$FoldersByID=array_flip($FolderIDs);
	if(is_string($CurItems=PullKnownItems($FoldersByID)))
		return $CurItems;

	//Store all items
	$FolderIDs['']=null; //Change empty folder to id=null for json insertions
	$NumErrors=StoreItems($Items, $CurItems, $FoldersByID);

	//Return completion information
	return "\nCompleted ".($NumErrors ? "with $NumErrors error(s)" : 'successfully');
}

I’ve been moving from Windows to Linux recently and my latest software move attempt is Roboform, a password manager that I use in offline mode. Just running it under Wine works fine for the primary software interaction, but I was unable to get it fully integrated with its chrome extension. Below is the information for my attempt to get it working, in case someone could use the information or wanted to try continuing my attempts.

To move your RoboForm profile to Linux, copy the data from C:\Users\USERNAME\AppData\Local\RoboForm\Profiles\Default Profile to ~/.wine/drive_c/users/USERNAME/Local Settings/Application Data/RoboForm/Profiles/Default Profile.

The chrome extension talks to its parent, rf-chrome-nm-host.exe, through the native messaging API. To direct the extension to talk to the windows executable you have to edit ~/.config/chromium/NativeMessagingHosts/com.siber.roboform.json and change the path inside it to /home/USERNAME/chrome-robo.sh, a script file that you will create. You can’t link directly to the rf-chrome-nm-host.exe because it has to be run through Wine, and the path cannot contain arguments.

Create a file with executable permissions at ~/chrome-robo.sh and set its contents to:

cd "/home/USERNAME/.wine/drive_c/Program Files (x86)/Siber Systems/AI RoboForm/9.6.1.1/";
/usr/bin/wine ./rf-chrome-nm-host.exe chrome-extension://pnlccmojcmeohlpggmfnbbiapkmbliob/ --parent-window=0

It should have worked at this point, but it still wasn’t, so I had to go further into debug mode. The full copy of the C source code can be found at the bottom of this post. Make sure to replace USERNAME in the source (and instructions in this post) with your username, as using “~” to specify your home directory often doesn’t work in this setup. You may also need to replace version numbers (9.6.1.1 for this post).

First, I created a simple C program to sit in between the chrome extension and the rf-chrome-nm-host.exe. All it did was forward the stdin/stdout between both programs (left=chromium, right=rf-chrome-nm-host.exe) and output their crosstalk to a log file (./log) that I monitored with tail -f. I pointed to the generated executable in the ~/chrome-robo.sh file.

All that was generated on the Linux config was: left=ping, right=confirm ping, left=get product info, END.

I then modified the program to specifically handle chrome native messaging packets, which are always a 4 byte number specifying the packet length, followed by the packet data (IsChrome=1). If this variable is turned off there is a place in the code where you can set a different executable with parameters to run.

Next, I ran the program in Windows with a working left+right config so I could see what packets were expected.

I then added a hack to the program (AddRoboformAnswerHacks=1) to respond to the 2nd-4th packets sent from the left (get-product-info, get-product-info, Initialize2) with what was expected from the right. rf-chrome-nm-host.exe crashed on the 5th packet, and debugging further from there would have taken too more time than I was willing to put in, so at that point I gave up.

I next decided to see if I could get the chromium extension to play nicely by talking directly to the rf-chrome-nm-host.exe on my Windows machine via an SSH tunnel. To do this, I changed the char *cmd[]= line to:

char *cmd[]={
  "/usr/bin/ssh",
  "USERNAME@HOSTNAME",
  "cd c:/Program\\ Files\\ \\(x86\\)/Siber\\ Systems/AI\\ RoboForm/9.6.1.1; ./rf-chrome-nm-host.exe chrome-extension://pnlccmojcmeohlpggmfnbbiapkmbliob/ --parent-window=0",
  NULL
};

While the first 4 packets succeeded in this setup, the following packets (rf-api-request) were all met with: pipe open error: The system cannot find the file specified. (error 2).

This was another stopping point because debugging this would also have taken too long. Though I did do some initial testing using Process Monitor and handle lookups in Process Explorer.

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/wait.h>
#include <string.h>

FILE *SideLog; //This logs all events to the file "log"
FILE *RecordLeft; //This logs the exact packets that the left side sends to the file "left"
int IsChrome=1; //True for roboform chrome compatibility, false to just pass through data as is
int DisplayStatus=1; //Whether to show the "Status: x" updates
int AddRoboformAnswerHacks=0; //Whether to force send fake responses that the rf-chrome server is not responding to
const char* HomePath="/home/USERNAME/.wine/drive_c/Program Files (x86)/Siber Systems/AI RoboForm/9.6.1.1/"; //This must be set.
const char* ExePath="/home/USERNAME/.wine/drive_c/Program Files (x86)/Siber Systems/AI RoboForm/9.6.1.1/rf-chrome-nm-host.exe"; //This must be set. You can make sure rf-chrome-nm-host.exe is running with a 'ps'

//Send back a custom packet to the left. This will show in the log file BEFORE the packet it is responding to.
void SendLeft(char *str)
{
	char Buffer[255];
	int StrSize=strlen(str);
	*(int*)Buffer=StrSize;
	strcpy(Buffer+4, str);

	write(STDOUT_FILENO, Buffer, StrSize+4);
	fprintf(SideLog, "OVERRIDE REQUEST: %s\n\n", Buffer+4);
	fflush(SideLog);
}

//Forward data from left to right or vice versa. Also saves to logs and "left" file
void ForwardSide(int InFileHandle, int OutFileHandle, fd_set* ReadFDS, int IsLeft)
{
	//Exit here if no data
	if(!FD_ISSET(InFileHandle, ReadFDS))
		return;

	//Create a static 1MB+1K buffer - Max packet with chrome is 1MB
	const int BufferLen=1024*(1024+1);
	static char* Buffer=0;
	if(!Buffer)
		Buffer=malloc(BufferLen);

	//If not chrome, just pass the data as is
	char* Side=(IsLeft ? "left" : "right");
	if(!IsChrome) {
		int ReadSize=read(InFileHandle, Buffer, BufferLen);
		write(OutFileHandle, Buffer, ReadSize);
		if(IsLeft)
			fwrite(Buffer, ReadSize, 1, RecordLeft);
		Buffer[ReadSize]=0;
		fprintf(SideLog, "%s (%d): %s\n\n", Side, ReadSize, Buffer);
		fflush(SideLog);
		return;
	}

	//Read the 4 byte packet size and store it at the beginning of the buffer
	unsigned int PacketSize;
	read(InFileHandle, &PacketSize, 4);
	*(unsigned int*)Buffer=PacketSize;

	//Read in the packet and zero it out at the end for the string functions
	read(InFileHandle, Buffer+4, PacketSize);
	Buffer[PacketSize+4]=0;

	//Send fake product-info packet since rf-chrome-nm-host.exe was not responding to it
	if(AddRoboformAnswerHacks && IsLeft && strstr(Buffer+4, "\"name\":\"getProp\"") && strstr(Buffer+4, "\"args\":\"product-info\"")) {
		//The return packet with header at the front
		char VersionInfo[]="{\"callbackId\":\"2\",\"result\":\"{\\\"version\\\":\\\"9-6-1-1\\\",\\\"haveReportAnIssue\\\":true,\\\"haveBreachMon\\\":true,\\\"haveLoginIntoAccount\\\":true}\"}";

		//Simplistic version counter hack since chrome always sends 2 version info requests at the beginning
		static int VersionCount=2;
		VersionInfo[15]='0'+VersionCount;
		VersionCount++;

		SendLeft(VersionInfo);
	//Send fake initialization info packet since rf-chrome-nm-host.exe was not responding to it
	} else if(AddRoboformAnswerHacks && IsLeft && strstr(Buffer+4, "\"name\":\"Initialize2\"")) {
		SendLeft("{\"callbackId\":\"4\",\"result\":\"rf-api\"}");
	//Forward the packet to the other side and store in the "left" file if left side
	} else {
		write(OutFileHandle, Buffer, PacketSize+4);
		if(IsLeft)
			fwrite(Buffer, PacketSize+4, 1, RecordLeft);
	}

	//Output the packet to the log
	fprintf(SideLog, "%s (%d): %s\n\n", Side, PacketSize, Buffer+4);
	fflush(SideLog);
}

int main(void) {
	//Create pipes
	int pipe1[2]; //Parent writes to child
	int pipe2[2]; //Child writes to parent
	if(pipe(pipe1)==-1 || pipe(pipe2)==-1) {
		perror("pipe");
		exit(EXIT_FAILURE);
	}

	//Fork the current process
	pid_t pid = fork();
	if(pid==-1) {
		perror("fork");
		exit(EXIT_FAILURE);
	}

	//New (child) process
	if(pid == 0) {
		//Close unused ends of the pipes
		close(pipe1[1]); // Close write end of pipe1
		close(pipe2[0]); // Close read end of pipe2

		//Redirect stdin to the read end of pipe1
		dup2(pipe1[0], STDIN_FILENO);
		close(pipe1[0]);

		//Redirect stdout to the write end of pipe2
		dup2(pipe2[1], STDOUT_FILENO);
		close(pipe2[1]);

		//Move to the roboform home directory
		if(IsChrome) {
			if(chdir(HomePath) == -1) {
				perror("chdir");
				exit(EXIT_FAILURE);
			}
		}

		//Execute a command that reads from stdin and writes to stdout. The default is the chrome command. If not in chrome, you can fill in the exe and parameter you wish to use
		char *cmd[] = {"/usr/bin/wine", (char*)ExePath, "chrome-extension://pnlccmojcmeohlpggmfnbbiapkmbliob/", "--parent-window=0", NULL};
		if(!IsChrome) {
			cmd[0]="/usr/bin/php";
			cmd[1]="echo.php";
		}
		execvp(cmd[0], cmd);
		perror("execlp");
		exit(EXIT_FAILURE);
	}

	//---Parent process - forwards both sides---
	//Close unused ends of the pipes
	close(pipe1[0]); // Close read end of pipe1
	close(pipe2[1]); // Close write end of pipe2

	//Open the log files
	SideLog = fopen("./log", "w+");
	RecordLeft = fopen("./left", "w+");

	//Run the main loop
	int max_fd=pipe2[0]+1; //Other pipe is STDIN which is 0
	while(1) {
		//Create the structures needed for select
		fd_set read_fds;
		FD_ZERO(&read_fds);
		FD_SET(STDIN_FILENO, &read_fds);
		FD_SET(pipe2[0], &read_fds);

		struct timeval timeout;
		timeout.tv_sec = 10;
		timeout.tv_usec = 0;
	
		//Listen for an update
		int status = select(max_fd, &read_fds, NULL, NULL, &timeout);

		//Display "Status: x" if its setting is true
		if(DisplayStatus) {
			fprintf(SideLog, "Status: %d\n", status);
			if(status==0)
				fflush(SideLog);
		}

		//Exit on bad status
		if (status==-1) {
			perror("select");
			break;
		}

		//Check both sides to see if they need to forward a packet
		ForwardSide(STDIN_FILENO, pipe1[1], &read_fds, 1);
		ForwardSide(pipe2[0], STDOUT_FILENO, &read_fds, 0);
	}

	//Close pipes
	close(pipe1[1]);
	close(pipe2[0]);

	//Wait for the child process to finish
	wait(NULL);

	return EXIT_SUCCESS;
}

var DataToSend=new FormData();
DataToSend.append(PostVariableName, VariableData); //Send a normal variable
DataToSend.append(PostFileVariableName, FileElement.files[0], PostFileName); //Send a file
var xhr=new XMLHttpRequest();
xhr.open("POST", YOUR_URL, true);
xhr.send(DataToSend);

function DownloadFile(
    FileURL,     //http://...
    Callback,    //The function to call back when the file download is complete. It receives the file Blob.
    ContentType) //The output Content-Type for the file. Example=image/jpeg
{
    var Req=new XMLHttpRequest();
    Req.responseType='arraybuffer';
    Req.onload=function() {
        Callback(new Blob([this.response], {type:ContentType}));
    };
    Req.open("GET", FileURL, true);
    Req.send();
}

//User Variables
var DownloadURL="https://www.castledragmire.com/layout/PopupBG.png";
var PostURL="https://www.castledragmire.com/ProjectContent/WebScripts/Default_PHP_Variables.php";
var PostFileVariableName="MyFile";
var OutputFileName="Example.jpg";
//End of User Variables

DownloadFile(DownloadURL, function(DownloadedFileBlob) {
    //Get the data to send
    var Data=new FormData();
    Data.append(PostFileVariableName, DownloadedFileBlob, OutputFileName);

    //Function to run on completion
    var CompleteFunction=function(ReturnData) {
        //Add your code in this function to handle the ajax result
        var ReturnText=(ReturnData.responseText ? ReturnData : this).responseText;
        console.log(ReturnText);
    }

    //Normal AJAX example
    var Req=new XMLHttpRequest();
    Req.onload=CompleteFunction; //You can also use "onreadystatechange", which is required for some older browsers
    Req.open("POST", PostURL, true);
    Req.send(Data);

    //jQuery example
    $.ajax({type:'POST', url:PostURL, data:Data, contentType:false, processData:false, cache:false, complete:CompleteFunction});
});

Unfortunately, due to cross site scripting (XSS) security settings, you can generally only use ajax to query URLs on the same domain. I use my Cross site scripting solutions and HTTP Forwarders for this. Stackoverflow also has a good thread about it.

So I was recently hired to set up a go-between system that would allow two independent websites to directly communicate and transfer/copy data between each other via a web browser. This is obviously normally not possible due to cross-site browser security settings (XSS), so I gave the client 2 possible solutions. Both of these solutions are written with the assumption that there is a go-between intermediary iframe/window, on a domain that they control, between the 2 independent site iframes/window. This would also work fine for one site you control against a site you do not control.

1. Tell the browser to ignore this security requirement:
   • For example, if you add to the chrome command line arguments “--disable-web-security”, cross-site security checks will be removed. However, chrome will prominently display on the very first tab (which can be closed) at the top of the browser “You are using an unsupported command-line flag: —disable-web-security. Stability and security will suffer”. This can be scary to the user, and could also allow security breaches if the user utilizes that browser [session] for anything except the application page.
2. The more appropriate way to do it, which requires a bit of work on the administrative end, is having all 3 sites pretend to run off of the same domain. To do this:
   1. You must have a domain that you control, which we will call UnifyingDomain.com (This top level domain can contain subdomains)
   2. The 2 sites that YOU control would need a JavaScript line of  “document.domain='UnifyingDomain.com';” somewhere in them. These 2 sites must also be run off of a subdomain of UnifyingDomain.com, (which can also be done through apache redirect directives).
   3. The site that you do not control would need to be forwarded through your UnifyingDomain.com (not a subdomain) via an apache permanent redirect.
      • This may not work, if their site programmer is dumb and does not use proper relative links for everything (absolute links are the devil :-) ). If this is the case:
        • You can use a [http] proxy to pull in their site through your domain (in which case, if you wanted, you could inject a “domain=”)
        • You can use the domain that you do not control as the top level UnifyingDomain.com, and add rules into your computer’s hostname files to redirect its subdomains to your IPs.

This project is why I ended up making my HTTP Forwarders client in go (coming soon).

Here are a few functions I’ve been finding a lot of use for lately. They are basically the JavaScript equivalent for PHP’s htmlentities and html_entity_decode. These functions are useful for inserting HTML dynamically, and getting values of contentEditable fields. These functions do replace line breaks appropriately, and HTML2Text removes a trailing line break.

var TextTransformer=$('<div></div>');
function Text2HTML(T) { return TextTransformer.text(T).html().replace(/\r?\n/g, '<br>'); }
function HTML2Text(T) { return TextTransformer.html(ReplaceBreaks(T, "\x01br\x01")).text().replace(/\x01br\x01/g, "\n").replace(/\n$/, ''); }
function ReplaceBreaks(TheHTML, ReplaceText) { return TheHTML.replace(/<\s*br\s*\/?\s*>/g, ReplaceText || ' - '); }

Sigh, I just realized after writing this post that I had already covered this topic... oh well, this version has some new information the other one is missing.

I find people very often asking me to move data from an Excel spreadsheet to a MySQL database, so I thought I’d write up the procedure I follow when doing so. This assumes no multi-line cells or tabs in the excel spreadsheet data.

1. You need a good text editor with regular expression support. I highly recommend EditPad Pro (a free version is available too), and will be assuming you are using it for the steps below.
2. Make sure all data in the Excel spreadsheet is formatted for SQL insertion, for example:
   To convert a date “mm/dd/yyyy” to SQL:
   1. Copy the entire row to your text editor
   2. Run the following regular expression replace:

      Find Text	Replace Text
      ^(\d+)/(\d+)/(\d+)$	$3-$1-$2

   3. Copy the text back to the spreadsheet row
3. Copy all the data into the text editor, and run the following regular expressions:

   Find Text	Replace Text	Explanation
   \\	\\\\	Escape backslash
   '	\\'	Escape single quotation mark
   \t	','	Change separators so that all values are encased as strings
   ^	('	Line prefix to insert a row and stringify the first value
   $	'),	Line suffix to insert a row and stringify the last value

4. Change the very last character on the last line from a comma to a semi colon to end the query
5. Add the following to the top of the file:

   SET NAMES 'utf8' COLLATE 'utf8_general_ci';
   SET CHARACTER SET 'utf8';
   TRUNCATE TABLE TABLE_NAME;
   INSERT INTO TABLE_NAME (Field1, Field2, ...) VALUES
   		

6. Make sure the file is saved as UTF8: Menu -> Convert -> Text Encoding -> (Encode the data with another character set ...) AND (Unicode, UTF-8)
7. Make sure the file is saved with Unix line breaks: Menu -> Convert -> To Unix (LF Only)
8. Save the file and run the following in your MySQL command line prompt to import it:

   \u DATABASE_NAME
   \. FILE_NAME
   		

There are of course easier solutions, but they can often be buggy, and I figured this is a good primer on regular expressions and simple data manipulation :-)

I had the need to pass a program’s [standard] output to a web browser in real time. The best solution for this is to use a combination of programs made in different languages. The following are all of these individual components to accomplish this task.

Please note the C components are only compatible with gcc and bash (cygwin required for Windows), as MSVC and Windows command prompt are missing vital functionality for this to work.

The first component is a server made in C that receives stdin (as a pipe, or typed by the user after line breaks) and sends that data out to a connected client (buffering the output until the client connects).

PassThruServer source, PassThruServer compiled Windows executable.

• This compiles as C99 under gcc:

  gcc PassThruServer.c -o PassThruServer

• Define “WINDOWS” when compiling in Windows (pass “-DWINDOWS”)

#include <stdio.h>
#include <malloc.h>
#include <fcntl.h>
#include <sys/types.h> 
#include <sys/socket.h>
#include <netinet/in.h>
#include <signal.h>

//The server socket and options
int ServerSocket=0;
const int PortNumber=1234; //The port number to listen in on

//If an error occurs, exit cleanly
int error(char *msg)
{
	//Close the socket if it is still open
	if(ServerSocket)
		close(ServerSocket);
	ServerSocket=0;

	//Output the error message, and return the exit status
	fprintf(stderr, "%s\n", msg);
	return 1;
}

//Termination signals
void TerminationSignal(int sig)
{
	error("SIGNAL causing end of process");
	_exit(sig);
}

int main(int argc, char *argv[])
{
	//Listen for termination signals
	signal(SIGINT, TerminationSignal);
	signal(SIGTERM, TerminationSignal);
	signal(SIGHUP, SIG_IGN); //We want the server to continue running if the environment is closed, so SIGHUP is ignored -- This doesn't work in Windows
	
	//Create the server
	struct sockaddr_in ServerAddr={AF_INET, htons(PortNumber), INADDR_ANY, 0}; //Address/port to listen on
	if((ServerSocket=socket(AF_INET, SOCK_STREAM, 0))<0) //Attempt to create the socket
		return error("ERROR on 'socket' call");
	if(bind(ServerSocket, (struct sockaddr*)&ServerAddr, sizeof(ServerAddr))<0) //Bind the socket to the requested address/port
		return error("ERROR on 'bind' call");
	if(listen(ServerSocket,5)<0) //Attempt to listen on the requested address/port
		return error("ERROR on 'listen' call");

	//Accept a connection from a client
	struct sockaddr_in ClientAddr;
	int ClientAddrLen=sizeof(ClientAddr);
	int ClientSocket=accept(ServerSocket, (struct sockaddr*)&ClientAddr, &ClientAddrLen);
	if(ClientSocket<0) 
		return error("ERROR on 'accept' call");

	//Prepare to receive info from STDIN
		//Create the buffer
		const int BufferSize=1024*10;
		char *Buffer=malloc(BufferSize); //Allocate a 10k buffer
		//STDIN only needs to be set to binary mode in windows
		const int STDINno=fileno(stdin);
		#ifdef WINDOWS
			_setmode(STDINno, _O_BINARY);
		#endif
		//Prepare for blocked listening (select function)
		fcntl(STDINno, F_SETFL, fcntl(STDINno, F_GETFL, 0)|O_NONBLOCK); //Set STDIN as blocking
		fd_set WaitForSTDIN;
		FD_ZERO(&WaitForSTDIN);
		FD_SET(STDINno, &WaitForSTDIN);

	//Receive information from STDIN, and pass directly to the client
	int RetVal=0;
	while(1)
	{
		//Get the next block of data from STDIN
		select(STDINno+1, &WaitForSTDIN, NULL, NULL, NULL); //Wait for data
		size_t AmountRead=fread(Buffer, 1, BufferSize, stdin); //Read the data
		if(feof(stdin) || AmountRead==0) //If input is closed, process is complete
			break;
		
		//Send the data to the client
		if(write(ClientSocket,Buffer,AmountRead)<0) //If error in network connection occurred
		{
			RetVal=error("ERROR on 'write' call");
			break;
		}
	}
	
	//Cleanup
	if(ServerSocket)
		close(ServerSocket);
	free(Buffer);
	
	return RetVal;
}

The next component is a Flash applet as the client to receive data. Flash is needed as it can keep a socket open for realtime communication. The applet receives the data and then passes it through to JavaScript for final processing.

Compiled Flash Client Applet

import flash.external.ExternalInterface;
import flash.events.Event;
ExternalInterface.addCallback("OpenSocket", OpenSocket);

function OpenSocket(IP:String, Port:Number):void
{
	SendInfoToJS("Trying to connect");
	var TheSocket:Socket = new Socket();
	TheSocket.addEventListener(Event.CONNECT, function(Success) { SendInfoToJS(Success ? "Connected!" : "Could not connect"); });
	TheSocket.addEventListener(Event.CLOSE, function() { SendInfoToJS("Connection Closed"); });
	TheSocket.addEventListener(IOErrorEvent.IO_ERROR, function() {SendInfoToJS("Could not connect");});
	TheSocket.addEventListener(ProgressEvent.SOCKET_DATA, function(event:ProgressEvent):void { ExternalInterface.call("GetPacket", TheSocket.readUTFBytes(TheSocket.bytesAvailable)); });
	TheSocket.connect(IP, Port);
}
function SendInfoToJS(str:String) { ExternalInterface.call("GetInfoFromFlash", str); }
stop();

var NewSock=new XMLSocket();
NewSock.onData=function(msg) { GetPacket(msg); }
NewSock.onConnect=function(Success) { SendInfoToJS(Success ? "Connected!" : "Could not connect"); }
SendInfoToJS(NewSock.connect(IP, Port) ? "Trying to Connect" : "Could not start connecting");

JavaScript can then receive (and send) information from (and to) the Flash applet through the following functions.

• FLASH.OpenSocket(String IP, Number Port): Call this from JavaScript to open a connection to a server. Note the IP MIGHT have to be the domain the script is running on for security errors to not be thrown.
• JAVASCRIPT.GetInfoFromFlash(String): This is called from Flash whenever connection information is updated. I have it giving arbitrary strings ATM.
• JAVASCRIPT.GetPacket(String): This is called from Flash whenever data is received through the connection.

This example allows the user to input the IP to connect to that is streaming the output. Connection information is shown in the “ConnectionInfo” DOM object. Received data packets are appended to the document in separate DOM objects.

JavaScript+HTML Source

var isIE=navigator.appName.indexOf("Microsoft")!=-1;
function getFlashMovie(movieName) { return (isIE ? window[movieName] : document[movieName]);  }
function $(s) { return document.getElementById(s); }

function Connect()
{
	getFlashMovie("client").OpenSocket($('IP').value, 1234);
}

function GetInfoFromFlash(Str)
{
	$('ConnectionInfo').firstChild.data=Str;
}

function GetPacket(Str)
{
	var NewDiv=document.createElement('DIV');
	NewDiv.appendChild(document.createTextNode(Str));
	$('Info').appendChild(NewDiv);
}

Next is an example application that outputs to stdout. It is important that it flushes stdout after every output or the communication may not be real time.

inc source, inc compiled Windows executable.

inc counts from 0 to one less than a number (parameter #1 [default=50]) after a certain millisecond interval (parameter #2 [default=500]).

./inc 10 #Counts from 0-9 every half a second

#include <stdio.h>
#include <stdlib.h>

int main(int argc, char *argv[])
{
	int NumLoops=(argc>1 ? atoi(argv[1]) : 50); //Number of loops to run from passed argument 1. Default is 50 if not specified.
	int LoopWait=(argc>2 ? atoi(argv[2]) : 500); //Number of milliseconds to wait in between each loop from passed argument 2. Default is 500ms if not specified.
	LoopWait*=1000; //Convert to microseconds for usleep

	//Output an incremented number every half a second
	int i=0;
	while(i<NumLoops)
	{
		printf("%u\n", i++);
		fflush(stdout); //Force stdout flush
		usleep(LoopWait); //Wait for half a second
	};
	
	return 0;
}

This final component is needed so the Flash applet can connect to a server. Unfortunately, new versions of Flash (at least version 10, might have been before that though) started requiring policies for socket connections >:-(. I don’t think this is a problem if you compile your applet to target an older version of Flash with the ActionScript v1.0 code.

This Perl script creates a server on port 843 to respond to Flash policy requests, telling any Flash applet from any domain to allow connections to go through to any port on the computer (IP). It requires Perl, and root privileges on Linux to bind to a port <1024 (su to root or run with sudo).

Flash Socket Policy Server (Rename extension to .pl)

#!/usr/bin/perl
use warnings;
use strict;

#Listen for kill signals
$SIG{'QUIT'}=$SIG{'INT'}=$SIG{__DIE__} = sub
{
	close Server;
	print "Socket Policy Server Ended: $_[0]\n";
	exit;
};

#Start the server:
use Socket;
use IO::Handle;
my $FlashPolicyPort=843;
socket(Server, PF_INET, SOCK_STREAM, getprotobyname('tcp')) or die "'socket' call: $!"; #Open the socket
setsockopt(Server, SOL_SOCKET, SO_REUSEADDR, 1) or die "'setsockopt' call: $!"; #Allow reusing of port/address if in TIME_WAIT state
bind(Server, sockaddr_in($FlashPolicyPort,INADDR_ANY)) or die "'bind' call: $!"; #Listen on port $FlashPolicyPort for connections from any INET adapter
listen(Server,SOMAXCONN) or die "'listen' call: $!"; #Start listening for connections
Server->autoflush(1); #Do not buffer output

#Infinite loop that accepts connections
$/ = "\0"; #Reset terminator from new line to null char
while(my $paddr=accept(Client,Server))
{
	Client->autoflush(1); #Do not buffer IO
	if(<Client> =~ /.*policy\-file.*/i) { #If client requests policy file...
		print Client '<cross-domain-policy><allow-access-from domain="*" to-ports="*" /></cross-domain-policy>'.$/; #Output policy info: Allow any flash applets from any domain to connect
	}
	close Client; #Close the client
}

1. Start the servers
   • In your [bash] command shell, execute the following

     Server/FlashSocketPolicy.pl & #Run the Flash Policy Server as a daemon. Don't forget sudo in Linux
     ./inc | ./PassThruServer #Pipe inc out to the PassThruServer

   • Note that this will immediately start the PassThruServer receiving information from “inc”, so if you don’t get the client up in time, it may already be done counting and send you all the info at once (25 seconds).
   • The PassThruServer will not end until one of the following conditions has been met:
     • The client has connected and the piped process is completed
     • The client has connected and disconnected and the disconnect has been detected (when a packet send failed)
     • It is manually killed through a signal
   • The Flash Policy Server daemon should probably just be left on indefinitely in the background (it only needs to be run once).
2. To run the client, open client.html through a web server [i.e. Apache’s httpd] in your web browser. Don’t open the local file straight through your file system, it needs to be run through a web server for Flash to work correctly.
3. Click “connect” (assuming you are running the PassThruServer already on localhost [the same computer]). You can click “connect” again every time a new PassThruServer is ran.

A client of mine wanted their website to have an applet that played streaming music from a SHOUTcast server. The easy solution would have been to just embed a Windows Media Player applet into the page, but that would only work for IE.

I thoroughly searched the web and was unable to find a Flash applet (or other solution) that already did this (and actually worked). Most of the information I was finding was people having problems getting this kind of thing working in Flash with no answer provided. After giving up on finding a resolution online, I decided to load up Flash and see what I could find from some tinkering.

Quite frankly, I’m shocked people were having so many problems with this. I started an ActionScript 2.0 project and put in the following code, and it worked right away in Flash CS3 (v9.0) with no problem:

var URL="http://example.shoutcast.castledragmire.com:1234/" //The URL to the SHOUTcast server
var MySound:Sound=new Sound(this);
MySound.loadSound(URL,true);

Unfortunately, once I exported the Flash applet and loaded it up in my browsers, it was no longer working. After a few minutes of poking around, I had a hunch that the SHOUTcast host might be sending different data depending on the [Browser’s] User Agent. I changed Firefox’s User Agent to “Flash” through a Firefox add-on (User Agent Switcher), and it worked :-D.

Once again, unfortunately, this was not a viable solution because I couldn’t have every user who visited the client’s web page change their browser User Agent string :-). The quickest solution at this point to the problem was to just create a passthrough script that grabbed the live stream on their server and passed it to the client. The following is the PHP script I used for this:

$streamname='example.shoutcast.castledragmire.com';
$port      ='1234';
$path      ='/';

header('Content-type: audio/mpeg');
$sock=fsockopen($streamname,$port);
fputs($sock, "GET $path HTTP/1.0\r\n");
fputs($sock, "Host: $streamname\r\n");
fputs($sock, "User-Agent: WinampMPEG/2.8\r\n");
fputs($sock, "Accept: */*\r\n");
fputs($sock, "Connection: close\r\n\r\n");
fpassthru($sock);
fclose($sock);

1. Setting the Flash Applet’s URL variable to the PHP file
2. Turning off PHP output buffering for the file. This can only be done through Apache or the php.ini depending on the server setup. This is very important, as if it’s on, the data will never get sent to the user.

The only problem with this method is that it taxes the server that is passing the data through, especially since it uses PHP... This kind of thing could very easily be done in C though (as a matter of fact, I will be writing a post on something very close to that very soon).

To continue the subject in my last post, these next cross-window bugs also derive from objects not being recognized properly when being passed between windows in JavaScript.

I needed the ability to dynamically run functions in the secondary window form the primary window where the parameters are taken from an array. Since a “function” from a secondary window is not seen as a function object from the primary window in IE, the apply member was not working.

I have included a fix for this below in the “RunFunctionInRemoteWindow” function, which is just a wrapper function in the second window that calls the apply function. This function manually copies the array through a for loop, instead of using slice, because in IE7 (but not IE8), the passed arrays were not seen as valid JSObjects, so the slice method (which is a standard method used for copying arrays by value) was not working.

<html><body>
<input type=button onclick="RunTest();" value='Click me when the second window has opened to run the test'>
<script type="text/javascript">

//Spawn the second window
var NewWindow=window.open('RemoteWindow.html');

//Run the test
function RunTest()
{
	LocalAlertString('This is an alert generated from the local window');
	NewWindow.RemoteAlertString('This is an alert generated from the remote window');
	alert('The local window alert function is of type function: '+(LocalAlertString instanceof Function));
	alert('The remote window alert function is of type function: '+(NewWindow.RemoteAlertString instanceof Function));
	LocalAlertString.apply(window, ['This is an alert generated from the local window through the APPLY member']);

	try {
	NewWindow.RemoteAlertString.apply(NewWindow.window, ['This is an alert generated from the remote window through the APPLY member. This will not work in IE because the remote window\'s function is not actually a function.']);
	} catch(e) { alert('The REMOTE APPLY failed: '+e.message); }

	NewWindow.RunFunctionInRemoteWindow('RemoteAlertString', ['This is an alert generated from the remote window through the FIXED APPLY function.']);
}

//Generate an alert in the local window
function LocalAlertString(TheString)
{
	alert('Local String: '+TheString);
}

</script></body></html>

<html><body><script type="text/javascript">
//Generate an alert in the remote window
function RemoteAlertString(TheString)
{
	alert('Remote String: '+TheString);
}

//Call functions in this window remotely through the "apply" member
function RunFunctionInRemoteWindow(FunctionName, Parameters)
{
	//Manually copy the passed Parameters since "Parameters" may not be a valid JSObject anymore (this could be detected and array.slice used if it is still valid)
	var ParametersCopy=[];
	for(var i=0;i<Parameters.length;i++)
		ParametersCopy[i]=Parameters[i];
	
	window[FunctionName].apply(window, ParametersCopy);
}
</script></body></html>

I was doing some research around April of 2009 on JavaScript interaction between web browser windows. I was doing this because web browsers are starting to split off each tab/window into separate processes/threads (Firefox is lagging in this), which can lead to some useful new implementations in the browser world, including multithreading. I wanted to explore the interaction between these windows to make sure there were no caveats that might creep up if I decided to take advantage of this.

The first one I found was that each browser window has its own instance of all of the base object classes, so prototypes do not carry over, and instanceof will not work as expected.

For example, if in WindowOne, you add a prototype to the Array class called IsArray, it is only accessible by arrays created in WindowOne. If you pass an array created in WindowOne into a second window, the prototype is still available on that one array (IIRC this was not true of some of the browsers at the time, but I tested again today, and it worked for IE8, Firefox3, and Google Chrome). Also, since the base object class in Window1 and other windows are not the same, an object created in Window1 and passed to another window will return false in a instanceof Object operation in that other window.

<html><body>
<input type=button onclick="RunTest();" value='Click me when the second window has opened to run the test'>
<script type="text/javascript">
Array.prototype.IsArray=true;
var NewWindow=window.open('RemoteWindow.html'); //Spawn the second window
function RunTest() { NewWindow.RunTest({}, [], new ExampleObject()); }; //Send the test data to remote window
function ExampleObject() { } //An example class
</script></body></html>

<html><body><script type="text/javascript">
function RunTest(AnObject, AnArray, AnExampleObject)
{
   var MyTests=[
      'AnObject instanceof Object',
      'AnObject.IsArray',                               //Object.prototype does not have this (Array.prototype does)
      'AnArray instanceof Object',
      'AnArray instanceof Array',
      'AnArray.IsArray',                                //This was added to the Array.prototype in the parent window
      'AnArray instanceof opener.Array',                //This example does not work in IE7 because opener.x cannot be properly accessed
      'AnExampleObject instanceof opener.ExampleObject',//This example does not work in IE7 because opener.x cannot be properly accessed
      'AnExampleObject instanceof ExampleObject'        //This test should error because "ExampleObject" does not exist in this window
   ];
   
   for(var i=0;i<MyTests.length;i++) //This runs each test like the following: alert("TEST: "+(TEST));
      try {
         eval('alert("'+MyTests[i]+': "+('+MyTests[i]+'));');
      } catch(e) {
         alert('Error on test "'+MyTests[i]+'": '+(e.hasOwnProperty('message') ? e.message : e.toString()));
      }
}
</script></body></html>

This is a modification of the utility I made in yesterday’s post, chunk, and most of the information mentioned in that post applies to this one too. This utility instead writes bytes from STDIN to a certain byte offset of a preexisting file. Below is the source code for the result, which I call chunkwrite (Windows Executable).

Chunkwrite writes bytes to a file at a given offset from STDIN. The parameters are:
1) The file to write to
2) The byte offset to write at (hex is supported like 0xA)

//Copyright 2009 by Dakusan (http://www.castledragmire.com/Copyright). Licensed under Dakusan License v2.0 (http://www.castledragmire.com/Misc/Software_Licenses/Dakusan_License_v2.0.php).
//See http://www.castledragmire.com/Posts/chunkwrite for more information

#define __LARGE64_FILES
#include <stdio.h>
#include <stdlib.h> //strtoull

#ifdef WIN32 //STDIN only needs to be set to binary mode in windows
	#include <io.h> //_setmode
	#include <fcntl.h> //_O_BINARY
#endif

typedef unsigned long long UINT64;
const UINT64 MaxSizeToRead=1024*1024*10; //The maximum number of bytes to read at a time to our buffer (Must be < 2^31)

UINT64 GetNumberFromString(const char* S) //Extract both hexidecimal and decimal numbers from a string
{
	bool IsHex=S[0]=='0' && (S[1]|32=='x'); //If string starts as 0x, then is a hex number
	return strtoull(S+(IsHex ? 2 : 0), NULL, IsHex ? 16 : 10); //Hex number starts after 2 characters and uses base 16
}

int main(int argc, char *argv[], char *envp[])
{
	//Determine if proper number of parameters are passed, and if not, output help info
	if(argc!=3)
		return fprintf(stderr, "Chunkwrite writes bytes to a file at a given offset from STDIN. The parameters are:\n1) The file to write to\n2) The byte offset to write at (hex is supported like 0xA)\n") & 0;

	//Open the file to output to
	FILE *TheFile=fopen64(argv[1], "r+b");
	if(TheFile==NULL)
		return fprintf(stderr, "File not found or cannot open file\n") & 0;

	//Determine the requested start offset
	UINT64 Offset=GetNumberFromString(argv[2]);

	//Write the data 10MB at a time from STDIN to the file
	char *Buffer=new char[MaxSizeToRead];
	fseeko64(TheFile, Offset, SEEK_SET); //Seek to the beginning write offset of our file
	#ifdef WIN32 //STDIN only needs to be set to binary mode in windows
		_setmode(_fileno(stdin), _O_BINARY);
	#endif
	do
	{
		size_t AmountRead=fread(Buffer, 1, MaxSizeToRead, stdin); //Read the data from STDIN
		fwrite(Buffer, AmountRead, 1, TheFile); //Write the data to the file
	}
	while(!feof(stdin)); //Keep reading and writing until STDIN is complete

	//Cleanup
	delete[] Buffer;
	fclose(TheFile);
	return 1;
}

I needed a command line utility for Bash (for both Windows and Linux) that only outputs bytes between 2 points in a file to STDOUT. head and tail weren’t really cutting it so I figured I’d throw something together. Below is the source code for the result, which I call chunk (Windows Executable).

I compiled the file as c++, but it should be c99 compatible. The file has been tested as compilable for: GCC4 on Slackware, GCC3 on Red Hat, and GCC3 on MingW (Windows [WIN32 should be defined by the compiler]).

Chunk outputs bytes between 2 points in a file to STDOUT. The parameters are:
1) The file
2) The byte offset to start at (hex is supported like 0xA)
3) The number of bytes to output. If not given, the end of the file is assumed.

//Copyright 2009 by Dakusan (http://www.castledragmire.com/Copyright). Licensed under Dakusan License v2.0 (http://www.castledragmire.com/Misc/Software_Licenses/Dakusan_License_v2.0.php).
//See http://www.castledragmire.com/Posts/chunk for more information

#define __LARGE64_FILES
#include <stdio.h>
#include <stdlib.h> //strtoull

#ifdef WIN32 //STDOUT only needs to be set to binary mode in windows
	#include <io.h> //_setmode
	#include <fcntl.h> //_O_BINARY
#endif

typedef unsigned long long UINT64;
const UINT64 MaxSizeToRead=1024*1024*10; //The maximum number of bytes to read at a time to our buffer (Must be < 2^31)

UINT64 GetNumberFromString(const char* S) //Extract both hexidecimal and decimal numbers from a string
{
	bool IsHex=S[0]=='0' && (S[1]|32=='x'); //If string starts as 0x, then is a hex number
	return strtoull(S+(IsHex ? 2 : 0), NULL, IsHex ? 16 : 10); //Hex number starts after 2 characters and uses base 16
}

int main(int argc, char *argv[], char *envp[])
{
	//Determine if proper number of parameters are passed, and if not, output help info
	if(argc!=3 && argc!=4)
		return fprintf(stderr, "Chunk outputs bytes between 2 points in a file to STDOUT. The parameters are:\n1) The file\n2) The byte offset to start at (hex is supported like 0xA)\n3) The number of bytes to output. If not given, the end of the file is assumed.\n") & 0;

	//Open the file and get its length
	FILE *TheFile=fopen64(argv[1], "rb");
	if(TheFile==NULL)
		return fprintf(stderr, "File not found or cannot open file\n") & 0;
	fseeko64(TheFile, 0, SEEK_END); //Get the length by seeking to the end
	UINT64 FileSize=ftello64(TheFile);

	//Determine the requested start offset
	UINT64 Offset=GetNumberFromString(argv[2]), SizeToOutput;
	if(Offset>=FileSize)
	{
		fprintf(stderr, "Offset is larger than file's size\n");
		fclose(TheFile);
		return 0;
	}

	//Determine the size to read
	if(argc==3) //If no final parameter, read to the end of the file
		SizeToOutput=FileSize-Offset;
	else //Determine from the 3rd parameter
	{
		SizeToOutput=GetNumberFromString(argv[3]);
		if(Offset+SizeToOutput>FileSize)
		{
			fprintf(stderr, "Requested size is larger than the file, truncating to end of file\n");
			SizeToOutput=FileSize-Offset;
		}
		else if(!SizeToOutput) //If nothing to output, exit prematurely
		{
			fclose(TheFile);
			return 1;
		}
	}

	//Output requested data 10MB at a time from the file to STDOUT
	char *Buffer=new char[SizeToOutput>MaxSizeToRead ? MaxSizeToRead : SizeToOutput]; //Only allocate as many bytes to our read buffer as is necessary
	fseeko64(TheFile, Offset, SEEK_SET); //Seek to the beginning read offset of our file
	#ifdef WIN32 //STDOUT only needs to be set to binary mode in windows
		_setmode(_fileno(stdout), _O_BINARY);
	#endif
	while(SizeToOutput) //Keep reading and outputting until requested data is complete
	{
		UINT64 SizeToRead=SizeToOutput>MaxSizeToRead ? MaxSizeToRead : SizeToOutput; //Number of bytes to read and write
		fread(Buffer, SizeToRead, 1, TheFile); //Read the data
		fwrite(Buffer, SizeToRead, 1, stdout); //Write the data to STDOUT
		SizeToOutput-=SizeToRead; //Decrease number of bytes we still need to read
	}

	//Cleanup
	delete[] Buffer;
	fclose(TheFile);
	return 1;
}

Since I just released my AJAX Library, I thought I’d post a useful script that uses it. The function CrossDomainGetURL below uses the AJAX Library to make requests across domains in Firefox. It takes one more parameter (not in order) than the AJAX Library's GetURL function, which is an array of domains to pull cookies from for the AJAX request.

function GetCookiesFromURL(Domains) //Return all the cookies for Domains specified in the Domains array
{
	var cookieManager = Components.classes["@mozilla.org/cookiemanager;1"].getService(Components.interfaces.nsICookieManager); //Requires privileges, which is granted in CrossDomainGetURL
	var iter=cookieManager.enumerator, CookieList=[], cookie; //The object used to find all cookies, the final list of cookies, and a temporary object
	while(iter.hasMoreElements()) //Loop through all cookies
		if(((cookie=iter.getNext()) instanceof Components.interfaces.nsICookie) && Domains.indexOf(cookie.host)!=-1) //If a cookie whose host matches one of our domains
			CookieList.push(cookie.name+'='+cookie.value); //Add it to our final list
	return CookieList.join("; "); //Return the cookie list for the specified domains
}

function CrossDomainGetURL(URL, Data, CookieDomains, ExtraOptions) //See AJAX Library GetURL function. CookieDomains is an array specifying what domains cookies are pulled from for the AJAX call. 
{
	//Access universal privileges in Firefox (Required to get cookies for other domains, and to use AJAX with other domains). This functionality is lost as soon as this function loses scope.
	try { netscape.security.PrivilegeManager.enablePrivilege("UniversalXPConnect"); }
	catch(e) { return alert('Cannot access browser privileges'); }

	if(CookieDomains instanceof Array) //If an array of domains is passed to get cookies from...
	{	
		ExtraOptions=((ExtraOptions instanceof Object) ? ExtraOptions : {}); //Make sure extra options is an object
		ExtraOptions.AdditionalHeaders=((ExtraOptions.AdditionalHeaders instanceof Object) ? ExtraOptions.AdditionalHeaders : {}); //Make sure extra options has an additional headers object
		ExtraOptions.AdditionalHeaders.Cookie=GetCookiesFromURL(CookieDomains); //Get cookies for the domains
	}
	
	return GetURL(URL, Data, ExtraOptions); //Do the AJAX Call
}

I am often asked to transfer data sets into MySQL databases, or other formats. In this case, I’ll use a Microsoft Excel file without line breaks in the fields to MySQL as an example. While there are many programs out there to do this kind of thing, this method doesn’t take too long and is a good example use of regular expressions.

First, select all the data in Excel (ctrl+a) and copy (ctrl+c) it to a text editor with regular expression support. I recommend EditPad Pro as a very versatile and powerful text editor.

Next, we need to turn each row into the format “('FIELD1','FIELD2','FIELD3',...),”. Four regular expressions are needed to format the data:

• Add the start of the query: “INSERT INTO TABLENAME VALUES”
• End the query by changing the last row's comma “,” at the very end of the line to a semi-colon “;”.

a	b	c
d	e	f
g	h	i

INSERT INTO MyTable VALUES
('a','b','c'),
('d','e','f'),
('h','h','i');

Sometimes queries may get too long and you will need to separate them by performing the “2 more steps to complete the query” from above.

After doing one of these conversions recently, I was also asked to make the data searchable, so I made a very simple PHP script for this.

This script lets you search through all the fields and lists all matches. The fields are listed on the 2nd line in an array as "SQL_FieldName"=>"Viewable Name". If the “Viewable Name” contains a pound sign “#” it is matched exactly, otherwise, only part of the search string needs to be found.

<?
$Fields=Array('ClientNumber'=>'Client #', 'FirstName'=>'First Name', 'LastName'=>'Last Name', ...); //Field list
print '<form method=post action=index.php><table>'; //Form action needs to point to the current file
foreach($Fields as $Name => $Value) //Output search text boxes
	print "<tr><td>$Value</td><td><input name=\"$Name\" style='width:200px;' value=\"".
		(isset($_POST[$Name]) ? htmlentities($_POST[$Name], ENT_QUOTES) : '').'"></td></tr>';//Text boxes w/ POSTed values,if set
print '</table><input type=submit value=Search></form>';

if(!isset($_POST[key($Fields)])) //If search data has not been POSTed, stop here
	return;
	
$SearchArray=Array('1=1'); //Search parameters are stored here. 1=1 is passed in case no POSTed search parameter are ...
                           //... requested so there is at least 1 WHERE parameter, and is optimized out with the MySQL preprocessor anyways.
foreach($Fields as $Name => $Value) //Check each POSTed search parameter
	if(trim($_POST[$Name])!='') //If the POSTed search parameter is empty, do not use it as a search parameter
	{
		$V=mysql_escape_string($_POST[$Name]); //Prepare for SQL insertion
		$SearchArray[]=$Name.(strpos($Value, '#')===FALSE ? " LIKE '%$V%'" : "='$V'"); //Pound sign in the Viewable Name=exact ...
			//... value, otherwise, just a partial patch
	}
//Get data from MySQL
mysql_connect('SQL_HOST', 'SQL_USERNAME', 'SQL_PASSWORD');
mysql_select_db('SQL_DATABASE');
$q=mysql_query('SELECT * FROM TABLENAME WHERE '.implode(' AND ', $SearchArray));

//Output retrieved data
$i=0;
while($d=mysql_fetch_assoc($q)) //Iterate through found rows
{
	if(!($i++)) //If this is the first row found, output header
	{
		print '<table border=1 cellpadding=0 cellspacing=0><tr><td>Num</td>'; //Start table and output first column header (row #)
		foreach($Fields as $Name => $Value) //Output the rest of the column headers (Viewable Names)
			print "<td>$Value</td>";
		print '</tr>'; //Finish header row
	}
	print '<tr bgcolor='.($i&1 ? 'white' : 'gray')."><td>$i</td>"; //Start the data field's row. Row's colors are alternating white and gray.
	foreach($Fields as $Name => $Value) //Output row data
		print '<td>'.$d[$Name].'</td>';
	print '</tr>'; //End data row
}

print ($i==0 ? 'No records found.' : '</table>'); //If no records are found, output an error message, otherwise, end the data table
?>