Sorry for the lack of posts and updates recently. I’ve been involved lately with both playing FF4 on the DS and rereading Eragon, as the next book is about to come out. Among other things... I’ll start posting regularly again come the new month.
This is a clip from the TV show “Malcolm in the Middle” in which the protagonist, Malcolm, demonstrates his freakish numeric abilities for the Krelboyne [the advanced learning/gifted class] Circus to save the day (episode “Krelboyne Picnic” Season 1 Episode 8).
I encoded this video, apparently, in February of 2007 and do not recall why. It’s a fun little clip, so instead of deleting it, since I recall that I could not find it at the time for whatever reason, I figured I’d put it here.
I am often asked to transfer data sets into MySQL databases, or other formats. In this case, I’ll use a Microsoft Excel file without line breaks in the fields to MySQL as an example. While there are many programs out there to do this kind of thing, this method doesn’t take too long and is a good example use of regular expressions.
First, select all the data in Excel (ctrl+a) and copy (ctrl+c) it to a text editor with regular expression support. I recommend EditPad Pro as a very versatile and powerful text editor.
Next, we need to turn each row into the format “('FIELD1','FIELD2','FIELD3',...),”. Four regular expressions are needed to format the data:
Search
Replace
Explanation
'
\\'
Escape single quotes
\t
','
Separate fields and quote as strings
^
('
Start of row
$
'),
End of row
From there, there are only 2 more steps to complete the query.
Add the start of the query: “INSERT INTO TABLENAME VALUES”
End the query by changing the last row's comma “,” at the very end of the line to a semi-colon “;”.
For example:
a b c
d e f
g h i
would be converted to
INSERT INTO MyTable VALUES
('a','b','c'),
('d','e','f'),
('h','h','i');
Sometimes queries may get too long and you will need to separate them by performing the “2 more steps to complete the query” from above.
After doing one of these conversions recently, I was also asked to make the data searchable, so I made a very simple PHP script for this.
This script lets you search through all the fields and lists all matches. The fields are listed on the 2nd line in an array as "SQL_FieldName"=>"Viewable Name". If the “Viewable Name” contains a pound sign “#” it is matched exactly, otherwise, only part of the search string needs to be found.
<?
$Fields=Array('ClientNumber'=>'Client #', 'FirstName'=>'First Name', 'LastName'=>'Last Name', ...); //Field list
print '<form method=post action=index.php><table>'; //Form action needs to point to the current file
foreach($Fields as $Name => $Value) //Output search text boxes
print "<tr><td>$Value</td><td><input name=\"$Name\" style='width:200px;' value=\"".
(isset($_POST[$Name]) ? htmlentities($_POST[$Name], ENT_QUOTES) : '').'"></td></tr>';//Text boxes w/ POSTed values,if set
print '</table><input type=submit value=Search></form>';
if(!isset($_POST[key($Fields)])) //If search data has not been POSTed, stop here
return;
$SearchArray=Array('1=1'); //Search parameters are stored here. 1=1 is passed in case no POSTed search parameter are ...
//... requested so there is at least 1 WHERE parameter, and is optimized out with the MySQL preprocessor anyways.
foreach($Fields as $Name => $Value) //Check each POSTed search parameter
if(trim($_POST[$Name])!='') //If the POSTed search parameter is empty, do not use it as a search parameter
{
$V=mysql_escape_string($_POST[$Name]); //Prepare for SQL insertion
$SearchArray[]=$Name.(strpos($Value, '#')===FALSE ? " LIKE '%$V%'" : "='$V'"); //Pound sign in the Viewable Name=exact ...
//... value, otherwise, just a partial patch
}
//Get data from MySQL
mysql_connect('SQL_HOST', 'SQL_USERNAME', 'SQL_PASSWORD');
mysql_select_db('SQL_DATABASE');
$q=mysql_query('SELECT * FROM TABLENAME WHERE '.implode(' AND ', $SearchArray));
//Output retrieved data
$i=0;
while($d=mysql_fetch_assoc($q)) //Iterate through found rows
{
if(!($i++)) //If this is the first row found, output header
{
print '<table border=1 cellpadding=0 cellspacing=0><tr><td>Num</td>'; //Start table and output first column header (row #)
foreach($Fields as $Name => $Value) //Output the rest of the column headers (Viewable Names)
print "<td>$Value</td>";
print '</tr>'; //Finish header row
}
print '<tr bgcolor='.($i&1 ? 'white' : 'gray')."><td>$i</td>"; //Start the data field's row. Row's colors are alternating white and gray.
foreach($Fields as $Name => $Value) //Output row data
print '<td>'.$d[$Name].'</td>';
print '</tr>'; //End data row
}
print ($i==0 ? 'No records found.' : '</table>'); //If no records are found, output an error message, otherwise, end the data table
?>
The unfortunate reality of different feature sets in different language implementations
I was thinking earlier today how it would be neat for C/C++ to be able to get the address of a jump-to label to be used in jump tables, specifically, for an emulator. A number of seconds after I did a Google query, I found out it is possible in gcc (the open source native Linux compiler) through the “label value operator” “&&”. I am crushed that MSVC doesn’t have native support for such a concept :-(.
The reason it would be great for an emulator is for emulating the CPU, in which, usually, each first byte of a CPU instruction’s opcode [see ASM] gives what the instruction is supposed to do. An example to explain the usefulness of a jump table is as follows:
Of course, this could still be done with virtual functions, function pointers, or a switch statement, but those are theoretically much slower. Having them in separate functions would also remove the possibility of local variables.
Although, again, theoretically, it wouldn’t be too bad to use, I believe, the _fastcall function calling convention with function pointers, and modern compilers SHOULD translate switches to jump tables in an instance like this, but modern compilers are so obfuscated you never know what they are really doing.
It would probably be best to try and code such an instance so that all 3 methods (function pointers, switch statement, jump table) could be utilized through compiler definitions, and then profile for whichever method is fastest and supported.
//Define the switch for which type of opcode picker we want
#define UseSwitchStatement
//#define UseJumpTable
//#define UseFunctionPointers
//Defines for how each opcode picker acts
#if defined(UseSwitchStatement)
#define OPCODE(o) case OP_##o:
#elif defined(UseJumpTable)
#define OPCODE(o) o:
#define GET_OPCODE(o) &&o
#elif defined(UseFunctionPointers)
#define OPCODE(o) void Opcode_##o()
#define GET_OPCODE(o) (void*)&Opcode_##o
//The above GET_OPCODE is actually a problem since the opcode functions aren't listed until after their ...
//address is requested, but there are a couple of ways around that I'm not going to worry about going into here.
#endif
enum {OP_ADD=0, OP_SUB}; //assuming ADD=opcode 0 and so forth
void DoOpcode(int OpcodeNumber, ...)
{
#ifndef UseSwitchStatement //If using JumpTable or FunctionPointers we need an array of the opcode jump locations
void *Opcodes[]={GET_OPCODE(ADD), GET_OPCODE(SUB)}; //assuming ADD=opcode 0 and so forth
#endif
#if defined(UseSwitchStatement)
switch(OpcodeNumber) { //Normal switch statement
#elif defined(UseJumpTable)
goto *Opcodes[OpcodeNumber]; //Jump to the proper label
#elif defined(UseFunctionPointers)
*(void(*)(void))Opcodes[OpcodeNumber]; //Jump to the proper function
} //End the current function
#endif
//For testing under "UseFunctionPointers" (see GET_OPCODE comment under "defined(UseFunctionPointers)")
//put the following OPCODE sections directly above this "DoOpcode" function
OPCODE(ADD)
{
//...
}
OPCODE(SUB)
{
//...
}
#ifdef UseSwitchStatement //End the switch statement
}
#endif
#ifndef UseFunctionPointers //End the function
}
#endif
After some tinkering, I did discover through assembly insertion it was possible to retrieve the offset of a label in MSVC, so with some more tinkering, it could be utilized, though it might be a bit messy.
It's always a bit of a risk giving out email addresses
Since I just added my résumé which mentions my LinkedIn page, I thought I’d mention something else I just discovered about LinkedIn.
I would normally never give out any of my contacts’ email addresses to third parties under any circumstance, but I decided there was very little risk to do so at LinkedIn because it is a largely used website with many users that is also eTrust certified. Unfortunately, I have also heard eTrust certification isn’t exactly hard to get and shouldn’t have too much stock put in it, but it is still something.
Anyways, after reading LinkedIn’s privacy policy, I decided it would be ok to list some of my email contacts to discover if they also used LinkedIn or not. I, of course, added in a dummy email address of mine into this to watch for spam or advertisements, and it has to date not received anything, though I’m sure any company that illegally released email addresses wouldn’t be stupid enough to let go of newly acquired addresses immediately, but then again, I always assume too much of people/corporations... but I digress. I have discovered that they keep all the emails you give them because one of the emails I gave was recently used to sign up for a new account and LinkedIn immediately informed me of this.
While this is a nice extension to the "find your contacts through their emails" function, LinkedIn really should have given me an option to opt out of this, or at the very least informed me that it was keeping the emails I gave it on record. Unfortunately, even if they do have a good privacy policy and abide by it, there is still the chance a rogue staff member could harvest the emails and sell them.
Oh, LinkedIn is also a very buggy system in and of itself. I very often get timeout errors and many other errors to the extent of “The server cannot perform this operation at this time, please try again later”. A friend of mine has also been having trouble linking our profiles together for more than a week now, with no response to his email to them… besides a type of auto response he got back that had absolutely nothing to do with the reported problem.
A friend just asked me to write a PHP function to list all the contents of a directory and its sub-directories.
Nothing special here... just a simple example piece of code and boredom...
It wouldn’t be a bad idea to turn off PHP’s “output buffering” and on “implicit flush” when running something like this for larger directories. Example output for “ListContents('c:\\temp');”:
A.BMP [230]
Dir1 [D]
codeblocks-1.0rc2_mingw.exe [13,597,181]
Dir1a [D]
DEBUGUI.C [25,546]
Dir2 [D]
Dir3 [D]
HW.C [12,009]
INIFILE.C [9,436]
NTDETECT.COM [47,564]
I decided to make it a little nicer afterwards by bolding the directories, adding their total size, and changing sizes to a human readable format. This function is a lot more memory intensive because it holds data in strings instead of immediately outputting.
function HumanReadableSize($Size)
{
$MetricSizes=Array('Bytes', 'KB', 'MB', 'GB', 'TB');
for($SizeOn=0;$Size>=1024 && $SizeOn<count($MetricSizes)-1;$SizeOn++) //Loops until Size is < a binary thousand (1,024) or we have run out of listed Metric Sizes
$Size/=1024;
return preg_replace('/\\.?0+$/', '', number_format($Size, 2, '.', ',')).' '.$MetricSizes[$SizeOn]; //Forces to a maximum of 2 decimal places, adds comma at thousands place, appends metric size
}
function ListContents2($DirName, &$RetSize)
{
$Output='<ul>';
$dir=opendir($DirName);
$TotalSize=0;
while($file=readdir($dir))
if($file!='.' && $file!='..')
{
$FilePath="$DirName/$file";
if(is_dir($FilePath)) //Is directory
{
$DirContents=ListContents2($FilePath, $DirSize);
$Output.="<li><b>$file</b> [".HumanReadableSize($DirSize)."]$DirContents</li>";
$TotalSize+=$DirSize;
}
else //Is file
{
$FileSize=filesize($FilePath);
$Output.="<li>$file [".HumanReadableSize($FileSize).']</li>';
$TotalSize+=$FileSize;
}
}
closedir($dir);
$RetSize=$TotalSize;
$Output.='</ul>';
return $Output;
}
Example output for “print ListContents2('c:\\temp', $Dummy);”:
A.BMP [230 Bytes]
Dir1 [12.99 MB]
codeblocks-1.0rc2_mingw.exe [12.97 MB]
Dir1a [24.95 KB]
DEBUGUI.C [24.95 KB]
Dir2 [0 Bytes]
Dir3 [20.94 KB]
HW.C [11.73 KB]
INIFILE.C [9.21 KB]
NTDETECT.COM [46.45 KB]
The memory problem can be rectified through a little extra IO by calculating the size of a directory before its contents is listed, thereby not needing to keep everything in a string.
Of course, after all this, my friend took the original advice I gave him before writing any of this code, which was that using bash commands might get him to his original goal much easier.
On computers equipped with certain brands of audio cards, when performing the system encryption pretest or when the system partition/drive is encrypted, the sound card drivers failed to load. This will no longer occur. (Windows Vista/XP/2003)
It is possible to access mounted TrueCrypt volumes over a network. (Windows)
I am quite impressed that they did this so quickly, and am sad I did not find out until now. They also fixed the other missing feature I reported to them within a month of that [version 5.1]
Support for hibernation on computers where the system partition is encrypted (previous versions of TrueCrypt prevented the system from hibernating when the system partition was encrypted). (Windows Vista/XP/2008/2003)
Also in the version history [5.1a], this little paragraph made me smile
[Update 2008-04-02: Although we have not filed any complaint with Microsoft yet, we were contacted (on March 27) by Scott Field, a lead Architect in the Windows Client Operating System Division at Microsoft, who stated that he would like to investigate our requirements and look at possible solutions. We responded on March 31 providing details of the issues and suggested solutions.]
Other very important features they have added for version 6.0 that I am super happy about:
Embedded backup header (located at the end of the volume)
Up to 20% faster resuming from hibernation when the system partition/drive is encrypted. (As I have always been super frustrated by super slow hibernation resume support on my now abandoned partition encryption software suite, BestCrypt.)
Multithreading support (Faster parallel processing, yay)
I did some speed tests of hibernation support in XP and got the following numbers: (Results are averages of at least 5 tests, in seconds)
*VMWare was running with 256MB of RAM and 1 virtual CPU on Laptop**. VMWare results were not always stable due to other processes on the host machine, so I terminated the worst offenders
**Laptop is a 2.4ghz Pentium Core Duo with 2GB RAM and 60GB hard drive running at 7200RPM
The decoy (first) partition holds a decoy OS and is accessible from the password prompt (password #3) at bootup. You should not have any sensitive data in it, and can give out the password if need be. TrueCrypt recommends using this decoy OS at least as much as the hidden OS so if someone checks out the decoy they are not suspicious of it. If the perpetrator is suspicious of the decoy due to non use, the size of the partition, or just the fact that you have TrueCrypt installed, you may need to fall back onto the second stage of the security in the below paragraph.
The outer (second) partition holds some decoy files and a hidden volume inside of it. It is accessible by either the decoy or hidden OS by opening the partition through a normal TrueCrypt device mounting (password #1). It is recommended to give out its password only if you have already been forced to mount your decoy OS and the perpetrator suspects a secure partition as is explained in the above paragraph. If any data is written to it after creation, it can destroy information at random within the Hidden OS (see “Partition Sizes” at the bottom).
The hidden partition holds its own OS and is hidden within the outer (second) partition. It is accessible from the password prompt (password #2) at bootup or by mounting the partition from TrueCrypt as a device when the decoy OS is open. The decoy partition/OS is NOT accessible while the hidden OS is open.
Basic installation procedure:
Create a computer with 2 partitions. The second (outer) partition must be 5% larger than the first (decoy) for a FAT file system, or 110% (2.1x) larger for a NTFS file system (see “Partition Sizes” at the bottom). You might as well make the outer partition FAT since it won’t be used much, if at all, and this won’t affect the hidden partition.
Install your operating system on the first (decoy) partition with all of your applications and data that are not sensitive.
Run the TrueCrypt hidden install, this does the following:
Asks for outer volume password (Password #1). Creates and formats the second (outer) partition/volume.
Lets you copy some “sensitive looking” files to the outer partition. Nothing should ever be changed or added to the outer partition after this, see “Partition Sizes” at the bottom.
Asks for hidden volume password (Password #2). The hidden partition is created within the outer partition.
Asks for decoy volume password (Password #3).
Rescue disk is created
All data from the first (decoy) partition is copied to the hidden partition, and then all data from the first (decoy) partition is encrypted.
And finally, things that bugged me, because I like to vent :-) :
Forced creation of rescue disk on full volume encryption. Having the file is more than enough since it can be copied to other hard drives, but it wanted proof of the rescue disc creation, so I just mounted the ISO to a virtual drive.
No customized pre-boot screens. This isn’t important really, but I loved my hokie ASCII art ^_^;.
Partition sizes: The hidden OS partition will be the exact same size as the decoy and the outer partition must be at least 5% larger for FAT and 110% larger for NTFS than the decoy.
The hidden OS partition will be the exact size as the decoy partition because they are originally duplicates of each other, including their original partition tables, which include the size of the partition.
The outer (second) partition that holds the hidden partition must be at least 5% larger for FAT and 110% larger for NTFS than the decoy. The reason for this is the file contents tables. NTFS, unfortunately in this case, stores its file table in the middle of the partition. The outer partition’s file table does not, however, affect the hidden partition in any way.
So, for example (these numbers are theoretical, I am not entirely sure if these are correct), if we have a 2GB decoy partition, the outer NTFS partition must be at least 4.2GB and the hidden partition will be 2GB. If we made the outer partition 6GB, then 0-3GB would be writable, 3.0GB-3.6GB would be used for the file table, 3.6GB-4.0GB would be writable, and 4.0GB-6.0GB would be used by the hidden operating system. So, theoretically, you could write 3.4GB to the outer volume before problems started occurring, but I wouldn’t trust NTFS to only write to the beginning of the drive.
So I jumped on the bandwagon and switched over to Firefox 3.0 when it came out a week or two ago, and was pleasantly surprised after some forced addon (used to be called extension) updates that everything worked brilliantly, including my favorite plugin, Firebug. I meant to write this post containing the addons I use and recommend a long time ago (once again, jumping on the bandwagon as everyone else and their dog that has a blog has done this topic too...), but now is as good as ever, especially since there are some updates for Firefox’s new version.
This is a software suite that always you to store passwords and personal information in encrypted (against AES) container files against a master password, so it’s pretty darn secure. It interfaces well with both IE and Firefox, and really helps with the filling out of personal info on those long tedious credit card forms and such.
Notes: I just wish it worked better outside of web browsers in the Windows environment... maybe one day I’ll make something for that, it would be fun.
Allows you to go to a cache for the page you are currently on from one of the many caching services like Google Cache, Coral Cache, and archive.org’s Wayback Machine.
Notes: I modified this to allow you to open all cache sites at once and to work for Firefox 3... maybe one of these days I’ll release the additions.
“Embedding Internet Explorer in tabs of Mozilla/Firefox”
Notes: Since IE is sometimes a necessity when people refuse to conform to standards; and for developers to make sure things look right in the (unfortunately) most used web browser.
Go to parent directories of your current URL by clicking
Hide the protocol (ex: “http://”).
Notes: I originally used this because it fixed a major problem that plagued Firefox and still plagues IE in which the address bars show escaped URLs (like “Firefox%20Extensions” instead of “Firefox Extensions”), so foreign URLs, which used lots of non-ASCII characters were next to impossible to read. I submitted this to Mozilla a ways back, and fortunately it was fixed for Firefox 3. This, IMO, is one of the most important fixes for Firefox 3, and it wasn’t even really advertised.
“Allows you to open ANY file (executables, etc.) from the internet into the default program assigned by your operating system, without needing to save it first.”
Notes: This is not marked as compatible with Firefox 3, but works fine. Firefox has added an “applications” tab to its options dialog that kind of takes care of this, but this still does at least allow direct opening of all file extensions without also mapping them in Firefox.
Viewing how you browsed to pages through a referrer tree.
Notes: This is not compatible with Firefox 3, hasn’t been updated for ages, and is extremely slow as it uses a brute force method to build the referrer tree. I might see if I can find a better version of something like this (or make it) if the need ever arises again.
Toggle completely anonymous web browsing at the push of a button.
Notes: I found using the tor network way too slow, so I have since abandoned it for faster methods, which I will post about some day. Tor still remains an excellent “full-proof” way to stay anonymous on the internet though.
Notes: This addon hasn’t been updated in a long time... I’m not sure if it works with Firefox 3. To be honest, I don’t even remember what it does completely.
I have seen rather complex code out there for style sheet swapping in web browsers through JavaScript, and just found out a much simpler way works.
I could have sworn I tried to do real-time style sheet swapping a very long while back and none of my tests turned out satisfactorily, but a friend was just asking me about it, and I was redoing the tests, and it all worked out perfectly in an incredibly easy fashion in IE 6 & 7 and Firefox 2.5 & 3. All that needs to be done is swap the href of the link object pointing to the external style sheet file.