Grumble Grumble

By Andrew Smith

A long long time ago I decided to do an experiment. I ported my open source application ISO Master to Windows, and kept the port as proprietary software.

The experiment paid off, and I definitely made enough money to pay myself for the porting work, but that was then.

In december last year my webserver uploaded 12GB worth of the shareware windows version of ISO Master, typical month. The sales have been slumping off year over year, and last month they were under 20$.

Now that I have my new project (OSTD) which I’m excited about, and I’m going to need bandwidth for – I am looking for ways to save computer resources, and one of those precious resources is bandwidth.

Should I keep the demo version online? Money isn’t really an issue, these days I make peanuts from it. But I have an archivist mindset, I hate removing stuff from the internet, especially if that stuff works. So I am leery of just removing that file.

Also there are links to the Windows version on all sorts of websites, including for example this and this. Knowing that I get so much traffic via those websites helps me feel more proud of my accomplishments, and beeing a negative kind of guy that’s definitely important.

Perhaps when the OSTD is ready for release I will run a suite of performance tests and decide whether the bandwidth the windows users are sucking up is one of the problems I need to address.

By Andrew Smith

There are a number of articles and blog posts out there trying to explain scope and closures, but overall I’d say that a majority of them aren’t crystal-clear.

No sh**. I’ve been trying to figure out how scope works in JavaScript on-and-off for the last several years. I was never a committed JS programmer though so it wasn’t important enough for me to make sure I learn it once and for all.

Why is it that something as simple as scope is so complicated? That was a rhetorical question, I don’t really want to hear excuses. Last week I was working on some moderately advanced JS code and sure enough I spent hours finding impossible bugs caused by assignments that overwrote values in the wrong variables.

I have no choice but to suck it up, but this blog is a partial registry of my complaints, and this one was definitely worth mentioning.

By Andrew Smith

If you read the slightly older post and look at its screenshot and do some thinking – you might like me wonder this: given a bunch of JSON with multiple selections which can be modified in JavaScript using a form.. wait, modified using a form?

One of the nice things about json is you can just do myjson[x].whatever[whatever].12 = “ABC” and it works. But if all you have is a <select> element and an onClick handler – that’s not so straightforward.

You can store the string “[x].whatever[whatever].12″ in the value field of the <option> but sadly you cannot just do myjson”[x].whatever[whatever].12″ = “DEF”, that’s a syntax error.

I had to wonder and look for a while, I even found something called JsonPath, which I got really excited about until I realised it’s only for reading (what exactly is the point of it then?). Today I found the solution: eval!

So continuing the lame example above I would simply do this: eval(‘myjson['+x+'].’+whatever+’['+whatever'+'].’+12+’ = “‘+”DEF”+’”‘) Roughly speaking, I haven’t tested this line. But you get the point?

Now that works but luckily because I’m testing with a real pot file I found a bug – all the ‘\n’ literals in the strings are replaced with newlines by eval, which is not what I want. I found a solution that not only fixes that but I hope will prevent my evals from being hacked: I escape all the backslashes and the single quotes in the string before giving it to eval:

eval(jsPath + ” = ‘” + text.toString().replace(/\\/g,’\\\\’).replace(/’/g,”\\’”) + “‘;”);

Yey!

By Andrew Smith

PHP has this really neat function, json_encode(). It can take an object of whatever type, including my own class with child arrays/classes, and make a valid JSON string out of it. I was going to write this function myself but I found PHP already has it.

There’s one concern I have about it – it takes the entire object tree and makes json out of it, and all my member variable names end up as keys in the JSON. So without much difficulty you can look at the JSON and see exactly how I structure the data on the server in the PHP.

I’m not sure I really care but this makes me uneasy. It just doesn’t feel right. Maybe it will be ok.

By Andrew Smith

I got to implementing one of the primary use cases for OSTD – user uploads a template .pot file and gets a bunch of .po files with as many translated strings as possible.

From a design point of view this isn’t a big deal: parse the .pot into a data structure, make a query per string to the database, and save the results.

But wait, what if there is more than one translation (in the same language) per english string? I expect that to happen quite often, so I have to handle that case from the start.

What I decided to do was not create the po files but only create in-memory representations of them, send them over to the user as json, generate the entire body of the webpage from that json, allow the user to make his selections, and post the modified json back to the server.

Only then would I generate the actual po files for the user to download. This will save some disk space (not that I care much about that) but also it’s pretty interesting technically. Possibly this saved me quite a bit of work too – otherwise I’d have had to reparse a bunch of po files I myself have generated.

Here’s a snippet, the blue string has more than one option. All the tables with all their contents are JS-generated:

By Andrew Smith

I think I mentioned earlier how I almost started writing a website in C and quickly realised that wasn’t the right tool for the job, and switched entirely to PHP.

For the work I did today I needed a good set of data structures:

• A set of files
• Each with a set of english strings
• Each with a set of 0 or more translations

Pice of cake, right? Right, it would have taken me a half an hour in C, took me over six hours in PHP.

One of the problems was a surprise in scoping. Turns out in PHP there is no such thing as block scoping, and they forgot to mention that in the manual (isn’t it obvious there’s only per function and per file scope?). This created some very weird bugs that took a lot of printing to figoure out.

Then there’s the arrays. How can you have a programming language without 0-indexed arrays? PHP forces you to manage the indicees yourself, since their ‘array’ is actually a hash table. No vector either, not list, basically only a hash table. Can get used to it I guess but was it really so hard to have something more structured?

Then there are the classes. I needed to use them because there is no concept of a struct and the arrays are so retarded. I won’t go too much into it, let me just say that I suspect classes in PHP resemble classes in Perl. Sure you can have them, but don’t expect them to be easy to use.

All this complaining, you say, but didn’t it take you years to learn how things work in C, what’s the problem with taking the time to learn how things work in PHP? Well – I would accept that if it weren’t for the prevailing opinion that PHP is easy to learn. It isn’t: you can get started using it very quickly but its similarities to C in syntax make it harder (not easier) to learn it well.

And yeah, the same applies to JavaScript, but whatever. That’s for when I’m ready to do something with this same data I mentioned above in the browser. Gonna be fun :)

By Andrew Smith

In the last year I’ve had to use CPAN several times. Perl people are all fancy and insist on all the perl modules to have unit tests. Commendable, except when it doesn’t work.

At least half of the times I used CPAN the unit tests failed. CPAN smarty pants decides not to install modules that have unit tests that failed. That’s a good thing, right? Wrong.

The problem is that unit tests usually fail not because the software is broken but because the unit tests are broken. Writing unit tests is somewhat similar to writing design documents – generally speaking they are obsolete as soon as you’ve written them, they are never complete, and they don’t account for all the possible environments.

So when CPAN fails, what am I as a user supposed to do, go file a bug report and wait a few months for someone to possibly fix it? Be real. I just run cpan -f to force an install despite failed unit tests. And given that I’ve had to do that 50% of the time I wonder if the typical cpan user simply adds that flag in by default.

By Andrew Smith

I have this silly little website called Linux Stuff, where I post things I learn about Linux that I think might be useful for other people as well.

When a new article is posted I get an email about it, piece of cake using the PHP mail() function.

For some reason (I already forgot why) I now want a new post on this blog whenever a new article on Linux Stuff was added. It turned out to be not so hard.

First I found a file on the internet called remotepost.class.php, and edited that to look like this:

<?php
class remotePost
{
  private $client;
  private $wpURL = 'http://localhost/grumble/xmlrpc.php ';
  private $ixrPath = '../grumble/wp-includes/class-IXR.php';
  private $uname = 'newuser';
  private $pass = 'newpass';
  public $postID;
  function __construct($content)
  {
    if(!is_array($content)) throw new Exception('Invalid Argument');
    include $this->ixrPath;
    $this->client = new IXR_Client($this->wpURL);

    $this->postID = $this->postContent($content);
  }
  private function postContent($content)
  {
    $content['description'] =  $content['description'];
    if(!$this->client->query('metaWeblog.newPost','',$this->uname,$this->pass,$content,true)) throw new Exception($this->client->getErrorMessage());
    return $this->client->getResponse();
  }
}
?>

I made that file only readable by Apache, just in case, even though this is not a multi-user server. The newuser I created in the wordpress Users panel.

Then in my Linux Stuff PHP on submit I added these few lines of code:

    /* Send it to Grumble Grumble too */
    $content['title'] = $got_articlename;
    $content['categories'] = array('Open Source');
    $content['description'] = "New post added to Linux Stuff: $got_articlename.";
    try
    {
      $posted = new remotePost($content);
      $pid = $posted->postID;
    }
    catch(Exception $e)
    {
      echo $e->getMessage();
    }

Piece of cake! Now I can post interesting things either here in wordpress or on Linux Stuff, and LuckyYou can learn about them :)

By Andrew Smith

Ever since I learned how CGI works (a couple of lifetimes ago) I was bothered by the fact that the source code is accessible by the web server, and by extension – by anyone on the internet.

If your CGI module is properly loaded and configured then Apache will execute the files rather than display them, but if there is a problem loading your module then Apache will stupidly display the contents of your source code in the web browser.

This is a real problem because the source code usually includes credentials for your database and who knows what else.

With PHP it got a little better when it became almost completely integrated into Apache and it was very challenging to break PHP without breaking Apache too.

Today I thought – wait a minute. My php will interpret .php files, but what about all those .inc files? They are PHP yeah but with a different extension. Sure enough I looked at ostd/parsePoFile.inc in Firefox and the whole source is dumped right out.

It was an easy enough fix, adding a Files section to my httpd.conf, but come on guys! How hard would it have been to add .inc files to the default config? .ht* is there. Lame.

By Andrew Smith

If someone updates a po file with 100 translations – I need to figure out whether each translation is already in the database, and if not – insert it. The result looks like this (more about “looks” later):

This is just a snippet.

I am concerned that running an SQL query like this:

"SELECT Translation.TranslatedString FROM Translation,Language WHERE " .
"Translation.LanguageID = Language.LanguageID AND Language.LanguageCode = '%s' " .
"AND Translation.EnglishString = '%s'"

for every line uploaded might be a bit too much. Is it?

The prevailing wisdom on the internet seems to be that more than a few queries per page is too much. But I wonder if that’s for viewing content rather than uploading. Perhaps for uploading content this is not too bad. I mean I hope a lot of uploading is going to happen but that’s not the most common use case.

Regardless, I can’t think of an easy way to optimise this. I would have to spend a lot more time on the database design. Maybe when (if?) the site gets popular I will have the motivation to learn more fancy DB optimisations.