Grumble Grumble

By Andrew Smith

andrew@littlesvr:~$ uptime
13:26:34 up 1104 days, 10:14,  4 users,  load average: 0.61, 0.31, 0.21

I guess 1100 days doesn’t sound like a lot but 3 years does :)

It’s still the same machine I mentioned a year ago. It’s been running and running and running, and not crashing!

Theoretically 3 years is not a lot, but in the real world such uptime requires a combination of luck, great software, and good hardware. A power failure will kill it. A kernel update will kill it. A kernel that can’t handle the load will kill it. An admin who doesn’t know how to upgrade or restart services without a reboot will kill it. A bad fan will kill it. And mine is still up :)

Even cdot, which is running Slackware 9 (littlesvr is running 12.2) has only been up for 208 days today though I’m guessing it’s sitting in a real server room and has real admins taking care of it.

Yesterday (this was after the 3 year mark) I thought my server was finally about to die. When I realised what was going on there were 115 httpd processes running, I had 25MB of physical memory and 36MB of swap space left. Sendmail fell over, refusing to work with a load average over 15 (it got to 44). Sshd stopped accepting connections. imapd could barely serve requests. Interestingly Apache still worked.

I tailed the apache logs (this took me a half an hour, working entirely off swap is very slow) and saw nothing unusual. I have no idea what got into it, why the suicidal behaviour. In the logs there were only the typical 2-3 requests and 2-3 errors per minute. I tried to run apachectl status but that was taking too long. So I did the obvious, apachectl stop. After 5-10 minutes the harddrive light stopped blinking, and littlesvr breathed a sigh of relief.

As for me – I’m not really sure that I cared if it died. 3 years is a lot, I’m starting to get itchy to upgrade software (though it works perfectly fine) and to upgrade the hardware (512MB of RAM is too little). Even during normal operations I’m using almost all my RAM, and hopefully the load will be heavier in the future.

Maybe when the next Slackware comes out I’ll decide whether today’s 13.37 is a stable enough version for the long term.

By Andrew Smith

Here’s an OSTD feature I was really excited about and looking forward to implement: allow the user to put in some version control info for a project so that I can run a nightly cron job and pull any new translations that have been added to that project. Would have been a great feature, and it would have provided a self-improvement mechanism for the OSTD.

And then I went and read about the capabilities of Git and Mercurial. Did you know that unlike that archaic version control system SVN the new and fancy distributed version control systems do not allow partial clones of repositories?

In other words, I cannot pull from a public repository only the po folder with the po files. I have to clone the entire repository. There’s an option to only pull the latest revision, but even that can be huge.

I cannot possibly afford the disk space, bandwidth, or time necessary to clone and do updates on thousands of full repositories which I was hoping to have linked into the OSTD. It would have worked perfectly with SVN, but not with the new systems.

So now what? My feature has been erased from the design board by the growing popularity of Git. I’m going to have to find a different solution, perhaps I can arrange something with Debian who have some kind of mechanism for indexing po files inside their packages.

Git.

By Andrew Smith

I’ve started to run into serious performance issues with my SQL queries. I mentioned my concerns earlier, but now (still long before production time) I’m already experiencing clearly unacceptable performance.

I’ve added a couple of thousand translated strings to the database, and uploaded another PO file for the record. That would run the following query for each line in the PO file:

"SELECT Translation.TranslatedString FROM Translation,Language WHERE " .
"Translation.LanguageID = Language.LanguageID AND Language.LanguageCode = '%s' " .
"AND Translation.EnglishString = '%s'"

This I think is called a join, and my boring data structures and algorithms experience says it’s an n^2 algorithm. But I figured a couple of things:

1. This has got to be almost the most basic kind of SQL query you can write. Select from table1 where the foreign key in table1 is the same as the primary key in table2. Database 101? Why is MySQL not smart enough to cache the result of the subquery, and only repeat it if table2 is changed in the meantime? Was that really such a hard optimisation to make, or did they leave it slow on purpose to encourage better query design?
2. My training as a real software developer tells me to avoid race conditions, and despite the extremely low likelihood that the LanguageID for french is going to change during the runtime of this query, I should never assume, and rely on the DBMS for making that association.

I figured wrong. The query above, after I inserted enough records into Translation would take ages, my page would take 5 minutes to load during which time MySQL would use 97% of the CPU. Unacceptable.

I had to step over my reluctance of caching a key outside the DBMS and run a separate query to obtain Translation.LanguageID, and ended up with this query in my loop:

"SELECT Translation.TranslatedString FROM Translation,Language WHERE " .
"Translation.LanguageID = '%s' AND Translation.EnglishString = '%s'"

Works much faster. So much faster that I’ve almost stopped thinking about extra hardware for MySQL. Shame that I had to do this.

By Andrew Smith

Here’s something not many people working with PO files have run into. What happens when your english/translated string contains a <b> tag and you try to display that string on a webpage? Luckily I have one of those (in Asunder, where it’s actually a GTK formatting tag, not an HTML tag), so I ran into this problem already.

Piece of cake to fix, PHP has a function called htmlspecialchars() which will escape the special-meaning characters such as “<” in strings. I can use that function before sending my strings over as HTML.

Now what happens if my PHP sends the browser not HTML but JSON, and I construct the page using the data in that JSON? Any guesses? Nothing happens. Because I create a DOM text node and put the string into there – it just shows up with the “<” and ‘”‘ and all the other special chars.

Cool.

By Andrew Smith

Back in 2009 I wrote a post complaining about the needless complexities plural forms introduce to the i18n process. Now I ran into them again.

Working on the OSTD I have to make sure I work with all kinds of PO files, and that has to include PO files with plural forms. The format of PO files is not a standard, mostly beacuse there’s only one implementation for fully handling them (GNU gettext). Heck, it’s not even a spec. I was barely able to piece together an understanding of possible variations of the format using the gettext manual and looking at existing real PO files. That and the fact that the format is kind of loose and everything else I mentioned in the previous post made me hate plural forms even more than I did before.

My implementations of the PO file parser and writer do not handle plural forms. I am hoping that I’m ignoring them properly when reading, and it’s ok that they’re missing when writing. And I have no plans of supporting them in the future. I will use what little influence I have to convince app maintainers to not use them, or to get rid of them.

By Andrew Smith

A long long time ago I decided to do an experiment. I ported my open source application ISO Master to Windows, and kept the port as proprietary software.

The experiment paid off, and I definitely made enough money to pay myself for the porting work, but that was then.

In december last year my webserver uploaded 12GB worth of the shareware windows version of ISO Master, typical month. The sales have been slumping off year over year, and last month they were under 20$.

Now that I have my new project (OSTD) which I’m excited about, and I’m going to need bandwidth for – I am looking for ways to save computer resources, and one of those precious resources is bandwidth.

Should I keep the demo version online? Money isn’t really an issue, these days I make peanuts from it. But I have an archivist mindset, I hate removing stuff from the internet, especially if that stuff works. So I am leery of just removing that file.

Also there are links to the Windows version on all sorts of websites, including for example this and this. Knowing that I get so much traffic via those websites helps me feel more proud of my accomplishments, and beeing a negative kind of guy that’s definitely important.

Perhaps when the OSTD is ready for release I will run a suite of performance tests and decide whether the bandwidth the windows users are sucking up is one of the problems I need to address.

By Andrew Smith

There are a number of articles and blog posts out there trying to explain scope and closures, but overall I’d say that a majority of them aren’t crystal-clear.

No sh**. I’ve been trying to figure out how scope works in JavaScript on-and-off for the last several years. I was never a committed JS programmer though so it wasn’t important enough for me to make sure I learn it once and for all.

Why is it that something as simple as scope is so complicated? That was a rhetorical question, I don’t really want to hear excuses. Last week I was working on some moderately advanced JS code and sure enough I spent hours finding impossible bugs caused by assignments that overwrote values in the wrong variables.

I have no choice but to suck it up, but this blog is a partial registry of my complaints, and this one was definitely worth mentioning.

By Andrew Smith

If you read the slightly older post and look at its screenshot and do some thinking – you might like me wonder this: given a bunch of JSON with multiple selections which can be modified in JavaScript using a form.. wait, modified using a form?

One of the nice things about json is you can just do myjson[x].whatever[whatever].12 = “ABC” and it works. But if all you have is a <select> element and an onClick handler – that’s not so straightforward.

You can store the string “[x].whatever[whatever].12″ in the value field of the <option> but sadly you cannot just do myjson”[x].whatever[whatever].12″ = “DEF”, that’s a syntax error.

I had to wonder and look for a while, I even found something called JsonPath, which I got really excited about until I realised it’s only for reading (what exactly is the point of it then?). Today I found the solution: eval!

So continuing the lame example above I would simply do this: eval(‘myjson['+x+'].’+whatever+’['+whatever'+'].’+12+’ = “‘+”DEF”+’”‘) Roughly speaking, I haven’t tested this line. But you get the point?

Now that works but luckily because I’m testing with a real pot file I found a bug – all the ‘\n’ literals in the strings are replaced with newlines by eval, which is not what I want. I found a solution that not only fixes that but I hope will prevent my evals from being hacked: I escape all the backslashes and the single quotes in the string before giving it to eval:

eval(jsPath + ” = ‘” + text.toString().replace(/\\/g,’\\\\’).replace(/’/g,”\\’”) + “‘;”);

Yey!

By Andrew Smith

PHP has this really neat function, json_encode(). It can take an object of whatever type, including my own class with child arrays/classes, and make a valid JSON string out of it. I was going to write this function myself but I found PHP already has it.

There’s one concern I have about it – it takes the entire object tree and makes json out of it, and all my member variable names end up as keys in the JSON. So without much difficulty you can look at the JSON and see exactly how I structure the data on the server in the PHP.

I’m not sure I really care but this makes me uneasy. It just doesn’t feel right. Maybe it will be ok.

By Andrew Smith

I got to implementing one of the primary use cases for OSTD – user uploads a template .pot file and gets a bunch of .po files with as many translated strings as possible.

From a design point of view this isn’t a big deal: parse the .pot into a data structure, make a query per string to the database, and save the results.

But wait, what if there is more than one translation (in the same language) per english string? I expect that to happen quite often, so I have to handle that case from the start.

What I decided to do was not create the po files but only create in-memory representations of them, send them over to the user as json, generate the entire body of the webpage from that json, allow the user to make his selections, and post the modified json back to the server.

Only then would I generate the actual po files for the user to download. This will save some disk space (not that I care much about that) but also it’s pretty interesting technically. Possibly this saved me quite a bit of work too – otherwise I’d have had to reparse a bunch of po files I myself have generated.

Here’s a snippet, the blue string has more than one option. All the tables with all their contents are JS-generated: