I always liked the web. I see little difference between writing PHP and C. But the powers of the masses of stuck-up computer science graduates won’t let the web be as it should: an incredible ecosystem of the latest great stuff built on awesome, stable technologies.

For the longest time I blamed only JavaScript – a mountain of hacks built on top of other hacks, ridiculous standards, complete disregard for efficiency of any sort, and a mockery of any long-standing-and-proven-to-be-working programming practices.

But today I started to realise that attitude applies not just to the development and design of JavaScript. I’ll give you a summary of the story:

Around 2013 I decided to build an Android app called Everyone’s Timetable (by the way – that Java .apk build still runs to this day). The app’s backend (written in PHP, also running unmodified to this day) needed a list of everyone’s current timetables (stored in MySQL, you guessed – with the schema not touched since then). Those timetables needed to be scraped off a website.

After searching high and low for a way to automate a web browser I found only one realistic option: iMacros. Everything else required months of setup and would guarantee at best a fragile solution. In that version of iMacros I could write a JavaScript program that would drive Firefox to different pages, wait for a load, navigate the DOM, extract the data I needed, and post it to my PHP web service. The process would take about two days. All the code worked great, the only problem was the website I was scraping – it would crash sometimes.

After about a year my screen scraping code stopped working. It took a lot of searching to figure out that the newest version of iMacros wouldn’t allow the old JavaScript code to run. So I got an older version if iMacros, which wouldn’t run on the current version of Firefox, so I had to get an older version of Firefox, which of course updated itself without asking me to the newest version, and disabled the old iMacros extension. The fault was with Electrolysis (which unapologetically broke iMacros for Firefox as well as every other Firefox extension I was using)

After a lot of headache I made a setup with and old version of Firefox (47.0.2), an old version if iMacros, a separate profile, and all updates disabled. That kept working until today, though it will stop working when that version of Firefox is declared obsolete.

Today I have a separate project that requires me to scrape a website. Do you think things got better? Of course you know they only got worse.

There was also Selenium, which from what I’m reading sort of kind of worked most of the time for some people to automate website testing. I went to use it today to find that – yeah, it was also broken.

But wait, there was Web Driver, developed internally by Google (who I guess like to test their websites) – but don’t use it, because:

Selenium has merged with Web Driver, to create something “better” but officially undocumented to this day:

NOTE: We’re currently working on documenting these sections. We believe the information here is accurate, however be aware we are also still working on this chapter. Additional information will be provided as we go which should make this chapter more solid.

I thought that makes sense, until I realised that the merger happened years ago.

There are downloads and tutorials for all of the above, from old iMacros to some kind of Selenium Web driver, but almost 100% of the software and tutorials are obsolete. I can’t even tell from the official Selenium website what’s supposed to work now.

I found a tutorial about setting it up – it’s a crazy combination of Eclipse, Selenium Client & WebDriver Language Bindings, and an officially unsupported Mozilla GeckoDriver.

I’ll probably try it, since I have little choice. But seriously, what the fuck!? Why is the web development field so full of people who don’t give a damn whether their software will work next year? Have they given up on taking pride in their work, or did they never learn what that pride feels like?

Bloody shame. So much potential never realised, and so much more of this garbage to come in the next decade. I don’t expect this will be fixed any time soon, in fact I see it spreading also via Android/iOS development and into Windows development as well.