Category Archives: Programming

Erik’s programming projects.

Uncharted Atlas

Posted on by

I discovered a new website that is doing random maps. The highlight is a twitter channel that has a new map every hour. And I must say, they look very nice. Martin O’Leary has even given the source code for his algorithms. I am now going to have to go through them and discover what he is doing.
It has already given me some ideas. In my maps, I use a grid for everything. He is using polygons. I can see advantages of both methods. I’ve been having some doubts about my grid system, but I’m thinking of a combination of the two.
With 3D graphics and video cards these days, I am probably not shackled to the tyranny of the grid anymore.
Martin is doing a good job of generating rivers and I want to look at how he is doing water flow. I’ve been planning a good method, but a brief look over his tutorial, and I find he is dropping the names of some algorithms I should know more about.
My biggest issue with his maps is that he treats sea level as the end result. Look at this:

There is a big lake, but it doesn’t have an outflow to the ocean. He is probably stopping all water flow as soon as it hits sea level. And that helps with quickly generating everything. But in reality there are freshwater lakes that have a depth that puts their bottoms below sea level. Those lakes still have outflows to the ocean, but their surface is well above sea level.

Actually, looking at his maps, there are no lakes at all. This is the closest:

They still feel like they are just another set of sea level lakes.

Religious Epics, both Modern and Classical

Posted on by

I am back to working on my father’s French epic manuscript index. I use “Epic” in the literal sense; stories about Charlemagne and his cast of characters.
I’ve gotten frustrated at getting my father to do some of the work, so I’ve decided to make an end-run around him and just try and do it myself. Which I probably should have done in the first place. In the past, I’ve gotten some of the data into a website, but that is mostly the actual libraries and the manuscripts. I still have to get the contents of those manuscripts.
I have all the files that he wrote thirty years ago. The first problem is that these files are written in something called “Waterloo Script”. There is nothing I’ve been able to find that reads those files or converts them to anything modern. I have not even been able to find documentation of it. So I will have to write a converter myself.
The files are basically text with a lot of codes to do different things. I have to extract the data out of it and eventually put it into SQL. However, I’m going to put it into XML as a middleman first and then use XSLT to convert as appropriate. That way I can convert to HTML easily to see how things are going. (I know that’s a lot of acronyms, but they make sense.) I am tempted to try and write something that will be generic and handle any Waterloo Script file, but that may be too ambitious.
My first task was to pick a language to do the parsing in. This was important.
I have most of my experience in C# or Swift these days, but those are full featured languages tied to Microsoft and Apple respectively. And I don’t know how long they will last, as they are specific to an operating system. I need to use an interpreted language that can be used on any machine. I was looking at this as a learning opportunity to get better with a language I don’t use much.
My first thought was Perl. It is an old language but still used and is well known for being good at text parsing. Sounds ideal. I even have the Camel book that is considered its bible. I haven’t looked at it in 15 years, but an old language doesn’t change much.
The other thought was PHP. It is a language more devoted to web servers and is what I will be using to serve the website eventually, but it can be used for parsing as well. It just isn’t known for it.
I could do it in Ruby which I’m learning, but that is VERY tied to web servers and will come with too much baggage that I don’t want.
I wasn’t sure which to use. I looked online for advice. But most articles seemed to be more religious than secular. People are very devoted to a language they have spent time learning.
For Christmas I wanted to start work on this project. But I didn’t want to take a laptop around with me during my vacation. I have a new iPad Pro with a keyboard; can I do work on that? I even have an app, Coda that should allow me to edit files. Unfortunately, the iPad doesn’t have a lot of compilers on it.
So, I loaded a whole bunch of files on to my web server. The plan was that I could use the iPad to edit the files remotely and then run them off my server. I wouldn’t be able to do development on an airplane, but who am I kidding? There is no way I would be THAT motivated.
I did some experiments to prove it could work. And I had a magnificent failure. The concept worked in practice and I was able to do exactly what I had theorized. However, I learned something else that changed everything.
In this day and age, being a programmer is fairly language agnostic. Languages have evolved a lot, but mostly in parallel. Many are derived from C/C++ and if you know the basics, it is easy to move from one to the other. (The valuable software development skills are more along the lines of being able to think properly.) That was the thought process I had going in. Although Perl was based off of C, it was nearly 30 years old. It did a lot of things in a way that no longer felt natural. It would not be easy to do development in it.
PHP was much closer to C/C++. That would be the better way to continue.
But then I realized, there is an even better option. If I wanted to use something not too foreign, why don’t I use Javascript? I’ve developed other things in it. I hadn’t thought of it before because it is a language for running off a client machine, not a server. But, it is powerful, and it will also run in any web browser. And my iPad has a web browser.
Using Coda, I was able to get the data on to the iPad and run it from there. Then I could actually use it on a plane if I wanted. The biggest issue was accessing the files wasn’t natural. Every other language had ways to open a read files from a file hierarchy. Because Javascript assumed it would be running on a client machine it could only access them from a URL, but I could work around that.
It has so far been working quite well.

Learning Cool Things

Posted on by

The company that I work for has occasional seminars for if you are interested in new technologies. Free to sign up. And if you want to actually use them for your own purposes, you can sign them out and go to town.
Yesterday I had the opportunity to learn about the Arduino system. Nominally it was to learn how to use Neopixels, but it was more learning about Arduino than anything else. Arduino is basically a way to create electronic devices. In the class we mostly controlled a strip of lights (the aforementioned Neopixels) but there are lots of things that you could use the Arduino for.
That is part of the problem. I wish I had something like this when I was a child. Back then I had an interest in electronics, but I never had an easy way to make anything. Or good ways to learn how to create anything. The Arduino would have let me experiment with things.
At this stage in my life, my creativity is shot. If I had the time to create devices, I don’t have any idea what I would want to make. If I was a child I could make completely stupid things. But right now, there is nothing electronic I need to make my life more interesting.
It’s kind of sad; to acknowledge the loss of youth. In such a geeky way.
Today though we learned about drones. It was an hour where we got an idea of how to fly a drone, what are the issues and problems, and what are the cool things you can do with them. Unfortunately it was too wet (Vancouver!) to actually take them out.
If I had the motivation, it would be awesome to have one and take video of people running an ultra marathon. The problem is that, in those cases, I’m usually busy doing the running. And I don’t think my company wants me give the drone to someone, untrained, outside the company.
And let’s be honest: I take a lot of pictures that I never do anything with. Why would video be any different. It would be cool to do, but wouldn’t go anywhere.
Next week is 3D printing, which I already have plans for how to abuse.

Unemployed Day One

Posted on by

Today is my first day without work.
Today I worked.
I had a project I’ve been wanting to get at. I recently tried to improve my knowledge of SQL. When I went to Pluralsight for training, I found their courses… wanting. They were fine courses, but they were at the wrong level; improvements to the server itself. Most of them could be summarized with “Turn on statistics and refresh often.” Their simple examples were fine for showing how the server does its thing. None were very good at giving tips on writing a better SQL in the first place.
I’ve written SQL that is hundreds of lines long. (That is a lot.) I’ve done optimizations which changed the run time of a report from minutes down to seconds. (That is also a lot.) I felt better qualified to write a course than what I found.
Which means I had to.
Today I finished it and put it up on my website.
It is dry. It is technical. But it is mine. And it will be useful for me to consult at whatever job I end up at.

Of course, I forgot to put my name on it. Then again, I haven’t put my name on anything on my website. That will be a project for tomorrow.

SQL Optimization

Posted on by

I’m the SQL guru at work, but I have no idea how I got to be that person. It just seemed to have happened. I know SQL better than a lot of other people. I’ve known some people who were better at it, and I followed their examples. Eventually, enough soaks into your brain that you gain new tools in your toolbox. Then you change jobs and you find that you are the local expert.
My work encourages me to continue self-development. They help do this by giving me access to Pluralsight so I learn more about technologies. Since I have doubts about my guru-ness of SQL, I thought I would take some courses on it. Mostly I wanted to learn how to optimize the queries. I’ve made some basic queries and watch them take minutes to perform the task. Then you make a few small changes and your result is back in under a second. I want to be more confident in my ability to do that.
The course titled “SQL Server: Optimizing Ad Hoc Statement Performance” seemed like it would do that. After about two hours in, I got the impression this is not actually going to help me optimize my SQL, but optimize how to construct the SQL so that it is cached properly. (I’ll finish the course so that I can be sure of my initial suspicion.)
I started to think that I could make a better course on SQL. Then I thought, why shouldn’t I? I know a lot of SQL, and I could write a document of all the tricks I’ve learned to improve the performance of it. Heck, that is what I’m frequently doing at work.
The biggest issue is one of self-doubt. I could write a lot of good tips. But I would always wonder if one of them was just wrong and thus negate the credibility of all of the other tips. I’m writing it out here to point out that that is silly. I really should write this document. At the very least so I can reference it myself after I’ve forgotten my tricks.

My empire grows

Posted on by

I gave fair warning to some friends that I would register their baby’s name as a website. I gave them 24 hours notice. I told them I was not joking. Domain names are cheap. They are priced to own. I didn’t really want it, but a threat will usually get people to get their act together.

Now what the hell do I do with borgtron.com?

In other news, I apparently still own rottentrailers.com. I really should do something with that. At the very least, stop parking it and point it to my own website. I originally planned to make a variation on www.rottentomatoes.com but on trailers. How much does this trailer spoil its movie?
I had another idea where I rate a movie, based off of how its trailer portrays it. For instance, Lost in Translation is a terrible comedy. But Transformers is an excellent giant-robots-beating-each-other-up movie.

Living epics

Posted on by

I got back to the work on the manuscript index. I finally finished the logic on the web pages. They all work and show what they need to. The links between them all are operating as expected.
So, let’s make this live!
I did some experimenting, and actually got it up on my website. There were a number of issues that I had to get through first.

  1. You need to actually put data in the database for the website to access. Simple enough, but it is an important step not to miss.
  2. The code for checking that parameters aren’t being made evil, actually requires a connection to the database first. It worked locally without it, but on the website, not so much. That was confusing, but a good lesson to learn.

I probably should place a link to it here, but I don’t feel THAT confident about it. The biggest reason I even did this, is so that my father, who is the author of all the data, can look it over. And that leads to the chief reason not to let people know about it: it doesn’t look right.
My father is preparing the data from ancient databases and word processing documents from twenty five years ago. (And that is prehistoric in computer terms, not just ancient) that lived on a mainframe computer. Since the programs and computer don’t exist anymore, a certain amount of translation has to happen. It hasn’t all been completed. And some of the data is oddly missing. I know there is a manuscript in Innsbruck, with a call number of “Codex 1342”, but I have no idea where in town to find it. There is a library “Oesterr. Nationalbibliothek (anc. d’un couvent d’Autriche)” in Vienna, with a single manuscript. (Did I mention that the site is supposed to be in French?) It does not have a call number, and I have no other information about it.
Hopefully, things will start to make more sense when we start getting the information about contents of each manuscript in. Plus we will get to list all the epics/legends that have been written.
Hint: Roland figures heavily.

Cleaning up after myself

Posted on by

Continuing on with being a creator, one of my first acts involved destruction.
A year ago, I started a second blog. The new one was devoted to my programming projects. After two entries, I never updated it. This seems like a waste.
Part of the reason to start the second one was to experiment with running my own blog. Since I have moved my main one from LiveJournal to hosting my own WordPress, that reason has fallen away. I can also use categories to mark posts that are programming related. I could even mark all posts that are not-programming. This way, readers (that would be you guys) could choose whether they wanted to hear about programming or not.
I’m still experimenting with that, so give me time.
In any case, I will shortly pull down the other blog. I’ve already transferred the two posts to this one and marked them appropriately.

Working on libraries

Posted on by

It wasn’t the authors page I was supposed to make, it was the Libraries page. Every manuscript is located somewhere and it seems the easiest part of the process to tackle, because it is simple. The other parts look like a headache, because each manuscript can have several parts that can each refer to a different epic, and they might be in poetry or prose.

It would be very convenient if every library in the world had a unique identifier associated with it. I could use that as the key to the database. Because it doesn’t, I have to make up my own that will only be true for this site. I’d rather use something an international body agreed on.

When my father wrote this, he wasn’t thinking in terms of database, so it is difficult to organize. It is also written in French, because it was dealing with Medieval French Epics. I don’t read French, so I can’t just look at what he wrote and figure out what is going on. It looks like a lot of the data uploading will need to be done by hand. Thankfully my father is onboard and looking through it.

I’m making progress on the PHP side. The hardest part is getting used to using a non-compiled language. If this was C++, it would happily inform me that I haven’t declared a variable, or I forgot a bracket. PHP is only processed when it is opened in a webpage. So if my code has an error, I won’t find out about it until after I run it. And then there seems to be a dearth of error messages. And if I accidentally misspell the name of a variable, the page will assume it is a different one.

Blech!

I should look into getting an IDE for this. That might make life easier.

Testing, testing…

Posted on by

Alright, first actual post.

The project I’ve decided to work on is for my father. He, long ago, wrote an index to Medieval French Manuscripts. He never finished it, and then went on to other things. I think that is a shame and something needs to be done about it. So, I’m going to see if I can.

At the very least, this looks like a good way to learn PHP and MySQL. I’ve already started the process by downloading MAMP. So I’ll be playing around with that and seeing what trouble I can get into.

First goal I think is to make an Authors page.