Personal Blog Entry 2

Picking up where my last personal blog left off, finished the introductory week of DevOps on Friday with a very early finish at 1pm. This was so that the trainers could get away for a conference over the weekend. It was a Consultancy convention in Barcelona. Jealous that I'll have to wait a whole year for one I can go on. They even got to do a Hackathon! :o

Weekend: It's all fun and games till your healer sets you on fire

A portable Flux Capacitor! Isn't
Magiotech just amazing?

Finally got around to seeing Kubo at the cinema. Damn good film that met and exceeded my expectations. Always enjoy Studio Ghibli-esque animations. Original story too.

Speaking of bards, the DnD group had our first proper adventuring session. It was heaps of fun and I enjoyed playing a bumbling elf hacker named Dawn-Willow-Eustace-Eragon-Brook-Iris-Earthmore (or Dweebie for short :P ) Having L337 HAXX0R skillz in a fantasy setting is rather over-powered. Not that it is much of an advantage when your character is so incompetent they manage to get knocked out by the party's healer :D

To the Right is a (googled) artists conception of Dawn-Willow-etc's Hacking deck. I imagine it looks as old as this but grimier.

Why yes, our theatre DOES look like it's
wearing a hat

Week Three: Off to the theatre for Enterprise Architecture

Week was first week of studying Enterprise Architecture, a topic much more relevant to going into a company as a consultant. Finding out how a business works (and politely telling them all the things wrong with it) We were working  in the Lowry Theatre which is a rather nice place. Had lots of business terminology to learn and created dozens of diagrams. mapping the messy, complicated, ad-hoc rules of human organisations onto the much nicer structure of a computer program. Quite a tricky task involving mock interviews and avoiding all those pesky assumptions about business practises...

Big thing of the week was interviews with tutors pretending to be part of the business we were pretending to be consultants for. It was rather weird but I think our group did well, managing to deal with some of the more, 'eccentric', performances our tutors put on during interview.

My fondest memories of working in the Lowry will be those tea breaks, so many nice varieties and oh my, the biscuits, so good!

One last surprise at the end of the week was that last thing on Friday we were asked what specialisations we wanted to go into. Gareth wrote up the available specialisations on the board and, once he had explained what the hell a Mulesoft was, we made our choice by going to a corner of the room. Bit of a moot point for me as I had had my heart set on DevOps since first hearing about QA Consulting. Cloud development might have been interesting, a lighter, more fluffy and nebulous version of DevOps. Other option was Pega which I believe has strong ties into Enterprise Architecture.

Social time and drink(s)

Two mortal enemies fused into an unholy whole...

There were lots of social events (read: drinks) on this week with Dan, a fellow trainee (and our DM), having birthday drinks on Tuesday.
Then on Thursday we had another DnD session: we had planned to complete the first dungeon but events and inanimate objects conspired against us. Who knew arguments about tables and rope could get so heated?. :o Friday was the usual social evening with free drinks. Good fun, even if I ended up round at Dan's once more with most of the DnD group.

Because the week had been rather busy with work and socialising it was tough to get the technical blog completely finished on time. I got it submitted on Friday with most of the content there but it took a little bit more work to get all the web applets set up completely. Don't worry it's all finished now! Pretty pleased with the Bi-gram generation. Strong temptation to keep tinkering with it but next weeks technical blog will be on another topic :)

Weekend: Trip into town and museum-going
At the weekend had family come to visit so was busy doing things with them. Got lots of new useful things for the flat, went out for dinner and a chat with some family friends. Sunday was out at the Museum of Science and Industry. Brilliant place :D

Check out the rack on that Baby!

 There was a whole exhibit on Graphene which had information on some novel new uses being researched. Another cool exhibit was a replica of the very first stored program computer; the Manchester Small-Scale Experimental machine, affectionately known as the Baby. On the 50th anniversary of it's construction some computer enthusiasts created a working  replica of the machine which is now on display at the Museum.

It was a wonderful piece of kit with nothing but valves as logic components. The memory which could store a program? It is a Cathode Ray tube with a wire mesh across it! :D The code was easy to visualise as it was literally stored upon the screen display as an array of dots. The memory was only 1000 bits with values in 5 bit words. I got heaps of information from the exhibitor, including details of an online emulator. He was so impressed with my questions and interest that he suggested I volunteer to help out at the museum. I have to say I wouldn't mind doing that!




Week Four: Enterprise Architecture, the second half

Weekend ended up rather busy so I started the second week far too tired. We seem to have gotten through the most interesting parts of Enterprise Architecture and have just been working towards the final presentation and test. Few more lectures here and there which gives a little bit of a break from preparations. Alas this part of consultancy isn't something we can automate away :/

On Thursday there were some Welcome drinks to chat with other members of QA. Of course it was at the Dockyard, best bar in the Quays :D Nice of them to have a company tab on the bar but with exam and presentation looming nobody was really in the mood to drink to excess (perhaps according to the trainer's plan, sneaky!)

Finally Friday came around and we had a chance to prove ourselves. The exam was pretty solid, good thing I had revised well for it. The most stressful part actually was some eleventh hour tech troubles, obviously a practical example for why good Ops management and planning is essential!

Before the presentation the office had all the atmosphere of a funeral, I think the week-long wait  had been getting to all our nerves. The questioning after the presentations was brutally direct, just like real management. I feel my group handled it with aplomb though, we had some rather novel, well-thought out solutions. The mood lifted as each presentation was done though and I got to indulge in my annoying habit of asking questions at the end too :3

We were all in need of a drink (or several) after three hours of presentations. I was nodding off to sleep whilst still in the office, slept well that night. It's good to have a proper work-week routine, makes the weekends all the more sweeter.

Wrapping up

Flat: now with 100% more spaceship

Enterprise Architecture was interesting but I would really want to be given more training and the chance to work with an experienced team before undertaking it in the real world. Telling people that the way they do their jobs is wrong and what *you* think is best is not my strong point. Then again in the real world people are a little more complex than characters in a course exercise.

Anyway what I am really looking forward to is the specialisation training starting next week, this is where the properly challenging training should begin. Can't wait to be learning how to be a DevOps Puppet Master!

Oh and my flat is looking a little nicer now that I've unpacked all my goodies :)

Technical Blog Entry 1

For these technical blog posts I shall be doing a review of methods for:

- Procedural Generation -

What is Procedural Generation?

Instead of explicitly recording information an algorithm or procedure is used to create it when required. Whilst the phrase is now most commonly seen in reference to content generation in computer games it has been used by people long before the development of computers to produce interesting results.

The most basic form of procedural generation is to use randomness to determine somethings state. Imagine shuffling a pack of cards to play hearts of solitaire. The order that they end up in has been "procedurally" created in the shuffling and with have a large impact upon the game and whether it will be possible to complete.

Complete randomness is very noisy and not very interesting. The human mind is a powerful pattern finder and can identify and ignore data with no pattern at all. To avoid this the output can be structured and made to follow rules of generation. The created patterns and shapes will be much more appealing to the eye. Some examples of procedural generation are to the right, map generation algorithms that create landmarks and geographic features.

John Carmack famously disparaged Procedural Generation as simply another form of compression algorithm. (link) In some sense he was correct, procedural generation is fantastic for creating textures for organic surfaces like wood, stone or the swirling flux of a fire.
A well known example of 3D models and textures was the demoscene project kkrieger. Imagine a whole Doom-like shooter in just X KB!

In this first blog post I will be starting with some basics of procedural generation in the realm of text generation. To that end I have created several web applets which you can play around with to get procedurally generated output :D

I do hope they work ok, so long as you are using a modern browser (i.e. NOT Internet Explorer) they should function. It stands to reason that they won't work without JavaScript enabled!

Why Randomness is not enough

Random Stream of Letters

-Output of random letter script goes here-

The most naive way to generate text which is guaranteed to be novel, (that is non-repeating) would be to string letters together with no rhyme or reason to their order. Give the web applet to the side a try and see what it produces...

Doesn't look great does it? It's only a slight improvement over opening up a binary file as text and getting a garbled mess. The main issue is that words don't have normal lengths (~5 letters). Here a "word" might span multiple lines!

Another aspect of this output that instantly marks it out as artificial is that the punctuation characters are incorrectly placed. In real text they should only appear a sentence endings every 5-10 words. To get more useful output you require the generation algorithm to observe some aspects of the structure of real sentences.

Simple Example: Weighted Text Generation

Sentence Generator

-Output of sentence generation script goes here-

A marked improvement over purely random text can be observed if you use a pre-existing piece of text as your source. By reading through it and noting when a token (a word or letter) occurs and what follows it you can create a table of frequencies. By looking at these pairs of tokens (also known as bi-grams) you can create an algorithm to randomly pick the next token, weighted by how frequently the pair occurs.

The web applet above does so with three different source texts, using words as the tokens it tracks. Ignore the punctuation marks as they don't really work (ideally punctuation would be tracked as tokens too)

A neat example of procedural text generation is if you successively click on predictive text options on a smart phone. You might learn a lot about your most frequently used words and sentences!

What my phone's predictive text produces.
I, uh, talk about tech a lot...

Like many algorithms the output of this bi-gram analysis depends on the "correctness" of the input data. If you put in nonsense sentences you'll get the same out. Another property to consider is how tightly fitted the model is to the input training data. It can often be advantageous to have an aspect of noise or randomness to the algorithm to avoid repeating the sequences in the input data *exactly*

Whilst bi-grams can produce impressive results in terms of looking real-ish sentences composed of them tend to be meaningless; all structure and no content. Whilst it is still an open problem of how many rules go into producing real human language texts there is always the option of throwing more computing power after deeper frequency analysis.



While bi-grams are concerned with the immediately subsequent token there is nothing stopping an enterprising computational linguist measuring tri-gram (next two token) frequency or any n-gram frequencies. There is fascinating work being done in terms of word vectors at the moment. See here for tutorial: (www.tensorflow.org)

New Word and Name Generation

New Word Generator

-Output of new word generation script goes here-

In the other direction, instead of trying to replicate real languages procedural generation can create fantastic results when creating completely novel words and languages.

Finer grained procedural generation of novel words can be done similarly as the weighted sentence generation above. Instead of tracking frequency of word bi-grams you track pairs of letters. The final Web Applet gives and example for this along with consonant-vowel-consonant random generation for comparison. The source texts (also referred to as corpus) for this Applet are a bi-gram analysis from Google Book's English texts: (norvig.com/mayzner.html) and an equivalent for German texts from here: (www.staff.uni-mainz.de/pommeren/Cryptology/Classic/8_Transpos/BigramsD1.html)

A further step in creating new words would be to create the bi-gram frequencies procedurally. What would a language with 'z' as common as 'e' look like? Alternatively you could improve the 'englishness' of a text by looking at longer combinations of letters, triplets, quadruplets and so on (tri-grams, quad-grams or n-grams).

How many pieces has the Triforce
been split into??

Summary

I've enjoyed creating this first blog on procedural generation, especially the interactive Web Applets. I hope my enthusiasm has been well received by you the reader. Thank you for having a read-through and perhaps you've learnt something too!

I could happily continue on procedural text generation for another three weeks, alas there are many more topics to discuss. Next fortnight's technical blog post will be on generation of shapes. I'm looking forward to it. Perhaps I'll do fractals, Menger Sponges or tree generation (what is an L-System exactly?) :D

Personal Blog Entry 1

This blog is a recording of my time on the QA Consulting IT training course at the QA Academy in Manchester. Training began on the 5th of September so at the time of writing I have had two weeks of instruction, they're been pretty fun!

Anchorage building at Salford Quays, our offices are in the wing to the left

QA Consulting has offices for training in the Salford Quays area, just a stones throw away from Media City where the BBC and ITV have studios. Suffice to say the location is very nice what with the tram stops right outside the offices and the area crisscrossed with canals and green spaces.

The training course is up to 12 weeks and will cover various parts of Software Development. After training I'll be assigned a 2 year consultancy role at one of QA's clients.

Week One: Of flats and Java

Home base

Main order of business on arriving in Manchester was to secure proper accommodation, getting a three month lease, right when University terms were starting was proving to be a difficult task. Thankfully I found a place by Tuesday and moved in on the spot. The flat is in Old Trafford to the South so I'll be getting a bit of exercise each morning on the way to the tram stop. Perfect Excuse to play Pokemon Go :)

It was a great, busy week. Settled into Manchester and training. Java is very fun to return to and re-explore. Flat shows promise and landlord is friendly chap and a software developer too. The Java tasks we've been given are good fun, the final one is to implement the board game Battleship!

Enjoyed getting an intro to design patterns, better way to structure code to be more readable and flexible. Had to give a surprise presentation on the Flyweight  design pattern on Thursday but it went fine.

Friday there was a short test on Object Orientated Programming, no issues there. The test assignment was to make a Jukebox Application's Object hierarchy. If only we had access to online materials I could have used an ACTUAL Jukebox program I made previously! Started late in the week on the Battleship task, too much time spent messing around making a library for a default project GUI :P Still I was pleased with what I managed to create:

Battleship!

Weekend: DevOps and Dragons

On Friday the work day finished early for some office social time. Played an enjoyable game of Pirate dice and there was way too much free alcohol on offer.

It was very nice to get to lie in on Saturday, spent morning relaxing in bed. The DnD group actually got together for a session in the afternoon! I'll be playing as an incompetent magical hacker in future sessions. What wonderful typecasting.

Sunday I sorted out the bedding in my flat (finally) and chatted to parents a bit. Been getting through more of the Java textbook that I brought with me. Other than that the day has been a well earned rest.





Week Two: Where we're going we don't need GUIs!

First Monday of a full work week! Amusingly I forgot about the late start on Mondays so I was in at 9am (along with most of the DnD group :P) meant we got to go to Costa for proper coffee. The tutor Aaron is back and we're diving into DevOps (trans: setting up computer systems for serious bizness) It sure involves a lot linux scripting... Been learning some interesting things about Continuous Integration too, software development in the internet era is fascinating.

Best way to use a computer, honest!

For our projects this week we've been using virtual machines and installing software by hand on the terminal in Linux. Surprisingly easy and a bit of silly fun. Using the CentOS Linux Distro which seems nice to use; same graphical user interface, Gnome, as Debian and has useful preset software for server use. Very nice to return to Linux, lots of comfy terminal tools:

Also been playing around with logging into each others VM's using ssh and running some cheeky scripts. Good practical lesson in the importance of setting up network firewall rules >:D

I could get used to this DevOps work, very satisfying to write a script to have a Provisioning tool such as Vagrant setup a server the way you want. Explored using Jenkins (yay) and Jira (boo) to set up a build server. I'll have to look into setting such a build server up for some of my existing Github projects.

In closing

This training has gotten off to a strong start, we'll be going onto Enterprise Architecture next few weeks, Might have less coding involved but it will be fun to be doing some group projects.

The Landlord is finally moving into his room in the flat, meaning that the internet should get sorted out soonish. I'm getting tied of having to piggyback off BT Openzone to get my daily internets.

Pic to the left is of QAbootie, caught him on a wander round the canals. The whole place is great for water Pokemon. I've found the tram is quite good for catching Pokes too, goes at a slow enough speed that the game doesn't twig that you're not walking :D

Hello World!