GitHub link for the lazy

Gameplay link for the lazy, not guaranteed work after March 31, 2013

What Is It?

As mentioned previously, I built an online multiplayer-only version of Spacewar! using Erlang for all of the game logic and Cowboy for websocket communication with clients. The only client at present is a very simple HTML canvas one, here is a direct link.

Why?

The Basics

I wanted to validate a couple of choices for building simple things requiring:

• the communication of some universal state to clients on a rapid basis
• changing that state based on input from a variety of clients

Looks like a game, sounds like a game but applicable to a lot of other stuff.

Learning Erlang

Aside from my (so-far) abortive attempt to build a URL-shortener, I haven’t found an excuse to really dig into Erlang yet. At the same time, I have been thinking for a while about the possibility of simple games via websockets. The concurrency and responsiveness requirements for something like this make Erlang a natural choice.

More With Websockets

They’re still sort of stuck in standards-hell and probably not as good as they could be but I wanted to try them out in more depth. I didn’t consider comet for this because the nature of Spacewar requires a lot of small, rapid updates and I suspected (but cannot yet confirm) that the repeated connection setup overhead might negatively impact this.

In actual fact, my assumption/suspicion is extremely likely to be totally wrong for the scope of this project. It probably only makes a difference at an extremely high rate of messages (eSpacewar does a minimum of 20/second to each client) which I’m obviously not dealing with here.

Derp Cloud Derp

This one’s easy. I just wanted to see if a tiny EC2 node could actually handle it. It mostly can but the t1.micro it’s hosted on now tends to stutter occassionally.

What I Liked

Erlang

I could probably go on forever about this (but won’t here). A few specific points that I ended up really loving:

• Pattern matching permeates everything to the point that if/then/else just seems medieval.
• So easy to avoid shared state. You just don’t have any option (slight lie, go with it) so it makes things easier to keep clean.
• Clean, simple and minimal. The core is so small and the emphasis on recursion so high that you’re naturally pushed to composability (although you’ll likely find gross contradictions in my code).

I didn’t actually make much use of supervision so can’t comment on that yet (but love it in the Akka/Scala world).

Turns Out People Like It

Who knew? Simple gameplay, easy to drop in and low requirements. I’ve seen some great 10 player moments of mayhem. Would have liked to see more but I think it would make playing almost impossible due to the number of torpedoes flying around.

What I Screwed Up

This is a non-exhaustive list, see code and comments for more but these are the big things I think I could have done better. Please feel free to hit me up with agreements/disagreements and suggestions on Twitter or in the comments.

Unit Tests

There aren’t any. I pretty much just slammed through this project to get things working and learn what I could.

Collision Detection

Collision detection occurs after an entity (torpedo, player) has been moved and is based on points in space. This means that if players and/or torpedoes are moving quickly, they can appear to pass through each other. This could be solved by using line segments to see if paths cross.

A further issue is that the server side has no concept of the ship’s shape. It considers a ship to be a point with a certain bounding area and only registers a collision if two entities overlap.

handle_info

My gen_server and gen_fsm implementations (see space.erl and player.erl respectively for examples) could possibly be simplified a bit by using handle_info over handle_cast or specific states in some cases. More importantly, this may have made the move from basic processes to gen_server and gen_fsm a little easier.

I would most definitely welcome opinions for and/or against the above, I clearly still have a lot to learn.

Records

I ended up not using records (just haven’t looked closely at them yet) and used tuples everywhere. I suspect I may have been able to simplify some state by using these but am unsure of how this would impact some of my pattern matching use. I’m aware they’re an extension of tuples but don’t know if this enforces a certain structure. An area for futher exploration.

Client

I’m clearly not an experienced JavaScript developer and need improvement (the playing space size is hard-coded, bad). One of the areas I wanted to get into but didn’t is WebGL but I’ll leave that for later.

Conclusions

I put this into a repo at a fairly early state so you will be able to dig back and see the progression (it was much uglier before).

As a result of this project, I’m confident enough with Erlang (and Cowboy) to start applying it to some more advanced/involved things so I’m going to push on with some new projects. I’ll undoubtably be posting progress on those at some time soon.

The problems above, unfortunately, likely won’t get fixed. I’ve essentially learned what I set out to and eSpacewar has served well for that process. Feel free to fork/host/modify, I’d appreciate a link to this blog if you decide to host it (or some derivative) somewhere else.

Comments/criticism most welcome on Twitter and in the comments below. I’ll consider pull requests on GitHub but don’t really want to mess with this too much further (feel free to fork and fix/extend).

What’s Erlang Spacewar?

A multiplayer version of Spacewar! built in Erlang with Cowboy using websockets. For the moment, it’s quite basic and really intended to be a proof-of-concept. The idea was to see if it is even possible to use websockets for this sort of thing.

Right now I’m just trying to get a sense of how it performs on a tiny little EC2 VM so I’d be most grateful if you could grab a couple friends and jump in when you have a few minutes to spare. Fair warning: I fully expect it to fall over and die if there are more than 50 or so people in there but I guess we’ll find out.

I’ll give you the link to play, don’t worry, just a couple quick things…

How Long Will It Work For?

I’m going to keep it live (aside from crashes, etc) for at least a couple of weeks I think.

What You Need:

A browser that supports websockets and HTML canvas. Think recent versions of Chrome and Firefox, Safari should work.

Problems You Can Expect

It’s Ugly

And I know. No explosions yet, very minimal.

It’s Choppy Sometimes

It’s running on a t1.micro in Amazon EC2’s us-east-1 region.

Translation: it’s got less than a full CPU core (really just uses spare cycles from other bigger VMs) and it’s in Virginia so if you’re on the West coast like me, expect some lag.

Crashes/Termination

I’ll be messing with the code for the next week so do expect unavailability due to upgrades and crashes.

Please Do

Play it and tell me what you think

I know about choppiness and ugliness but if you find any other frustrating bits (or even stuff you love), please do tell me in the comments or on Twitter.

Share the link

But please share the link to this blog post – I’d like to give all test players the same warnings/notes I’m giving you.

Are You Going To Release The Source?

Yes. It’ll be reasonably open (probably BSD or MIT license) on my github account within a couple of weeks. I’m still cleaning things up and moving to gen_server/proper OTP instead of bare Erlang processes.

For the record, the repository will contain the grim history of almost all my mistakes. It’s been a learning experience.

Shut Up And Let Me Play!

Of course. Here’s the game, don’t forget to enter a username and click “Start Playing”: Erlang Websocket Spacewar!

Or: WTF, A LISP in Scala?

Project link for the lazy

How Did It Come To This?

Some background first. At my day job, we’ve ended up writing a lot of boilerplate for REST endpoints with Netty (it’s relevant, I’ll get to the LISP bit soon). While I don’t want to get into it too much as we’ll likely be open-sourcing the results, we didn’t go with bigger solutions (e.g. Jetty, Scalatra, Play) because when you do, you’re making one or more of the following trade-offs (at minimum) in favour of an easier API for writing HTTP stuff:

• lack of visibility into how concurrency is handled at all levels of your application
• much bigger artifacts to deploy (unless you’re a ProGuard ninja)
• dealing with someone else’s assumptions about how HTTP-involved code (not necessarily a full “web app”) should be written

That third point is fuel for a much longer post/many discussions well outside the scope of this one. I’m happy to drink beer and yell about it with you if you’re ever in Vancouver.

To be clear: the above points DO NOT MATTER if you’re writing something that’s going on decent hardware (big servers) and/or you’re only dealing with a few hundred concurrent connections with DB/cache/distributed communication involved (echo and static file servers are a completely different story). Even that still depends greatly on the nature of your app and where you’re running it. The first point can also be mitigated by diving into the source for whatever library/framework you select but that will only get you so far, especially if you’ve got a larger team. We went the direction that we did because we need to deal with thousands of concurrent inbound connections per server with lots of DB and cache interactions on M1 medium through M1 large EC2 VMs. In this situation transparency and simple configurability at each layer of the application are necessities.

Just in case: there are other reasons we didn’t go with Finatra but I’ve already gone on too long.

Getting To The LISP Bit

The upshot of everything I said above is that we needed a basic routing DSL/library on top of Netty. Since we need to provide both static and dynamic paths, it seemed natural to start looking at combinatorial parsers (if you’re curious, go read this post by Daniel Spiewak). After some initial fumbling with regex and possible path parsers, I decided to see what I could do with S-expressions. This was the initial result:

Once I had progressed a bit further, it seemed natural to see how easily I could add simple evaluation of basic math operators (+, –, / and *).

Short answer: easy

Result: I got sucked in and went overboard, clearly.

A Slight Tangent

Before I get into what was good and bad about my first LISP implementation experience, it bears briefly demonstrating the way these experiments have resulted in a solution for the original problem: we need a simple, transparent, configurable routing layer on top of Netty. Our library “Floor” was built with a single goal in mind – ruthless minimalism while staying DRY.

Here’s a very simple example of what Floor lets you do:

There’s a bit more to it than that but I’m sure you get the idea.

The Good Stuff

I got to dig a bit further into a couple areas that I haven’t before and they’re worth mentioning. If/when Floor goes open source, you’ll see a lot of the same stuff in it.

Partial Functions and Pattern Matching

Looking at the various traits that are bolted onto the BadLisp interpreter, you’ll see that pretty much everything is done via filtering lists of partial functions based on pattern matching. Both special forms (like “if-then-else”) and the simple built-in functions (e.g. +, –, car) are implemented this way. Check out Conditionals.scala for a brief example. I generally like this approach and for the most part find it clean but it does have some downsides which I’ll touch on shortly.

Implicit Classes

New in Scala 2.10, I’m sure they’ll be as contentious as implicit defs and vals were before. Regardless, I stuck one in for the Comparisons trait. I treat it a little bit like a type class in that it allows certain value-based methods to be used with classes BadLisp considers to be concrete values. I suspect based on some of the reading I’ve done that this is not precisely what is meant by type classes in Scala but I think it’s a step in the right direction for the moment.

The Bad Stuff

Not Enough Implicit Classes

Again, probably a contentious point. I do get the feeling, however, that I could have simplified even more with judicious use of these. It’s not something I’m going to go back and refactor for but something I’m going to keep top of mind for future stuff. Please do weigh in via comments or Twitter if you have strong feelings one way or the other about this though.

Partial Functions and the Symbol Table

Because my symbol table is simply a big list of partial functions that I match against, it’s a bit clunky to look up or alias things in the case of higher order functions (I just wrapped this one up this morning). I’m not especially proud of how functions and scope are handled at present but I think I’m going to leave well enough alone as I seriously doubt this is the last time I try to implement something like this.

Next

I won’t be working on this particular interpreter further but am happy to talk about anything you find deficient and/or interesting on Twitter or in the comments. Next steps for me are getting back to PawnShop and/or experimenting with some Actor and Disruptor implementations I think.

Roughly 10 years ago, Cameron and I were roommates. At the time, he was getting into computer security, talking about NOP sleds and shell code while I ranted about how the Intel memory model could prevent it if people used it correctly. Thankfully he talked me out of writing something antagonistic and a brief introduction to Intel memory management resulted.

There are some notable weaknesses including:

• no mention of how caching figures in
• little to no talk of actually securing things
• it’s IA32, so kind of out of date (bear in mind this was 10 years ago)

I intended some sort of followup that covered safer process address space construction using non-executable heaps but unfortunately never got around to it. Nevertheless, I figured it might be of interest to some people so share and enjoy.

Background

Since pretty much everything I do at my day job involves Akka and Scala, Erlang’s been on my radar for quite a while. Using RabbitMQ in current products and considering Riak to solve some nasty storage issues has of course heightened my interest in it as well. Thanks to @ujm and @tavisrudd I’ve also been hearing a lot about Clojure for the past year or so and it has (not so) mysteriously turned up in a few testing tools at the aforementioned day job due to @ujm’s unbridled love for it. As I’ve been trying to improve my own approach to and understanding of functional programming, it seemed both natural and frankly exciting to dive into something involving both.

The Pitfall

This brings up a minor pet peeve of mine: while evaluating something new and interesting, people often come up with trivial example projects and in the past, I’ve been possibly more guilty than most. This is something I want very much to avoid now. The problems I’ve seen in myself with using something too trivial to evaluate and learn new things are:

• performance assumptions are never grounded in reality when your code doesn’t do anything realistic. A perfect example is testing concurrent websocket connection limits that simply echo input (I did this with Play! 2.0 and I’ve seen some naive numbers from the node.js crowd mentioned lately as well).
• simple projects rarely expose some of the strange corners of a language or platform. Anyone who has extensively used Scala’s implicits or made extensive use of futures and promises will understand this.
• most importantly overly trivial projects won’t expose your own weaknesses and misunderstandings early on.

Some Goals

Given the above, there are a few specific challenges I’m looking to work on with this project in both Erlang and Clojure:

• unique ID generation without a central locking resource. I want to be able to establish total ordering without any central coordination. Loose temporal ordering is a nice to have.
• horizontal scale, add and remove additional resources with no coordination required beyond some configuration. As long as nodes can get in touch with the database, they should continue to function.

And I have a few goals specific to Clojure and Erlang:

• learn to use their build tools (Leiningen and Rebar)
• loosen my reliance on object orientation
• get more comfortable with dynamic typing (I tend to yell a lot about the supremacy of static typing).
• experiment with a heterogeneous solution. I’ll be building the same thing in both languages so I think some interesting comparisons could arise with regard to things like deployment, configuration and performance.

Naturally I’m going to make a few rules for myself, the first and most important being that I’m not going to hide my mistakes. When I change my mind or screw up, all the details will be in the commit history. Second, the whole thing will be open source so that anyone will be able to see my pace, changes and progress in github at any time. Lastly, I’ll welcome feedback and optimization suggestions(with credit given where credit is due) but won’t accept pull requests or massive rewrites from anyone – this is mine to learn and mess up.

The Actual Project

I’m going to build a simple URL shortener and image hosting service called PawnShop with the following basic features:

• exchange a URL for a short hash
• redirect short URLs to originals
• upload an image (png/jpeg) in return for a short hash
• redirect HTTP requests for a URL hash to the original
• display the matching submitted image for a given requested hash
• track click stats for each hash generated. I’ll store these at millisecond resolution.
• display hosted images inside a minimal template
• provide a REST API to retrieve click stats, likely report by one hour intervals at the smallest at first.
• basic authentication via email address and password

The nice-to-have features will be:

• click stat reports/graphs with D3.js
• pluggable authentication

The basic technical bits:

• This will be a single monolithic project, no interdependent services.
• It will be heterogeneous in that I want to be able to run the Erlang version side by side with the Clojure version in the same cluster, off the same database, behind the same load balancer and things should Just Work ™.
• I’m going to use standard web frameworks, likely Noir and either Chicago Boss or Nitrogen
• The only data store I’ll be using is Riak.

Following Along

I’m going to aim for roughly a post-per-week here. Blog posts related to PawnShop will essentially be a notebook/journal of my reasoning for certain decisions as well as both negative and positive experiences throughout the learning and development process.

GitHub

Trello

Riak and the Data Model

A while ago, @jamesgolick told me I wasn’t allowed to use Riak until I had read the famous Dynamo paper in its entirety. He’s right, and you should read it too if you haven’t yet.

A URL shortener model seems a great fit for an eventually consistent Dynamo-style store like Riak.

• To shorten URLs and host pictures, you want high consistency for DB writes meaning that when a user submits new data, you want the write to complete with reasonable certainty.
• To retrieve URLs, low read consistency (small quorum) is OK. Since we’re not mutating URL and picture entries, we don’t have to worry about an update that might not have hit replicas yet.
• To add and retrieve click stats, low read and write consistency is fine. Missing one or two clicks out of thousands for a single report isn’t the end of the world, we just want reasonable accuracy when the stats are requested and eventually high accuracy. If it takes a few hours for this to happen, so be it.

Assumptions/Dependencies

statsd

Since I’m going to monitor the hell out of everything, I’ll assume something statsd-compatible is available wherever this project runs. These days I don’t think this is really an option at all for any sort of sane environment.

Next Steps

The next few posts will likely cover the following topics:

• decentralized ID/hash generation
• Riak model for shortened links and picture data
• Riak model for click stats

But those could change at any time.

Comments/questions always welcome on Twitter