Kids and Programming Resources

I have a pretty precocious child when it comes to using a computer - which is no surprise.  Years ago, I read the essay in Freakanomics about the studies that showed having a house full of books produced readers (true).  Having a house full of computers has a similar effect -- it turns children into programmers.  Computers are curiosity machines and it is no fun to simply be a consumer of media when one can be a media creator.


In my house, Scratch ( has been king of the kid programming.  My daughter has made dozens and dozens of interactive displays, animations, and games.  It's not just a visual programming language, it's also a curated and safe kid online community where they swap code and comment on each other's programs.

Scratch is visual and a completely contained IDE within a browser.  Kids drag and drop components with variables, loop constructs, and input/output devices for keyboard and mouse to build up visual subroutines.  From there kids can add in graphics (costumes), sound files, and animations to build complete projects.

We used the book "Super Scratch Programming Adventure (covers v. 2)." In fact, our copy has disintegrated.  

However, as a warning: clever kids hit the hard ceiling of Scratch pretty hard.  My daughter has blown through almost everything she can do with Scratch (which is alot) and we had to go to her computer teacher for other ideas.


If you ask me, Python is the end-all be-all of scripting languages.  It can be as simple or complex as you need it to be (and with Python 3.4.1 and coroutines and futures it can get pretty complex!) However, moving kids from the highly intuitive drag-and-drop Lego-like interface of Scratch into the VIM and EMACS world turned out to be a hard sell.  I have tried out PyCharm as an IDE and that works okay but it's still not the same ease as Scratch.

We are test driving Wyliodrin, a GUI IDE in a browser which has visual blocks much like Scratch and interfaces with the Raspberry Pi, the Arduino, and a large number of standard programming languages like Python.  We're trying out some Pygame for Kids programming with it.  It feels promising as a bridge between Scratch and making real things.

I personally like Python for Kids: A Playful Introduction as a starter book for kids and adults because I have a fondness for the Pygame library.  We have also had quite a bit of success with Code Academy, believe it or not.  The free Python course there is just enough information to whet the appetite and get into the flow of basic Python programming.  


Processing (found at is an interesting and compelling idea for getting kids into programming with instant and tactile feedback.  Harder than Scratch and arguably more complex than beginning Python, Processing is a programming language built on the JVM and Java designed explicitly for artists.  It comes with a huge number of drawing and interaction primitives out of the box and works not unlike LOGO from the days of yore.  

Processing is a complete system from point of download.  Parents do not even need to install java on their systems.  And it has a little programming interface and an IDE.  Unfortunately, unlike Scratch or Wyliodrin, it is geared more toward those with some experience programming and does not have a "fun" drag and drop experience.

But Processing is ridiculously powerful.  With a couple of quick primitives kids can draw animations, algorithmic designs, interactive demonstrations with mouse and keyboard, read in and display visualizations of data, or make simple games.  The books are kind of sparse -- I personally have read the Processing starter book from Make and the Visualizing Data book from O'Reilly.  Sadly, Processing: A Programming Handbook for Visual Designers and Artists 2nd Edition is not yet available -- that looks like the Processing Holy Grail.

For incredibly precious programming kids who want to do some visual graphics and get into Arduino quick it's worth a look and incredibly easy to download and install.

Raspberry Pi

If you do want to get a kid into programming and you don't feel like shelling out $1500 for a brand new 13" Macbook Pro, I cannot recommend enough going to AdaFruit and picking up a Raspberry Pi B+.  For about $150 you can buy:

  • A fully functioning Linux-based computer full of Scratch and Python already loaded and ready to go and usable on WiFi;
  • A bright rainbow case;
  • A mouse and keyboard combination;
  • A small television monitor perfect for small spaces and small hands.

Everything a curious mind needs to get started programming is right there in the box.  Very little maintenance, installation, patching, or even Unix-knowledge required.  My experience with it was: plug in the SanDisk with the OS on it, turn it on, wait for it to boot, and start working.

Two Other Possible Options

For parents who are more hard core there is:

For the first one -- I am not familiar with the Javascript ICE Code Editor (GamingJS) but it looks phenomenally Processing-like.  Javascript is an interesting idea but Processing is a better designed teaching language.  I feel perhaps sticking with Processing is a better direction here since it has enormously powerful 3D graphics support.

The second one is an interesting idea because Minecraft is Java and that will get kids programming hard-core, no doubt about it.  But now we've gone into the world of modding games so now parents need to build a server, install a client with code, figure out the right modding tools (the book will no doubt help), stand up an IDE and a Java build chain and... yep.  It's for older kids, I think.

What about LEGO NXT?

I'm going to be honest -- I don't really know what shape NXT is in these days.  I feel like kids robotics has gone the way of the Arduino, processing, python, and duct tape.  We're about to try out littleBits which also interfaces with LEGOs and the Arduino and presumably the Pi.  So that's a less cheap but interesting direction to take things.

Also there's python and Blender for 3D graphics... the world is full of too many cool toys!

A Quick Conclusion

If kids are just starting out programming and they are 12 and under I recommend Scratch, hand's down.  Scratch will get kids programming fast.

If they have blown through Scratch and are looking for a bigger challenge, I do recommend getting a Pi and getting into Python.  Python opens the world to not just Pygame but all of the mods and hardware hacking that can go along with the Pi.  

Standing Up Agile at a Lean Startup

I left my previous gig for greener pastures when the grass under my feet started to look less green and more napalmed.  The new gig is small, lean, and full of super smart and motivated people burned in previous lives by failed old-school SDLC practices so was leaning toward the anarchy side of the project management spectrum.  But one cannot ship product without a plan and shipping good, solid, reliable product to customers is the #1 job of any company - be it software or a gaming company or what have you.  We need a method to get from point A to point B.  

As an architect, in my old age, I discovered interest both complex distributed systems and how human beings interact with those systems.  How can we tweak the knobs just enough to preserve tight-knit culture and hit targets and produce something everyone is proud to put their names on?  To me, project management is another system of inputs, outputs, pipes and filters - another way to architect the system to build a better machine. I had just come from a gig where Agile had managed to both spectacularly succeed and spectacularly fail -- some my fault, some others.  But I knew what parts of Agile brought value.     

There's two ways to look at project management: a way to push engineers into a system of metrics, control, and dehumanization and a way to foster communication and teamwork around a project while setting reasonable expectations.  Having been an engineer turned into a 'resource' bean-counted by management and then found myself sitting in meetings fighting over 'resources' - my friends - my desire to commoditize well-educated smart engineers is low.  Project management can be a force for evil but also a force for overwhelming good.  Solid project management with good communication and a feel of buy-in from everyone on the team can take a good team who knows where they want to go but not entirely how to get there to a great team that can climb mountains, land on mars, and change the world.  

Agile gives me:

  • A way to manage expectations for everyone -- engineers, stakeholders, customers.
  • A way to capture ideas for the product, put them somewhere, and have a reasonable way to talk about them, analyze them, and give them proper airing.  (Aka "Put it on the backlog.")
  • Keep a focused list of work to get to a working and great deliverable.
  • Keep everyone focused on the near-term goals.
  • Iterate features fast like crazy people toward a great product.
  • Fail fast.
  • Squeeze bad ideas out of engineering and replace them with better ideas while keeping the scope of the ideas manageable.  
  • Foster open communication.
  • Have short horizons so we can discussion the state of the project and change course when needed.

Well run Agile is magical.  I wanted to bring Agile in because I had seen it succeed.  I want to be sensitive to the culture in place.  I want it to be team-owned.  And I want this to succeed.  Synergy!  


Bringing in the big bang wall of multicolored 3x5 cards, the giant pin-boards, the pens, and the actual standing up part of standups wasn't going to happen.  Neither was Atlassian's Jira.  Jira is fantastic for big companies where burn-down charts help to predict where multiple teams all working to a common goal will land based on release goals.  But Jira is a huge headache getting setup, working with it, and dealing with its schizophrenic personality.  I went with three tools:

  • Trello
  • Atlassian Confluence
  • Slack

Trello is more a Kanban board than a Scrum board but it can be pressed into service if all its little knobs are twisted.  And while, without plugins, it lacks burn down and velocity charts, what it does have in spades is user-friendliness.  It's flexible for setting up different boards around different classes of workflows for different projects: sprint boards, project backlog boards, bug tracking boards, and even individual boards to break down particularly complex stories.  It also works for shopping lists, travel planning, marketing planning, deciding when various blog posts are published... and it has mobile clients.  The comment feature captures details about a story.  One can use Trello to plan anything with a little bit of cleverness.  

As a hint: trello allows story cards to have checklists which are amazing for tasking.

Confluence is an architecture trick.  The most powerful weapon in the architect's arsenal is not static code analysis or a UML/CAD tool but a good wiki.  While I'm not a fan of Jira, I'm a big fan of Confluence.  While it doesn't allow Markdown in its hosted version, it is feature rich and dead simple to use.  Communication is key and a hosted wiki is a lovely place to host online communication with diagrams, documents, instructions, and information.  Also, the dirty trick is architecture archeology -- never delete pages from Confluence.  Just move them into an archeology folder.  Keep where you've been.  DRY.

Slack pre-dated me but it's the best Agile tool in the world.  It gets everyone out of email and into a continuous chat with a history and a back-scroll.  Email is a blight on mankind.  It slows down teams and gets everyone in the habit of merely answering email all day when an issue can be solved with a sentence in a chat.  Integrated chat keeps teams working and communicating together.  Slack is one of the best investments a company can make.  It's too bad one can't get Slack without a corporate account.

What I threw out:

Being highly sensitive to the company's anarchistic leanings, I took out anything that felt over-whelming project-managery.  Agile has a large number of ceremonies and decorations -- we call them meetings but they're ceremonies.  The trick is to figure out which to drop of and which to keep.  When introducing Agile to a brand new team who is a little raw from too much project management, I cut it to the bone.  What went out the window?

  • Walls of 3x5 cards
  • Time estimation
  • Burn down charts
  • Making well worded customer-focused stories - not stories themselves, just pedantic wording
  • Hyper detailed tasking

Time estimation and burn down charts are an important part of Agile, true.  Velocity allows the team to tune how much work it can take in a Sprint.  They are also a part of stress and heart burn.  And, worse, in a lean startup doing new things, stories are near impossible to estimate even after breaking them down into tasks because the tasks devolve into "and underwear gnomes happen here."  I have to trust these are well seasoned engineers who know what they are doing and are driven to hit deadlines.  And fail fast; see above.  Keep sprints as tight as possible, fail fast, and don't worry about the numbers of hours to do X.

Also, tasking is a little light.  That's a point to iterate on.  It will get better as time goes on.

What I kept:

But I kept most of it!  

  • Daily Standup
  • Sprint Planning
  • Release Planning
  • Backlog Grooming
  • Sprint Retrospectives

... and hoping, soon, Sprint Reviews.

Getting up and going was a little rough at start.  There was a huge product backlog to work through and priorities to set and stories to put into priority order.  Some stories turned into swimlanes.  Some were dropped, renamed, or broken into pieces.  It happens.  

I was also faced with the challenge with a globally distributed team on radically different time schedules so daily standup is held in a Slack channel used for Agile administration.  We maintain a ruthlessly managed product backlog and a release backlog so sprint planning is a focused affair designed to be two hours or less.  The team is involved with the backlog grooming.  All boards are available to everyone all the time on trello.  

Sprint Retrospectives are incredibly important for crafting an Agile process around the team and the company culture.  That way the Scrum Master can listen to what everyone has to say about the previous scrum and try to make some changes to improve process.  Just like code, Agile is an ever improving process where we always try to get better.  Again, we use a slack channel for the process to collect feedback and then the results are preserved on a Confluence page so we can have history.  


Selling Agile to the team was more a process of doing instead of telling.  I can teach classes in Agile and Agile Methologies all day long but it does not mean anyone listens... it's a process, not a proscription.  We just started doing.  The initial meetings to get the backlog in order and perform a round of release planning were painful but it's rapidly becoming less-so: ritual becomes habituated.  Engineers realize that they own the process as a group so everyone is compelled to contribute.  And there is nary a Gantt or PERT chart in sight.  Nor will there ever be one.  Nor will there be big bang architecture from an ivory tower architect.  SDLC is dead.  Long live short iterate-test-release software cycles.  Us architects need to learn how to design within the context of a sprint.

What we have gotten out of the process so far has been tangible: clarity, we know exactly where we are against our deadlines, clear communication to everyone in the company what is being delivered, and a plan.  Lower blood pressure.  Focus.

Overall, since the team is small and little overhead bureaucracy, Scrum Master duties take up only about 25% of my time -- the rest of my time is focused on architecting solutions and writing code, as it should be. There's probably quite a bit more I could add like how to deal with Issues in Github while trying to track stories and bugs in Trello (solved with a product), and how to deal with sprint planning when half the team is on the other side of the world.  But suffice to say, the process is remarkable as long as the touch is light and the management overhead is non-intrusive.

In other news, I also brought in Continuous Integration and Continuous Delivery, the other half of the Agile toolkit, but that's for another day.

Loosely Couple All the Things

Loose Coupling is an old electrical engineering concept applied to computer science and used mostly when talking about object oriented architectures.  To whit: objects are loosely coupled when there is a weak dependency - little to no knowledge - about components in a system.  It's not encapsulation vs non-encapsulation -- knowledge of the inside of a class vs its public methods.  It's knowledge about the class itself.

In highly threaded OO, loose coupling of systems is a holy grail.  The less interdependent a set of objects within an overall system are, the more the system can expand and plan for change.  (This is the core mantra of architecture -- plan for change.)  If one set of classes over here, say, can change the way they work and a set of classes over there can still use the data produced without having any knowledge of the changes, the system has an overall long-term stability.  

But to hell with it.  Table flip.  Giant monolithic OO systems are dead.  Functional programming, which was dead once, is alive again, like the mathematically-based computer science zombie it is.  

Loosely couple all the things!

We're no longer loosely coupling objects while adhering to good SOLID principles.  We're loosely coupling entire systems.  Fun things to divorce from other parts of a system:

* Whole chunks of software architecture!  We are calling it today microservices but it's just loosely coupled objects taken to an extreme.  Fragments of applications get broken out into small single serving REST services with coordination through queuing (services talk to workers who do things to databases) or caching (services talk to complex caching stores to send data to each other).  Why call methods on classes when one can fire and forget a message and expect Zookeeper or AMQP to get it there? We can do it with events, baby. 

* Configuration!  Configuration management systems (chef, puppet, ansible, saltsack, your favorite shell scripts) has configuration over... here... while the systems exist over... there.  Systems are described by their configuration but no longer defined by their configuration.  And this is how God intended when He created configuration files.  Now that loose coupling is managed.

* Build artifacts!  A build systems -- ha ha! -- kicks out things.  Otherwise it's not a very good build system because it isn't building anything.  Build systems have gotten smart and cunning and can now separate apart the things they build from the systems they stand up.  Gone are the days of capistrano scripts dipping into git (unless you still do that and you should not do that because that is bad on many levels not the least of which security) to roll builds but now we can smoothly loosely couple the code from the artifacts the build system rolls and the configuration system.

* Server builds!  Vagrant now loosely couples the build artifacts and the configuration from the actual server build.  Now server builds are fancy vagrant files all held together by some high level scripting.  Wrap it with docker (oooh docker) and now we have highly loose decoupling of servers, systems, configuration, code, and virtualization in little container bags.    

* Entire servers!  We're rapidly moving away from a world of huge datacenters and static servers which hang around for years and get gunky.  Highly temporal servers themselves are loosely coupled from the code, the system and, in fact, each other.  And why have huge servers running giant vertical workloads when one can have 1000 cores on single servers running Hadoop that can be stood up and shut down at a whim?  Having the networks/network architecture/servers/server architecture now decoupled from itself and everything around it forces a sort of fragmented, bite sized architecture all the way up the stack.  

We wander on embracing a sort of mad ephemerality where everything is so loosely coupled -- the code from itself, the code from its configuration, the code from its servers to its virtualization to its network hardware -- everything becomes a sort of slurry haze of pieces that do not depend on other pieces in a strong way except for long strings of data.  And this drives a certain kind of thinking about systems where everything should be independent of everything else but still come together as a coherent whole from a large distance.  

Instead of cloud computing it should be called particle computing.  Really, it's not a cloud.  It's lots of little tiny itty bitty bits held together with clever queuing and caching to make it look like a whole.  Kind of like the core principles of matter with atoms held together into complex molecules with, you know, clever uses of Cassandra instead of the strong nuclear force.  

Mad rant off. 

Ruby vs Elixir -- An Unfair Fight (and Elixir wins)

From the Ruby Weekly mailer, I found this post: Elixir vs Ruby Showdown - Phoenix vs Rails.  As a little background:

- Elixir is a wrapper on Erlang to make Erlang more palatable to human beings.  Raw Erlang should be consumed by no one, neither man nor beast, in the course of an afternoon. Elixir tames the beast by doing what Erlang does best -- writing macros in Erlang to add features to Erlang to make Erlang less Erlang.

- Phoenix is a nice little framework built in Elixir using the Erlang HTTP and Websockets library, Cowboy. Cowboy is ridiculously powerful as simple HTTP servers go.  It's received some serious love lately.  And, of course, Cowboy is built on Erlang/OTP and inherits all the power and flexibility Erlang/OTP provides.  It is, after all, the reason we learn Erlang and then wrap it in Elixir in the first place.

- To the article's credit, the ruby configuration is the most optimal for a current ruby implementation that still runs on MRI ruby.  

Unsurprisingly, the Elixir/Phoenix implementation crushes the Ruby/Rails implementation by an order of 10x and was resource constrained by the amount of pipe available to it for getting new requests.  This was not a fair fight.

- Erlang/OTP can theoretically scale to handle a nigh-infinite number of requests.  Erlang/OTP uses a lightweight concurrent threading model inside of a running VM within the process. Whenever a new request arrives, Erlang/OTP doesn't bother with thread pools or thread management. It simply throws off another new hyper lightweight thread to manage the request, manages the request functionally with no memory sharing between threads, and throws it away at the end of the request.  It will eventually run into hardware limitations - network bandwidth first, memory second, and then processor - but joining a second Erlang/OTP machine to the first to build a cluster scales linearly.  

- The Ruby/Rails process is confined to four worker processes, each which are single threaded and limited by Ruby's global interpreter lock.  Although the memory management and garbage collection algorithms are enormously improved in Ruby 2.1.x, it can still only manage those four requests at a time.  And Rails itself is a horrendous memory hog of a framework carrying with it an entire universe of baggage simply to get it started and listening on a socket. Puma is fast; requests are likely sitting in the connection queue waiting for ruby to finish putzing around and serve them.

This isn't a fair comparison: it is like comparing a Ford Pinto to a Ferrari and then pointing out that the Pinto tried to keep up but its engine lit on fire.  In no world would the limited, memory heavy Ruby worker processes keep up with Erlang/OTP's infinite lightweight threading model.  Erlang/OTP was designed by Ericsson from the ground up to be hyper fast to maintain highly distributed telecommunication systems.  Ruby was designed to be fluid and friendly by compromising speed and Rails was designed to provide a single server LAMP stack using a standard MVC2 architecture.  

I applaud the community's push to move off Ruby/Rails and onto real, robust, functional systems.  Phoenix looks small but promising; in a year it will be large and super promising.  I will likely build something small in it to take it out for a spin.  Erlang is a bear but Elixir makes it less so.  And Erlang is so overwhelmingly powerful that the simple prospect of turning, say, a 20 node web cluster into a 2 node web cluster with double the uptime is worth the look.

(I still think that Clojure/Ring-Compojure/JVM is the actual upgrade path for Ruby/Rails... but you know, me and Erlang...)

Some Thoughts About Clojure

The new Web 2.0 languages are trying to solve what is an old problem: making the most out of processor and RAM scarcity.  Except, this time, instead of having actual scarcity, we have manufactured scarcity in the form of cloud computing: lashing hundreds of tiny slices of computers together to make one whole.  Back in the day, when scarcity was real, and building networked and highly threaded applications was the way to go, and before servers got fat and bloated and lazy, functional programming was a theoretical way to make the most of resources.

Functional programming is designed for highly threaded apps in a scarce resource world.

Procedural programming is designed for event-driven applications in a universe of processor abundance.  

That leads us to the Erlang/Scala/Clojure trifecta and to Stuart Halloway's book, Programming Clojure 2nd Edition.  Which, in a change from anything resembling normality, I sat and read instead of randomly skimming the two chapters introducing the language and then ripping right into "how fast can I stand up a web application with this?"

And now with the bullet points, cuz I love bullet points.

  • Clojure, or at least Programming Clojure, is a much kinder, gentler introduction to FP than Erlang or Scala.  Erlang is a bit like being repeatedly punched in the face by someone holding a mathematical treatise on tail call optimization and screaming and how do you like them nuggets huh huh huh?*  Scala is just that guy in the corner with the neckbeard who has no patience for you not getting it.  Clojure, because it will blow its stack if it goes all in with TCO, has a much nicer model for easing into the FP paradigms.  The lengthy chapter on Functional Programming in the book with specific examples and pitfalls helps immensely.  
  • Ruby developers should feel at home immediately with the ISeq object and the sequence macros.  Much of it looks and acts exactly like ruby's FP and metaprogramming paradigms.  If a developer likes to iterate over sets of data from database calls using .map() then they will be comfortable here.  Clojure is a clear on-ramp from ruby.
  • The ability to easily include Java libraries and just use the calls is ridiculously powerful.  You can just pull in and use all of the file manipulation calls directly from Clojure without any sort of weird jury rigging.  The integration is smooth.
  • Anonymous functions are not confusing like they are in Scala. 
  • After a few hours of reading it, Clojure's version of LISP reads more like English than anything else.  But that's LISP bringing its crazy hot mojo to the table more than anything Clojure does.
  • Like any other language running on the JVM like Scala and Processing, Clojure actually can do anything Java can do -- except in LISP.   
  • Database connectivity is just through JDBC and works pretty much like calling a database through JDBC.  As long as you have a JDBC adapter, you're good to rock and roll.

I cannot comment directly on the threading model yet.  

If you're a ruby programmer and looking to get a 10x speedup on a tiny box out of the same sort of code, I feel like Clojure is the clear next step in evolution.  The line from ruby... Clojure simply isn't that long.  Two weeks of working with it, maybe, and getting used to the templating libraries and ring/compojure and should be able to rock and roll.  With a prolific ruby programmer who uses ruby's list comprehension library to its fullest, the change should be smooth.

I'm not sure Clojure is the next step for Python and Java programmers.  That still feels more like Scala still to me.  Python does have list comprehension but it's not used the same way ruby's is so it's a higher cliff to climb.  But it's to be seen. 

Anyway, the book is good.  The language feels sane.  I also found Quil, a Clojure wrapper on the Java Processing libraries for drawing and animation, so that's a new toy.

* I desperately love Erlang.  Don't get me wrong.  But it's not for beginners.  Or intermediate.  Or advanced.  Or really anyone.

Clojure vs Scala?

Ruby on Rails is deflating like an overfilled balloon.

There is nothing wrong with the ruby language per se.  It does a number of super neat things with its metaprogramming model and duck typing and mixins.  The issue is with Rails which makes tradeoffs of performance over ease of programmability.  Rails made itself a quick prototyping on-ramp for novice to intermediate-level developers but fancy metaprogramming has a cost of needing to manipulate and create large object models in memory on the fly and incurring a huge processing cost.  

Ruby itself is plenty fast, especially with the new garbage collection model.  To get that speed, though, the entire Rails framework has to be dumped into the dustbin and developers are forced to go back to standard models of development with threading models and direct optimized SQL queries.  And it's not the rails way.

So that's a thing.

Modern architectural patterns require that servers reach the C10K model because hardware is not an infinite resource.  As more customers get online and demand more out of servers, the programmability vs speed tradeoff is no longer an effective development model.  Developers need systems that can reach the C10K model with a rails-like amount of fuss and development speed.  Node.JS is a nice choice* but... it doesn't run on the JVM and the JVM gets developers there all the way from the outset.

Clojure, with ring and compojure, it's REPL environment and leiningen bootstrap environment for plug-ins, is awful close.  It's fast.  It's multi-threaded.  It has LISP MACRO powers. It's also not like ruby at all unless one is really into lambda() functions.  Then it is.  

Scala, which brings the entire Java programming language with it, has Akka and Play Framework.  It's also battle tested.  Fast as hell.  Feels like it is made to do heavy lifting.  But much harder to get started and moving with coming from a Rails background -- it is much more like Java and Erlang had a baby than anything coming out of regular procedural programming.  

Either way, Rails shops are going to need to consider going to one or the other.  The hardware costs and the 99.999 uptimes and the customer needs combined with mobile will push shops off Rails and onto either Clojure or Scala just for the pure speed coming out of a jetty server.  Combined with access to all the java libraries in existence, it's a ready made ecosystem.  The question really is if the community will just fragment violently between the two or if it will pick one over the other.  

* I like Node.JS.  I really like Node.JS.  I am finding I don't want to use it for heavy lifting. Why is this? I want to love node.js the same way I love python and take advantage of its web socket goodness but there's something weird.  Maybe it's the community explosion that is making me go "not in the face!."

Setting Up Travel APIs Pt 1

Hey, back to the point of this project, building a website that supports conventions and allows for a million users to hit it all at once because it's the biggest convention in the known universe! Enough with the DevOps logging and monitoring nonsense... back to the meat and potatoes of building a product.  Features!  We demand features!

(Despite that the big logging models are interesting...)

If we want people to buy a convention pass we also want to route them to a convenient and nearby hotel.  Setting up actual hotel programs with local hotels is largely out of the scope of writing a blog but looking into the developer APIs of various providers is not.  This goes into the world of having to partner with a SaaS.

I will simply walk my notes in putting together a solution.  Quasi real time... in text.  Exciting!

Pick an API Provider

Several of the API providers to hotel information have elaborate rules, partnering agreements, and requirements.  There are three of varying quality that allows developers to jump right in:

1. The Google Places API -- - which does all of the fun integration with google maps.  It brings up a nice list based on proximity and an integrated Google Map but doesn't allow for room and reservation information. Great for integrating with other things.

2. The Yahoo Travel API -- -- which looks a little long in the tooth and on its way out.  

3. The Expedia API -- -- which slices, dices, and allows listings to hotels although the site has to go through a complete review before Expedia will allow a production API key to their system.  They do post a fairly comprehensive list of criteria to meet.

This enormous list of travel APIs is available for perusal.

Expedia seems to want developers to integrate with their systems and pull their information to get the data flows through their systems.  Because of the sheer volume of their API documentation, due to it being XML/REST based, the ease of integration, and the availability of API keys -- developer access requires a sign up and then development can begin -- that one is the winner for pulling hotel information.  Project Butterfly will (in a vaporware sort of way) integrate with Expedia.

From a business development process there's likely a contract with an SLA attached that must be negotiated before a site integrating with them goes live along with their approval;  if Expedia goes down the system needs a plan or otherwise convention goers.  Also, there are enormous security requirements on both sides if booking actually works through our site, through Expedia (where they take a cut) and back again.   While Expedia can deal with the room booking via APIs, all communication must be secured via HTTPS.  Our site cannot keep credit cards information.  

Project Butterfly for now will only pull hotel lists and availability and post a link to send a user to Expedia to book their room (for now).  The site isn't ready to sell a pass so for now it will be only informational.

Looking through the Expedia API, one can request a hotel list based on a starting geolocation, page via cache on their side, proximity, rating, price list, room types available, room availability, etc.  

Part 2... will walk through implementation details and pitfalls with dealing with data from a 3rd party provider and the impact on the site.

Logging Architectures and Elasticsearch ELK

The original solution to building a centralized logging system for N+1 servers was to:

  1. Have code write directly to syslog based on the syslog library of choice for that particular language and/or have a syslog collector which tails logs and sends them into syslog;
  2. Forward the logs over UDP to a centralized collection point;
  3. Perform grep with a vengeance.

This required a centralized logging server with good log rotation scripts, some LDAP goodness for authentication, and some decent knowledge of a decent regex.  For somewhere around a cap of 40 data sources sending data to the central server, this is not a terrible solution -- especially if the logs are structured and fit within the syslog limitations for a message.  (1)  Now this is data sources instead of servers: if the servers are sending both application logs, web logs and syslogs, divide the number of sources by three to be effective.

The topology of a system looks much like so.

Rsyslogd logging topology

Rsyslogd logging topology

This works well if we get beyond the occasional message lost through UDP weirdness.  But if we have more than the servers than this can handle, and we don't have enough money for the all mighty Splunk, do we go straight to Hadoop?

The temptation is to get a jump start on the sexy world of Hadoop to get it on a resume tout de suite.  Hadoop is, ultimately, a distributed file system much like AFS before it using a combination of clever Java processes and Zookeeper to turn commodity hardware into a vast sea of storage.  Map/Reduce is a software framework embodied in Hive or Pig to sift all that data and find answers within its vast sea of non structured data.  But Hadoop is varsity level that gets better the more nodes added to the cluster.  Building it for a website that turns over 1M log lines or less a day is not worth the effort from the effort overhead of figuring it out and building the batch jobs that span the data store and surface useful data.

We need something a little more junior varsity level.  Something that does the job and will run in the public cloud or a private cloud and doesn't require a bunch of data scientists standing around and going hmm.  Something that will cough up good solid analytics.

This is a thing that maybe kinda sorta exists.  It's called Elasticsearch ELK.  Elasticsearch - Logstash - Kibana.

We need the following components:

  • Forwarders.
  • Search Indexers.
  • Search Engines.
  • Analytics.

Forwarders are the most difficult to pick because there are so many to choose from.  The best one is Apache Flume, a high performance java-based forwarder which has an elegant input-sink-output architecture and bridges the gap between CDH (Cloudera Distribution including Hadoop) (2) and Elasticsearch.  But there's also Fleuntd and Lumberjack which are far more logstash-friendly and designed to be lighter on a system than the memory hog Flume.

This is a your mileage may vary -- if eventually the cluster blows out into a Hadoop cluster, you want to use Flume because that architectural choice will serve you oh so well in the future.  If you love your Elasticsearch cluster -- and you will -- then it's a preference based on the log shape.  Fluentd and Lumberjack will be much happier with more structured logs as inputs.  Flume shines with unstructured logs, like log4j logs - although Flume really wants to write in Avro.

They have the same basic functionality: get logs away from the application and into the logging system quickly and efficiently without putting strain on the host system.

Indexers are a little more touchy and here.  Logstash will do the work of the extract/transform/load for the data coming in from the forwarders and load it into Elasticsearch.  The logstash configuration file requires well defined the inputs and outputs but the system has an enormous list of plugins on both sides of the equation.  It will use grok to parse logs into Elasticsearch format.  

For Hadoop, Flume will likely load into Avro or Thrift.  

For search, Apache Lucene is an amazing engine.  It's an incredibly fast full-text lexical search engine and all it asks for is RAM and iOPs.  It is what is lies at the heart of Elasticsearch. (3)

Elasticsearch does all sorts of super interesting things -- it clusters, it grows, it's fault tolerant, it provides a concise query language, it has APIs for inter-operability and it can be used as a searching engine for Hadoop when moving to a full Hadoop cluster -- or along side it for specific log analytics.  It's really a neat piece of software that slices and dices.  It will burn through 10s of thousands of logs a second given it has enough RAM.   

The query front end is a piece of software called Kibana.  Not too much to say here other than it's a graphical query engine that plugs directly into most of these technologies -- elasticsearch, logstash, fluentd, flume -- and delivers intelligence and analytics about the system.  If you have ever run an operation with a few hundred web servers and you can now get a nice live-updating trending graph of 502s across the entire system to find when the system is going down under load, you will appreciate a system like Kibana.  If you are in business development and you'd like to understand how users actually use your system, or if you are in security and want to see users brute force your authentication systems, you will also appreciate Kibana.  Graphs for everyone!

Here's an upgraded topology using all these technologies in concert to pull together a full working system:

A very basic, not scaled ELK topology.  Scales at every point.

A very basic, not scaled ELK topology.  Scales at every point.

Preferably, all of these components should run on separate VMs in production.  In a testing environment, enh.  Elasticsearch does want to run alone, though, because as a Java-based search head, it works best when it can consume the maximal cores and RAM on a system.  If it does not have all the RAM it needs, it will work but searching will under perform.  

It feels like a ton of work to pull together a system one could pay for but being able to spot trends, problems, and instability in a system is gold.  This can all be built out of commoditized hardware and, with some gotchas, in public cloud.  That logstash can also forward to ganglia and graphite is a golden bonus.  

Cool stuff.  We didn't get flying cars in the future but we certainly got hyper-fast log indexing, searching and trending.

One thing that I did not cover at all is scaling all these components.  All of these components horizontally scale -- and logstash has some gotchas.  If/when I go deeper I'll talk about that in more depth and breadth.

  1. Graylog2 has a format called GELF as a syslog replacement which gets around several limitations of the syslog format.  It's another whole way of building this system with Apache Kafka.
  2. For reasons unknown, Cloudera, who is otherwise amazing, does not tell you what CDH stands for. 
  3. This is an offside and a topic of another post: be very careful putting Elasticsearch nodes on cloud systems that charge/iOP.  ES can easily do 1M iOps a month on a moderate sized system.  Also, AWS specific: there's no multicast in AWS so nodes will not auto-discover.  Documented work arounds are available.

Logging Architectures and Splunk

If I could spend money on any underlying infrastructure component for a large distributed system it would be on Splunk

Splunk is a log aggregator.  It accepts data from any known data stream, nominally through sets of forwarders, ETLs the data, indexes the data, and provides the data for search.  So data can come in from a number of different sources -- syslog, application logs, nginx logs, firewall logs -- can be aggregated together across a common logging framework using Splunk's map reduce-like language.  Reports can span over time, be shown in a variety of formats, and can live update into monitors.  And it can alert.

For a system that has, say, several hundred virtualized nodes all playing different roles in a complex system, Splunk is a godsend.  It coughs up all the data as long as there are logs -- going back to my argument that a good distributed system must log.  This is what the system does with all those endless logs.  It sends them to Splunk.  

With some careful logging and mining, one can trace a user from the furthest edge to interactions with various deep in systems and then back out again at the end of their session, skipping from system to system to system. Splunk is every bit as sweet a system as the website plays up.


Let's talk about the howevers.

Splunk costs all the money.  Whatever money you have, it costs it.  It costs more than that.  Because the licensing is by the gigabyte, it costs more the more useful it is.  Sure when a few web logs go in, it's moderately useful.  Once it starts surfacing analytics, the voracious appetite for analytics never lets up.  It's like a heroin addiction.  The first high is great but then it costs more and more to maintain that high and get to new ones.  

Splunk does need some serious love from professional services to become useful.  It is a large complex system with multiple moving parts that cannot be understood by just reading the manual.   Without love indexers fill up, indexes are not updated, reports not maintained, and the system falls into disrepair.  It takes on a post-Apocalyptic air of unlove. 

And... Splunk needs to run on dedicated highly performant hardware with equally highly performant disk.  This feeds into the top point -- Splunk is expensive.  But if the model is to spend money on analytics and system intelligence instead of time, this is the far end of the money-time trade-off.  

If there's time to build and little money, this isn't the right buy because it is extremely pricy.  But if there's either little time, lots of money, or the business is risk-averse, it's a fantastic product.

I sound like a shill for Splunk -- and it is lovely, they can send me all the free t-shirts -- but this is all warm-up for "how do I build splunk without any money and... is it any good?"  


What is Heartbleed?

Putting on my cryptography nerd hat for a moment.  

Everyone is talking about heartbleed.  What is it and why should you care?

In December 2011, the OpenSSL project accidentally introduced a flaw into the optional heartbeat function for a SSL to allow an SSL tunnel to stay pinned up and active over an extended lifetime.  The flaw hides in a compilation flag which is built into all openssl binaries pulled down from repositories or stock with post-May 2012 builds Unix-based operating systems.  If a system uses one of the affected openssl binaries for their secure communications, all SSL communication on that system has that flaw.

The flaw allows an attacker to read 64 random plaintext bytes out of memory.   Anything that can be resident in a process's memory -- private keys, root passwords, secure configuration files, anything secured in the SSL channel -- an attacker will get by simply starting an SSL session over and over on the targeted server.  Since authentication systems on the Internet are both Unix-based and openssl-based, authentication systems with the flaw are leaking customer credentials to intruders.  This will allow attackers to own entire email or banking services.  Worse, if the vulnerable web services are running as root (a common, but bad, practice), reading the 64 random plaintext bytes will, sooner or later, leak the information used to start the process and possibly leak root passwords to servers to the Internet allowing attackers to compromise entire systems.

Nevertheless, the panic on this one is justified.  It is a bad bug.

It is trivial to determine if your systems are affected.  Simply log into the systems terminating SSL and type:

> openssl version

Here is the vulnerability list:

  • OpenSSL 1.0.1 through 1.0.1f (inclusive) are vulnerable
  • OpenSSL 1.0.1g is NOT vulnerable
  • OpenSSL 1.0.0 branch is NOT vulnerable
  • OpenSSL 0.9.8 branch is NOT vulnerable

If you are like many lazy people on the Internet and you haven't patched your server in a glacial ice age, you are not susceptible to this bug but likely dozens of other security flaws so you should patch.  If you are a Sound Patcher, you are likely vulnerable.  Check your servers. 

Also it's possible your load balancers, routers, firewalls, and other network gear are vulnerable. Call your vendor immediately and find out the state of heartbleed and your gear's OS revision.  Seriously, you might be vulnerable beyond your applications -- network gear uses openssl, too.    

Fix your servers.  There are absolutely exploits that take advantage of Heartbleed right now in the wild.