How to run 4.5 Million tests per day, and why

If you had to guess what software company has the expertise (and plain guts) to scale up the CI (continuous integration) concept to running almost 5 Millions tests per day, what would you say? Google, of course. This short talk by Mark Striebeck (a Google employee) was very interesting and somewhat illustrative of what Google is capable of when it sets its collective mind to something.

People at Google were finding it harder to apply CI because of the large size of their test base. Tests were taking a long time to run so they were run less often. To solve this problem, they decided to leverage the Google infrastructure and implemented their test harness as a scalable web service. The test execution is distributed on more than 1500 CI servers!

The interface to the test system is a nice dynamic web page where you can filter and display the results in many different ways, giving you the exact view you want on your test results. And as you would expect, the search works pretty well, too.

Big Balls

"Big Balls of Mud: Is This the Best that Agile Can Do?" was one of my favorite sessions, given by Brian Foote and Joseph Yoder. It was presented as a "debate", where both of the presenters answered a series of questions given by a moderator. This session was a reflection on what I consider a fundamental problem in our profession: why is the "Big Ball of Mud" still the de-facto standard software architecture?



Most of the material presented comes from an article (41 pages) the presenters wrote called "Big Ball of Mud" (html | pdf). Big Balls of Mud are

haphazardly structured, sprawling, sloppy, duct-tape and bailing wire, spaghetti code jungle. We’ve all seen them. These systems show unmistakable signs of unregulated growth, and repeated, expedient repair. Information is shared promiscuously among distant elements of the system, often to the point where nearly all the important information becomes global or duplicated. The overall structure of the system may never have been well defined. If it was, it may have eroded beyond recognition. Programmers with a shred of architectural sensibility shun these quagmires. Only those who are unconcerned about architecture, and, perhaps, are comfortable with the inertia of the day-to-day chore of patching the holes in these failing dikes, are content to work on such systems.

The architects preach better practices, with diagrams, principles, books, but in the end we still end up with mud. What makes the Big Ball of Mud software architecture so successful? Foote gives the example of Internet Explorer and Netscape, both of which he considers to be monstrous large balls of putrid mud. These products were successful, and made money. Some other examples of cleaner things that failed: Mac (smaller market share than Windows), lisp, Smalltalk, Betamax (versus VHS), FrameMaker (versus MS-Word).

Foote and Yoder came up with the idea of "The Selfish Class" (a reference to The Selfish Gene, by Richard Dawkins) to explain the evolution of software. Would this mean that Mud is more fit for survival than other types of software architectures? Is Mud normal? Is Mud the best we can do in real production systems? Why do people keep doing it? Why do we make money with Mud? In the global software environment, where different architectures fight and compete for survival, it would seem that the Big Ball of Mud is the dominant species.

Does Mud == Legacy? Not necessarily, Mud is produced daily in prodigious amounts. On the other hand, some old code is good (e.g., Smalltalk).

To illustrate his view that mud is the mostly used and deployed software architecture, Foote gave the dark matter analogy.



What are the reasons for Mud happening?

• Throwaway code
• Piecemeal growth
• Keep it working
• Copy / paste metastasis (faulty code is reproduced in many places and spreads)
  

When asked about the best and worst code he has ever seen, Yoder answered the following:

• Best code: the Smalltalk core library
• Worst code: a single C++ class with over 500 000 lines of code. This class is still in production at a large and well-known Silicon Valley company!

Finally, is Agile useful in preventing Mud? Over the years since the first software programs, we went through many changes to what should be the "best" process. But still, year after year, decade after decade, we produce gargantuan mountains of nauseous mud. Is Agile the definite "best" process for avoiding mud? Yoder answers that many aspects of Agile actually seem to lead directly to mud:

• Lack of upfront design
• Late changes to the requirements
• Late changes to the architecture
• Piecemeal growth
  

In the end, maybe teams of good people produce good software, while teams of bad people produce bad software. This means that average teams will have some of both (good and bad), thus slowly inching towards mud. Is it that simple? Did we overestimate the importance of the process?

Overall, it was a very interesting session, with excellent speakers.

Null References: the Billion-Dollar Mistake

At the QCon 2009 Conference in London, the inventor of QuickSort, Tony Hoare, talked about another one of his creations: null references (without pride, it would seem):

http://www.infoq.com/presentations/Null-References-The-Billion-Dollar-Mistake-Tony-Hoare

I didn't have time to watch the video, but I'm pretty sure it is very interesting. I was surprised to find out that such a fundamental concept as null references had an inventor!

This presentation was suggested by Dean Wampler during his session on Scala.

The Scala Programming Language

On Wednesday morning I was in the mood for a more technical presentation, and "Scala: Object-Oriented and Functional Programming for the JVM", by Dean Wampler, seemed to fit the bill perfectly. The presenter was a bit surprised that his session was accepted in the Agile 2009 schedule, since it does not really relate to Agile... Anyway, it was a very interesting introduction to a new language: Scala.

Why do we need a new language?

• We need functional programming
• for concurrency
• for concise code
• for correctness
• We need a better object model
• for composability
• for scalable designs
• But we want to keep our investment in Java and C#

Scala is also a statically typed language.

Here are some references if you are interested in learning more about the language:

• The Wikipedia entry entry on Scala
• The official Scala website
• The Scala Wiki (with many code examples)
  
• Scala on Literal Programs - interesting examples
  

Planning Poker: one card only!

On Tuesday night, during a marketing event, I had a short discussion with someone from Cyrus Innovation about planning poker. I had a specific question: "should we use multiple cards when making our estimate?" The idea behind using multiple cards is to form numbers that are not available in the list, for example, use 1 and 3 to play a 4. His opinion was that no, we should not do this. The cards have those values (0, ½, 1, 2, 3, 5, 8, 13, 20, 40, 100) for a reason. According to Wikipedia:

The cards are numbered as they are to account for the fact that the longer an estimate is, the more uncertainty it contains. Thus, if a developer wants to play a 6 he is forced to reconsider and either work through that some of the perceived uncertainty does not exist and play a 5, or accept a conservative estimate accounting for the uncertainty and play an 8.

The Ideal Agile Toolset

My last presentation for Tuesday was a talk by Brad Appleton (who was using crutches because he hurt his leg while breakdancing, true story!) about the characteristics of the ideal Agile toolset. Appleton feels that the current tools we have that support the Agile process (e.g., Eclipse, Confluence, TWiki, Trac, Jira, zAgile) all fail one way of another to deliver proper Knowledge Management.

Most current Agile toolsets treat source code as the primary deliverable of a project. This means that the source code gets special treatment: branches, labels, history, dependencies, etc. However, Appleton asserts that source is not the product: knowledge is the product delivered on a project. Source code is only one way to represent knowledge, it is a knowledge format. What follows is that the ideal toolset should manage all knowledge the same way, allowing tight revision control on diagrams, wiki entries, tests, etc.

Appleton then raises some interesting questions:

• How can we define "done" for non-code artifacts (e.g., a UML diagram)?
• Is it possible to apply TDD (test driven development) to non-code artifacts?
  

This talk by Brad Appleton got me thinking about a practice we have in our department. We usually manage our designs by checking in a complete design document (a Word .doc file, or a EAP file) in our VCS. Maybe we should try to have a finer level of control on those designs by splitting the documents into individual diagrams, or sections, etc. This would allow for a better management of dependencies (e.g., between code and individual diagrams), branches, etc.

On a final note, the presenter mentioned a book that could be an interesting addition to our library: The Laws of Software Process, by Phillip G. Armour. (I don't think we have it, but I could be wrong...)

Agile Project Management with Google Docs

This session was an experience report by a company who decided to use Google Docs as a lightweight approach to Agile project management for a distributed Scrum. For their daily stand up meetings, they would use a shared Google Spreadsheet while having a Skype conference call. They also used this approach for the planning poker. In both instances, a single person would edit the spreadsheet, but being a shared Google Spreadsheet, everyone would see the updates in real time.

One thing they do that I found interesting is that they have a couple of LCDs near their Scrum meeting area that constantly display information about the project, such as snag count, the status of the build, etc. The goal is to create or encourage participation through shared and visible information. The term usually employed for this is "mieruka" which means "making visible" in Japanese, and seems to have been pioneered by Toyota.

The Greatest Misses of an XP coach

J. B. Rainsberger is a software consultant that gave a talk about his greatest misses as an XP coach. He is a very good speaker, the talk was rather pleasant and interesting. What he basically said during 90 minutes is "people are more important than practices", and that the key to succeeding with Agile is good communication.

He mentioned two techniques that can improve your communication skills:

• The Satir Interaction Model. This model will help you "debug" the conversations you have with other people.
  
• The laws of "generous interpretation" and "harsh interpretation". When someone tells you something, try to come up with three different interpretations of what they said. Assume that most generous is the real one. On the other hand, when you want to say something to someone, make up three different interpretations and assume the other person will choose the most harsh. This should improve your reactions to what other people tell you, and also deepen your understanding of the reaction of others to what you say.
  

He also suggested a book, "The Psychology of Computer Programming", saying that it was the most influential book he ever read.

I would like to end this post with something that Rainsberger said that I found very interesting:

Doing the right thing for the right reason is not enough. [You also need to have the trust and commitment of your teammates and stakeholders.]

Covert Agile

I selected the "Covert Agile: Development at the Speed of…Government?" session to complete my Tuesday morning schedule. It turned out to be a pretty interesting experience report by a consultant who did a contract for the United States Department of Defense. The project was to re-write some legacy satellite scheduling software so it would work with modern hardware. It wasn't just for the thrill of using the latest and greatest hardware though. They were actually forced to go on ebay.com to find replacement parts because nobody produced the old hardware they were dependent on. Having DoD software running on machines bought on ebay is not very reassuring for American citizens.

They knew that at least one previous attempt at doing this by another company had failed. It didn't fail because that company didn't deliver, on the contrary, they delivered a complete working solution after many months of work. However, the operators simply didn't like the interface. They refused to use the new software so the project was a failure.

So it was obvious that if they were to succeed, they needed to include the operators very early in the process, and have relatively short iterations, with rather fluid requirements. They wanted to be Agile. However, in order to get the contract they needed to hide their Agility! They were required by DoD to provide "Waterfall" type artifacts at various milestones in the project. They actually had to hire people to produce all these "Waterfall" artifacts, and deliver them to DoD. These people were not contributing anything to the software, they were just producing heavy documents and sending them upward to be able to keep the contract.

In the end they succeeded, but the presenter said he hoped to never have to do such a contract ever again. I think a part of his soul died during this project and was buried in a large requirements binder.

As a last note on this session: the presenter mentioned that one thing they did that helped them was to leave a certain proportion of sprints (say 1 in 4, or 1 in 5) entirely dedicated to bug fixes, refactorings, etc. We sometimes do something similar, but in our team it seems to be reactive, and somewhat disruptive for the schedule. In their case, they were proactively planning for it from the beginning of the project.

Kill the Gatekeeper!

On Tuesday morning I attended an experience report given by Francis Lacoste, who's working on LaunchPad (not to be confused with our LaunchPad). He basically described the changes they made to their integration system to render it more Agile, more "continuous".

They initially had a single "Main" branch that they tried to never break, a bit like what we have in our department. They had a GateKeeper that would validate all changes submitted by developers, and only if everything passes (build, unit tests, etc.) would the change be applied to the main branch.

They realized this was preventing them to fully implement continuous integration, so they now work with a "stable main" and a "devel main". The developers can now push changes at any time they want to the "devel main". The gatekeeper resides between the "devel main" and the "stable main".

(Note: the gatekeeper in their case is not a human being, it is an automated tool)

Keynote

We were all staring at the freakish picture of the speaker, Alistair Cockburn, when a guy emerged from the door and crossed the entire room, playing bagpipes along the way. Cockburn then appeared on the stage and started reciting the "Agile" version of some part of Shakespeare's Julius Caesar... That's how the keynote speech, titled "I come to bury Agile, not to praise it", started.

It was an interesting talk overall. Here are some points Cockburn made during the presentation (not an exhaustive summary):

• When we pick items from the backlog, we should not just consider business value; technical risk is also an important consideration. We may want to start working on the riskier things earlier to get a lot of knowledge as fast as possible.
• The "gold card": when a developer gets a "gold card" (I'm not sure through what mechanism), that developer gets to work on anything he wants for a couple of days. Cockburn seems to be in favor of having some sort of "thinking time"!
• We should stop talking about "Agile". We should bury the term itself, since all those practices are now part of the current way to do software engineering. It is now time to consider experimenting with newer techniques.