Monday, August 31, 2009

Big Balls

"Big Balls of Mud: Is This the Best that Agile Can Do?" was one of my favorite sessions, given by Brian Foote and Joseph Yoder. It was presented as a "debate", where both of the presenters answered a series of questions given by a moderator. This session was a reflection on what I consider a fundamental problem in our profession: why is the "Big Ball of Mud" still the de-facto standard software architecture?



Most of the material presented comes from an article (41 pages) the presenters wrote called "Big Ball of Mud" (html | pdf). Big Balls of Mud are
haphazardly structured, sprawling, sloppy, duct-tape and bailing wire, spaghetti code jungle. We’ve all seen them. These systems show unmistakable signs of unregulated growth, and repeated, expedient repair. Information is shared promiscuously among distant elements of the system, often to the point where nearly all the important information becomes global or duplicated. The overall structure of the system may never have been well defined. If it was, it may have eroded beyond recognition. Programmers with a shred of architectural sensibility shun these quagmires. Only those who are unconcerned about architecture, and, perhaps, are comfortable with the inertia of the day-to-day chore of patching the holes in these failing dikes, are content to work on such systems.
The architects preach better practices, with diagrams, principles, books, but in the end we still end up with mud. What makes the Big Ball of Mud software architecture so successful? Foote gives the example of Internet Explorer and Netscape, both of which he considers to be monstrous large balls of putrid mud. These products were successful, and made money. Some other examples of cleaner things that failed: Mac (smaller market share than Windows), lisp, Smalltalk, Betamax (versus VHS), FrameMaker (versus MS-Word).

Foote and Yoder came up with the idea of "The Selfish Class" (a reference to The Selfish Gene, by Richard Dawkins) to explain the evolution of software. Would this mean that Mud is more fit for survival than other types of software architectures? Is Mud normal? Is Mud the best we can do in real production systems? Why do people keep doing it? Why do we make money with Mud? In the global software environment, where different architectures fight and compete for survival, it would seem that the Big Ball of Mud is the dominant species.

Does Mud == Legacy? Not necessarily, Mud is produced daily in prodigious amounts. On the other hand, some old code is good (e.g., Smalltalk).

To illustrate his view that mud is the mostly used and deployed software architecture, Foote gave the dark matter analogy.



What are the reasons for Mud happening?
  • Throwaway code
  • Piecemeal growth
  • Keep it working
  • Copy / paste metastasis (faulty code is reproduced in many places and spreads)
When asked about the best and worst code he has ever seen, Yoder answered the following:
  • Best code: the Smalltalk core library
  • Worst code: a single C++ class with over 500 000 lines of code. This class is still in production at a large and well-known Silicon Valley company!
Finally, is Agile useful in preventing Mud? Over the years since the first software programs, we went through many changes to what should be the "best" process. But still, year after year, decade after decade, we produce gargantuan mountains of nauseous mud. Is Agile the definite "best" process for avoiding mud? Yoder answers that many aspects of Agile actually seem to lead directly to mud:
  • Lack of upfront design
  • Late changes to the requirements
  • Late changes to the architecture
  • Piecemeal growth
In the end, maybe teams of good people produce good software, while teams of bad people produce bad software. This means that average teams will have some of both (good and bad), thus slowly inching towards mud. Is it that simple? Did we overestimate the importance of the process?

Overall, it was a very interesting session, with excellent speakers.

Null References: the Billion-Dollar Mistake

At the QCon 2009 Conference in London, the inventor of QuickSort, Tony Hoare, talked about another one of his creations: null references (without pride, it would seem):

http://www.infoq.com/presentations/Null-References-The-Billion-Dollar-Mistake-Tony-Hoare

I didn't have time to watch the video, but I'm pretty sure it is very interesting. I was surprised to find out that such a fundamental concept as null references had an inventor!

This presentation was suggested by Dean Wampler during his session on Scala.

The Scala Programming Language

On Wednesday morning I was in the mood for a more technical presentation, and "Scala: Object-Oriented and Functional Programming for the JVM", by Dean Wampler, seemed to fit the bill perfectly. The presenter was a bit surprised that his session was accepted in the Agile 2009 schedule, since it does not really relate to Agile... Anyway, it was a very interesting introduction to a new language: Scala.

Why do we need a new language?
  • We need functional programming
    • for concurrency
    • for concise code
    • for correctness
  • We need a better object model
    • for composability
    • for scalable designs
  • But we want to keep our investment in Java and C#
Scala is also a statically typed language.

Here are some references if you are interested in learning more about the language:

Planning Poker: one card only!

On Tuesday night, during a marketing event, I had a short discussion with someone from Cyrus Innovation about planning poker. I had a specific question: "should we use multiple cards when making our estimate?" The idea behind using multiple cards is to form numbers that are not available in the list, for example, use 1 and 3 to play a 4. His opinion was that no, we should not do this. The cards have those values (0, ½, 1, 2, 3, 5, 8, 13, 20, 40, 100) for a reason. According to Wikipedia:
The cards are numbered as they are to account for the fact that the longer an estimate is, the more uncertainty it contains. Thus, if a developer wants to play a 6 he is forced to reconsider and either work through that some of the perceived uncertainty does not exist and play a 5, or accept a conservative estimate accounting for the uncertainty and play an 8.

The Ideal Agile Toolset

My last presentation for Tuesday was a talk by Brad Appleton (who was using crutches because he hurt his leg while breakdancing, true story!) about the characteristics of the ideal Agile toolset. Appleton feels that the current tools we have that support the Agile process (e.g., Eclipse, Confluence, TWiki, Trac, Jira, zAgile) all fail one way of another to deliver proper Knowledge Management.

Most current Agile toolsets treat source code as the primary deliverable of a project. This means that the source code gets special treatment: branches, labels, history, dependencies, etc. However, Appleton asserts that source is not the product: knowledge is the product delivered on a project. Source code is only one way to represent knowledge, it is a knowledge format. What follows is that the ideal toolset should manage all knowledge the same way, allowing tight revision control on diagrams, wiki entries, tests, etc.

Appleton then raises some interesting questions:
  • How can we define "done" for non-code artifacts (e.g., a UML diagram)?
  • Is it possible to apply TDD (test driven development) to non-code artifacts?
This talk by Brad Appleton got me thinking about a practice we have in our department. We usually manage our designs by checking in a complete design document (a Word .doc file, or a EAP file) in our VCS. Maybe we should try to have a finer level of control on those designs by splitting the documents into individual diagrams, or sections, etc. This would allow for a better management of dependencies (e.g., between code and individual diagrams), branches, etc.

On a final note, the presenter mentioned a book that could be an interesting addition to our library: The Laws of Software Process, by Phillip G. Armour. (I don't think we have it, but I could be wrong...)

Friday, August 28, 2009

Agile Project Management with Google Docs

This session was an experience report by a company who decided to use Google Docs as a lightweight approach to Agile project management for a distributed Scrum. For their daily stand up meetings, they would use a shared Google Spreadsheet while having a Skype conference call. They also used this approach for the planning poker. In both instances, a single person would edit the spreadsheet, but being a shared Google Spreadsheet, everyone would see the updates in real time.

One thing they do that I found interesting is that they have a couple of LCDs near their Scrum meeting area that constantly display information about the project, such as snag count, the status of the build, etc. The goal is to create or encourage participation through shared and visible information. The term usually employed for this is "mieruka" which means "making visible" in Japanese, and seems to have been pioneered by Toyota.


Diagrams for understanding and improving Agile practice

Mon mardi après midi fut probablement ma meilleure journée. Trois bonnes conférences en trois dont deux salles combles. Plusieurs personnes se sont vu fermer la porte au nez. When is full, is full !

La conférence de Bonnie Aumann et Arlo Belshee, « Diagrams for understanding and improving Agile practice », commence avec un petit exercice pratique. Avec comme information seulement un estimé du « Business Value » (Low, Some, Good & High) associer à chacun des livrables de l’entreprise, nous devons choisir par quel livrable nous commençons. Nous retournons la carte et nous obtenons la durée et le bénéfice obtenu. Nous soustrairons la durée des 120 jours initiaux que nous disposons et nous répétons l’exercice tant qu’il nous reste des journées. La deuxième phase de l’exercice ce fait avec les même cartes cependant, nous disposons d’une information supplémentaire pour faire notre choix. Nous avons un estimé en temps pour chacun des livrables, ce qui correspond au coût du livrable. Après avoir fait cette deuxième phase, nous comparons les bénéfices pour nous rendre compte que nous obtenons un meilleur bénéfice avec seulement comme information le Business Value. Hasard ? Selon les deux conférenciers ce n’est pas le fruit du hasard. Après quelques courbes, graphiques et formules mathématiques nous comprenons que la nature humaine nous porte à prendre des mauvais choix. Cet exercice nous démontre que nous interprétons mal un surplus d’information parfois plus complexe que nous le croyons.

















Le reste de cette présentation a porté sur la performance versus l’overhead et la période (continues processing) versus la production incluant d’autre donné comme les frais fixe (taxes…). Une très bonne présentation qui avait comme objectif de nous démontrer qu’une courte période est souvent signe de $.

ScrumMaster

Le problème avec le ScrumMaster c’est le nom. Le nom porte à confusion. Que veut dire Master dans ScrumMaster? La courte conférence de Paul Hodgetts « ScrumMaster Considerer Harmful » nous aide à mieux définir le rôle d’une ScrumMaster. Je vous énumère rapidement ce qu’un ScrumMaster est :
ScrumMaster as a Scrum Change Agent
ScrumMaster as Our Process Conscience
ScrumMaster as a Remover of Impediments
ScrumMaster as Coach and Team Builder
ScrumMaster as Committed to the Team
ScrumMaster as an Ambassador to the Organization

Cependant, le ScrumMaster n’est pas un Manager ni un Project Manager. Le ScrumMaster à l’esprit et le goût d’aider les autres. Petite parenthèse, je ne veux certainement pas dire que le Manager ou PM ne veut pas aider les gens. D’ailleurs, j’ai appris que la majorité des ScrumMasters était Project Manager au paravent. Je crois que nous (mon organisation) devrions maintenant nous poser les questions suivantes : Avec toutes ces responsabilités, le ScrumMaster devrait-il codé ? Devrait-il avoir le temps ?

Thursday, August 27, 2009

Butterfly Effect

Lors de la conférence “What (Else) Can Agile Learn from Complexity?”, Jurgen Appelo nous fait la démonstration que rien n’est simple. Dans un monde complexe, il est impossible de tout prédire. La phrase qu’il a mentionné vers la fin de sa présentation, invoque bien le sens général: « All models are wrong, but some are usefull ».

Il n’y a pas de solution magique qui peut s’applique à une entreprise et encore moins à toutes les entreprises. Cependant, aujourd’hui Agile est probablement le model le mieux adapter au monde du software. À nous de l’appliquer du mieux que nous pouvons.

The Greatest Misses of an XP coach

J. B. Rainsberger is a software consultant that gave a talk about his greatest misses as an XP coach. He is a very good speaker, the talk was rather pleasant and interesting. What he basically said during 90 minutes is "people are more important than practices", and that the key to succeeding with Agile is good communication.

He mentioned two techniques that can improve your communication skills:
  • The Satir Interaction Model. This model will help you "debug" the conversations you have with other people.
  • The laws of "generous interpretation" and "harsh interpretation". When someone tells you something, try to come up with three different interpretations of what they said. Assume that most generous is the real one. On the other hand, when you want to say something to someone, make up three different interpretations and assume the other person will choose the most harsh. This should improve your reactions to what other people tell you, and also deepen your understanding of the reaction of others to what you say.
He also suggested a book, "The Psychology of Computer Programming", saying that it was the most influential book he ever read.

I would like to end this post with something that Rainsberger said that I found very interesting:
Doing the right thing for the right reason is not enough. [You also need to have the trust and commitment of your teammates and stakeholders.]

Covert Agile

I selected the "Covert Agile: Development at the Speed of…Government?" session to complete my Tuesday morning schedule. It turned out to be a pretty interesting experience report by a consultant who did a contract for the United States Department of Defense. The project was to re-write some legacy satellite scheduling software so it would work with modern hardware. It wasn't just for the thrill of using the latest and greatest hardware though. They were actually forced to go on ebay.com to find replacement parts because nobody produced the old hardware they were dependent on. Having DoD software running on machines bought on ebay is not very reassuring for American citizens.

They knew that at least one previous attempt at doing this by another company had failed. It didn't fail because that company didn't deliver, on the contrary, they delivered a complete working solution after many months of work. However, the operators simply didn't like the interface. They refused to use the new software so the project was a failure.

So it was obvious that if they were to succeed, they needed to include the operators very early in the process, and have relatively short iterations, with rather fluid requirements. They wanted to be Agile. However, in order to get the contract they needed to hide their Agility! They were required by DoD to provide "Waterfall" type artifacts at various milestones in the project. They actually had to hire people to produce all these "Waterfall" artifacts, and deliver them to DoD. These people were not contributing anything to the software, they were just producing heavy documents and sending them upward to be able to keep the contract.

In the end they succeeded, but the presenter said he hoped to never have to do such a contract ever again. I think a part of his soul died during this project and was buried in a large requirements binder.

As a last note on this session: the presenter mentioned that one thing they did that helped them was to leave a certain proportion of sprints (say 1 in 4, or 1 in 5) entirely dedicated to bug fixes, refactorings, etc. We sometimes do something similar, but in our team it seems to be reactive, and somewhat disruptive for the schedule. In their case, they were proactively planning for it from the beginning of the project.

Agile is not a religion

Ce titre est en fait un des 10 conseils pour un coache Agile selon Rachel Davies et Liz Sedley. J’ai assisté à leur conférence Top 10 Tips for Agile Coaches avec plaisir. Elles ont commencé par le dixième conseil : Get Introduced. C’est dans la logique des choses. On se présente, ont dit nos attentes, … Je me dis soudainement que je ne me suis pas présenter lorsque je me suis assis entre deux personnes. Normalement je me présente. Neuvième conseils, Agile is not a religion, ça fait sourire toute l’audience. L’homme à côté de moi griffonne sur un bout de papier et le tasse sur mon côté comme pour me le montrer. Je regarde et il est écrit qu’il est un évangéliste. Je suis surpris, … huitième conseils, Show respect ! C’est ce que j’ai fait.

Vous devinez, la présentation fut très intéressante, principalement pour un coache Agile. Ce que je ne suis pas. Mais certain conseil me seront très pratique lors de mon retour.


Si vous voulez connaitre les 10 conseils, voici leurs livres qui fut le premier meilleur vendeur de la conférence 2009.


Wednesday, August 26, 2009

Kill the Gatekeeper!

On Tuesday morning I attended an experience report given by Francis Lacoste, who's working on LaunchPad (not to be confused with our LaunchPad). He basically described the changes they made to their integration system to render it more Agile, more "continuous".

They initially had a single "Main" branch that they tried to never break, a bit like what we have in our department. They had a GateKeeper that would validate all changes submitted by developers, and only if everything passes (build, unit tests, etc.) would the change be applied to the main branch.

They realized this was preventing them to fully implement continuous integration, so they now work with a "stable main" and a "devel main". The developers can now push changes at any time they want to the "devel main". The gatekeeper resides between the "devel main" and the "stable main".

(Note: the gatekeeper in their case is not a human being, it is an automated tool)

Keynote


We were all staring at the freakish picture of the speaker, Alistair Cockburn, when a guy emerged from the door and crossed the entire room, playing bagpipes along the way. Cockburn then appeared on the stage and started reciting the "Agile" version of some part of Shakespeare's Julius Caesar... That's how the keynote speech, titled "I come to bury Agile, not to praise it", started.

It was an interesting talk overall. Here are some points Cockburn made during the presentation (not an exhaustive summary):

  • When we pick items from the backlog, we should not just consider business value; technical risk is also an important consideration. We may want to start working on the riskier things earlier to get a lot of knowledge as fast as possible.
  • The "gold card": when a developer gets a "gold card" (I'm not sure through what mechanism), that developer gets to work on anything he wants for a couple of days. Cockburn seems to be in favor of having some sort of "thinking time"!
  • We should stop talking about "Agile". We should bury the term itself, since all those practices are now part of the current way to do software engineering. It is now time to consider experimenting with newer techniques.

TDD Clinic: C++

My last session for Monday was a workshop on applying TDD (Test Driven Development) to C++ code given by James Grenning and Bas Vodde. Most of the session was spent working with actual code, doing pair programming. We needed to implement a circular buffer; I suggested to my team that we use a std::deque, but James considered that cheating and would not allow it.
;)

Working through the exercise was very instructive on the dynamics of TDD. As you can imagine, given the short time frame of the session, the exercise was not overly complex. However, the way the exercise was constructed lead us to cause (and find) a couple of regressions as we were implementing the circular buffer. One important thing I learned about TDD is that when you write code to pass a test, you really must write a minimal implementation, something that will just barely pass the test. Then, you write another test, and you implement something to barely pass that test, and so on. Only that way can you make sure that the test suite your end up with has maximum coverage.

At some point during the exercise, James Grenning mentioned a book by Michael Feathers that could prove useful for our department (if we don't already have it, I don't know): "Working Effectively with Legacy Code". I should also say that Michael Feathers said in another session (I'll post about it later) that "legacy code is code without [automated regression] tests".

XUnit Smells

I think unit tests are a bit like dental floss. We all know we should use it daily (the floss), but we still don't do it. The knowledge of its usefulness does not automatically mean we are going to actually do it. The same goes with unit tests. I believe every single developer in our department is aware that unit tests would improve that quality of our product, but we still don't do it.

I keep wondering: why can't we have some working (and useful) unit tests for our software, when everyone else in the industry seems to be doing it? I actually took active part in an effort to establish a unit test suite for our product a couple of years back, and it failed. It didn't fail because of a lack of commitment from the team, on the contrary, everyone was on-board trying to make this work. But it failed because the maintenance costs for the test suite were too high with respect to the benefits it provided. That is why I thought that the XUnit Test Patterns and Smells presentation would be useful for our team. What was wrong with our test code that made it unmaintainable?

The presentation was given by Gerard Meszaros. The first thing he said that really caught my attention was this (he called it a sobering thought): "there should be at least as much test code as there is production code, or maybe twice as much." For me this was a pretty shocking declaration. What he said next was not nearly as shocking, and it made a lot of sense given the first assertion: "test code should be much easier to maintain than production code". Another point the presenter made is that we should design for testability upfront.

Tuesday, August 25, 2009

Uncle Bob

Lundi après midi. Je m’étais fait recommander d’assister à la présentation de l’oncle Bob. http://agile2009.agilealliance.org/node/908 Quelle présentation (Craftsmanship), mais surtout quel orateur! Wow! Il a su garder l’attention de la salle du début à la fin. Ces exemples était frappant, un peu caricatural, drôle et toujours vers la direction qu’il veut nous amener à penser. Une bonne introduction à Agile avec des phases clef comme : the only way to go fast… is to go well. QA should find… nothing! Manual test scripts are Immoral. Bon, ok, hors contexte je ne sais pas trop ce que vous penser de ses phrases. Elles ont probablement besoin de chair autour mais vous voyez l’allure de la présentation.

A Developmental Path to Agile Leadership



Lundi avant midi, j’ai assisté à une excellente présentation donné par un psychologue: Developing Agile Leaders and Teams: A Developmental Path to Agile Leadership. Après quelques minutes de présentation, l’idée semble simple; comment devenir un meilleur “expert” ou leader? Avoir une meilleure écoute, ne pas être sur la défensive, le but n’est pas de savoir qui a raison mais de collaborer ensemble. On se fait dire qu’il faut s’améliorer, cette présentation donne une direction à prendre pour le développement personnel, de l’équipe et de la compagnie.

http://agile2009.agilealliance.org/node/2344




Monday, August 24, 2009

The Agile Testing Quadrants

I think one of the areas where our department has the most to improve is testing. We had a couple of regressions lately that seem to prove the point that a more robust testing scheme must be put in place. That is the reason why I decided to focus my first day at the Agile 2009 Conference on Agile testing.

I attended a presentation by Janet Gregory called "Using the Agile Testing Quadrants to Plan Your Testing Efforts". Right off the bat, in the five first minutes, I realized that the presentation was intended for a testers audience! That reminded me that software teams usually have both developers and testers. At this point I got in a discussion with the people at my table about the developer/tester ratio. For most of them, the ratio was between 2 and 4 developers for 1 tester. Another interesting point is that some companies have a separate Scrum team for testers.

The presentation was mostly about showing the contrast between the "old" approach, where most of the effort is spent on manual functional tests, and the Agile approach, where most of the effort is spent on automated/unit testing. The presenter insisted on the need for planning the tests and allocating a budget for building the test infrastructure. She also insisted that the testers should give the tests to the developers very early in the process, even before the coding actually takes place.

The "Agile Testing Quadrants" themselves (from the title of the presentation) are a way to classify the tests according to different dimensions. The diagram from Janet Gregory's presentation is probably the best way to explain this:


Some tools were mentioned by the presenter (mostly for web development though):

Agile Requirements Validation

Finally I have a little time to blog, so I'll talk briefly about the first presentation to which I assisted: "Agile Validation is Continuous and Collaborative: A Field Study of Agile Require[ments Validation]", by Rosalva Gallardo-Valencia. I wasn't particularly interested in the subject, however that was the only presentation given on Monday morning.

The researcher presented the results of a field study she conducted. She found out that in the Agile companies she surveyed, requirements are validated by the entire team in a continuous manner. She presented a UML sequence diagram with roles as "objects", where basically everyone ask questions to everyone else (product owner asks questions to the customer, developers ask questions to the product owner, etc.) to validate the requirements.

The presentation was not particularly interesting nor instructive, however I clearly saw a parallel with our efforts to integrate the test pilots in our development process a lot sooner, which is very good.

Wednesday, August 19, 2009

Tuesday, August 18, 2009

Flight booked, leaving on Sunday!

As you may know, François G and I will be attending the Agile 2009 conference in Chicago, August 24 to 28. We'll use this blog to share our experience with you as much as possible, and to keep a trace of what we found most interesting. You can use the excellent Google Reader to keep track of our postings and be notified when we write something.

Don't hesitate to use the comments to post, well, obviously, comments, but also questions, etc. You can also take a look at the schedule and tell us what sessions you think will be the most interesting.