Arrizza

Adopting XP/Agile

2011-05-09T01:07:00.000-07:00

Can I recommend an approach to you? Use the XP practices as solutions for problems you are already having.

For example, someone checks in some code that causes the build to break => install Hudson http://hudson-ci.org/ and add everyone on your team in the email list. Too heavy handed? Install Hudson anyway and just send the emails to yourself, then notify the person who broke the build (face-to-face, nicely).

Now the marketing: call it the "broken build early warning system" or something similar; use your current team's vocabulary or draw from your project's domain. The idea is you advertise the XP principle's benefit without necessarily mentioning XP or Agile or even a process change at all -- this takes the my-process-is-better-than-your-process argument out of the discussion.

Just added code which caused a regression error => see if you can add an automated test or xTest unit test that would catch it. Then show it off as almost-zero impact ("literally took me 2 minutes to write that test") with high impact ("never see that bug again!") activity.

Found it too hard to predict 6 months into the future => break it down into two week chunks. Call them "mini-milestones" or something similar from the PM's vocabulary.

etc.

Your adoption of the practices will be naturally guided by the failure modes your team is experiencing. These failure modes exist in your specific team because of specific underlying weaknesses in the team dynamics, team members, development environment, management, etc. As you address them one by one, it's easier to deal with the new ones that crop up. Fixing them in one large changeset is difficult and will commonly fail (see "Leading Change" by Kotter).

Note James Grenning mentions a similar approach in his reply. Perhaps this is similar to his -- with a slight twist?

The top benefits of this approach are :

You are helping to fix concrete, real, live problems your team is immediately facing. It gives a large amount of credibility to the practice. It isn't just "fluff" or some "guru" or some "fad" or you "just padding your resume", the XP practice is a real solution to a real problem. The adoption is not "too hard" or "too much work" simply because it needs to be done. It is very hard for someone to argue against adopting it because it solves a big problem kicking your team in the head.
You only introduce the Agile practices that your team really needs. This addresses the "if it ain't broke, don't fix it" argument. And again it adds credibility to the practices you adopt. There is no "extra baggage" or "fluff", everything you do is direct and concentrated on problems your team is actually facing. BTW all of the objections like "fluff", "fad", "too hard", "extra baggage", etc were actual arguments used from time to time on projects I've worked on.
Fixing one problem will expose or highlight the next problem in turn. The benefit here is that later, in retrospect, the adoption of the practices will seem natural and unforced. This addresses some individuals balking at authoritarian approaches. Also there is credibility added yet again. Changes were done not for the "fun of it" or because "some book said to", you added them only when those practices were absolutely necessary. The post-fact review of XP adoption is much easier to describe and justify.
The practices are introduced one at a time. This gives time for the team to adjust -- and there is always an adjustment that happens. Changing a software development process requires a "complex system" perspective on your part. In any complex system, changing one part of that system will have an impact on another part and that impact is generally unpredictable in magnitude and unpredictable in location. Read Gerald Weinberg and you'll see this is a common thread in many of his books (check the bibliography, Beck lists Weinberg). Also pick up "The Logic of Failure" by Dorner and again you'll see this theme is in his book as well. You need to give your team time, and yourself time too, to figure out what the full set of effects the change you just made will have on your project.
It gives time for the team to intellectually absorb the impact of the change. This will take longer than you think it will and in some individuals the reverberations of a change may take years to quieten. Also the changes are evolutionary rather than revolutionary. This addresses the tendency for some folks to dig in their heels whenever a large change is introduced.
The practices are introduced in priority order (roughly speaking of course). The more common process problems you face will occur more often and so you will get to address those by the first XP principles you introduce. This gets you big, visible benefits early which makes it easier to introduce other more subtle principles later (again see "Leading Change").
It prevents "process wars" from starting up. Changing the development process should be rooted in reality, not from a staunch dogma that X is better than Y. And it should be rooted in *your* team's reality not some hypothetical "average team" working on an equally hypothetical "average project".

The cons here are:

You will not get the full benefits of the entire XP development process. There is synergy among the practices and skipping a practice means you lose out.
It may take a very long time to get all of XP in place, if ever. On the other hand, your team may have a natural immunity to some problems that a given principle addresses.
You can't claim you are doing XP, you can only claim you are "just using some Agile principles". This causes some individuals some anxiety. They want a name for the development process. Also you won't be able to call it XP because if you do you will incite the "we aren't doing XP because we aren't doing principle ABC so why do XP at all?" argument.
There can be a backlash at some point in time. Some folks do not like the "stealth" approach; it seems "sneaky". And even though you have actually fixed real problems in your development process, just the accusation of "sneakiness" itself will be damaging. So simply don't claim you are doing XP, you are just using some Agile principles that seem to work really well in your shop. Imagine that, who knew?!

Refactoring Legacy Code

2010-02-22T21:27:00.000-08:00

Check out: http://www.amazon.com/Working-Effectively-Legacy-Michael-Feathers/dp/0131177052. The Feathers book is very good. Here are some additional strategies to consider.

You will use two basic refactorings: Extract Method and Extract Class. You will use Extract Method most of the time, but that's just a stepping stone to the real big-bang-for-buck workhorse, Extract Class.

The basic thing to realize is that a very large class is a set of smaller classes intertwined together. And that a class is a cohesive set of methods that associate strongly with each other and associate (very) strongly with a small set of variables.

1) The primary strategy to use is:

a) isolate one member variable of the large class and extract it into a private class as a public variable

old: private int mVar;

new:
private class SomeThingClass
{
public int mVar;
}

b) instantiate the new class in the large class

SomeThingClass mNewClass = new SomeThingClass();

c) search for all lines in the class that use or modify that member variable and replace them with calls into the new private class:

old: mVar++;
new: mNewClass.mVar++;

d) start extracting lines of code that use or modify that member variable into the new private class:

old : mVar++;

new: mNewClass.Bump();

and in SomeThingClass:
void Bump()
{
mVar++;
}

e) after a few extractions, you will probably need to rename the class since it's purpose should become clearer as you extract more lines.

f) check for member variables in the Large Class that are not standalone but are attributes of other member variables (for an analogy think "Third Normal Form"). Eg.

string[] mAppleTypes; // holds data
int mNumberOfAppleTypes; // it's an attribute of mAppleTypes

These two variables should be extracted out together.

2) You probably don't have unit tests in place. So add some -- any -- testing, whatever you can manage, it doesn't have to be pretty. They must be automated but they don't have to test at the unit level.

3) No unit tests mean high risk. So minimize the risk by looking for ways to simplify the large class by doing very small, very simple extractions. Typically look for 2 - 5 lines of code you can extract out into a void-void method:

...
stmt1;
stmt2;
stmt3;
...

replaced by:

...
SomeGoodFunctionName();
...

private void SomeGoodFunctionName()
{
stmt1;
stmt2;
stmt3;
}

The key is to find statements that are tightly coupled for some reason, e.g. they all act on a single member variable, they perform a very simple, but atomic (can't be reduced any further) kind of action, etc.

By the way, most programmers consciously or sub-consciously use the "white space" around statements to set them off (and some even put a comment or two above them). This is a coarse way to look for void-void sets of statements, but believe it or not, does work well since you are piggybacking on top of previous developer's understanding of "cohesion".

Once you have a whole set of these smaller functions, look for commonality amongst them. If two or more functions all act on a single member variable or they all modify/use a small number of member variables, extract them out into their own class.

4) Look for methods in the Large Class that repeat the same formal variables in their signatures. Extract these out into their own class. The methods of the new class are the old methods without the formal variables, the member variables are the old formal variables. E.g.

old:

void fnA(int val1, int val2, int val3);
void fnB(int val1, int val2, int val3);

new:

private class SomeNewClass
{
private int val1;
private int val2;
private int val3;

public SomeNewClass(int v1, int v2, int v3)
{
val1 = v1;
val2 = v2;
val3 = v3;
}

public void fnA() { ... }
public void fnB() { ... }
}

Why? because formal variables are tightly coupled to the method that uses them. If several methods use the same formal variables, then those methods are most likely tightly coupled to each other (i.e. they form a class).

5) Repeat 1 - 4 as often as you can. My experience has been that the original Large Class pretty much disappears leaving only a vestige of itself (just like the Chesire Cat).

6) As the primary purpose of the extracted classes becomes clearer, start adding Unit Tests for them that reinforces the intent of the new class. That actually helps clarify their purpose much further and may lead to a splitting of that class again (this is a good thing!).

7) Checkpoint the source files often. You will probably have to backtrack when you go down the wrong rabbit hole and so you'll need a quick way of going back to a known good point.

Not glamorous work, but does pay off. I had one chunk of code that started off as 60,000 lines of VB and C++ code, boiled it down to 12,000 over several months. Lot of duplicate code disappeared, a lot of dead code was deleted, and my personal favorite, useless code was deleted. Useless code is code that is executed, but never actually changes the state of the system (yes it can change variables but those variables aren't significant to the system's purpose, so they can be deleted along with all the code that acted on them).

Refactoring tools, an analogy

2009-03-17T23:51:00.000-07:00

Refactoring has nothing to do with tools (although they help speed things up). And refactoring has little to do with renaming variables (although that is one technique used).

Take a polynomial x**2 -1 and it factors out to (x-1)(x+1). Clearly you can evaluate x**2-1, just as easily as you can evaluate (x-1)(x+1). But from one perspective, "(x-1)" is an easier component to deal with and therefore "(x-1)(x+1)" is a "better design" because it communicates the intent of the polynomial better to humans.

If I had a tool that could automatically factor x**2-1 into it's constituent parts, that'd be great but it may not do it in the way that best communicates what the polynomial is trying to do. The audience for the factorization is the next mathematician who is going to be looking at my solution. So I want to help him/her out by formatting the polynomial to show my understanding of how it works.

If I renamed 'x' to 'y', that doesn't really help them. But if I renamed it to "nuclear_core_temperature", perhaps it would help them out, so I go ahead and rename it.

Refactoring is just like that, but for software.

Refactoring, OO and tools

2009-03-17T23:34:00.000-07:00

The refactoring yahoo group recently had a series of posts regarding OO and tools.

As originally defined, refactoring is not an isolated activity. It fits into an overall development sequence to allow good, fast and flexible solutions, followed by a slow, deliberate, reasoned task to clean up the design and implementation. More specifically:

1) get a tiny bit of your task implemented any way you can, writing the tests first.
2) refactor the bit of code you just wrote, paying a lot of attention to the current design and to the potential design your code could or should fit in to. Use the tests to confirm you haven't broken anything along the way.

#1 is the good, fast and flexible solution. Get it to work, get some proof it's working via the tests. Don't worry about the design at this point.

#2 is the post-fact, deliberate, reasoned approach to clean up the "mini-mess" you just made. This is where the design emerges (yes, "emergent design" is a by-product of refactoring).

Why do it in this order? Because #1 gets work done fast, very fast and #2 gets code that is clean, very clean. If you do #1 without #2, you get slop. If you do #2 without #1, after a period of time you bog down in a lousy code and you can't do #1 very quickly at all.

OO is a very nice way to structure code, to clearly specify a design "in the small". So refactoring with OO programming/design is a natural fit, but OO is not a prerequisite for refactoring. You can refactor procedural languages, OO languages, assembler, whatever. Each language has it's own set of idioms that are supported via it's syntax and/or supported via de facto idioms (read "patterns") accumulated over time. You refactor your quick and dirty code to these idioms, patterns and design so that it communicates your intent to the next developer.

And it should be clear now that refactoring tools are not necessary viewed from this perspective. They help you do the cleanup quicker, that's all. Design and architecture is and will most likely remain a very human exercise simply because it is for humans.

Moved my weblog here...

2009-01-26T21:01:00.000-08:00

I had started my weblog at http://www.arrizza.com/cgi-bin/pub?WebLog. Moved all the posts to here. The dates on the previous pages are when I originally posted.

Using the XP Planning Game in a non-XP site

2009-01-26T20:58:00.001-08:00

-- Jan 20, 2009 Using the XP Planning Game in a non-XP site

The Planning Game is probably one of the best parts of XP and it is loved by the Customers. They very much like the ability to control the project on an on-going basis and the visibility it gives them into the software development cycle. It clarifies the delineation between the business folks and the development folks and who makes what decisions.

To use it in a non-XP development environment, do the following:

Call them anything but "Stories". Call them Features, Enhancements, Change Requests, Thingies, or whatever.
Call it anything but "Planning Game". Call it Kick off, Pre-development Review, Feature Review, or whatever.
Do the entire project at once. Have all the Stories... err... Features laid out at the beginning of the project. This is no small task, you will probably have to help the Customers do it.
Calculate the Capacity (aka "Velocity") for the entire project in "man-weeks".
Use Excel, not index cards. Dunno why, just makes them more comfortable. One row per Feature.
Get the Customer to assign a priority number 1-10 to each Feature as a column in the spreadsheet.
Add a ballpark "work effort" for each Feature in "man-weeks".
If any Feature is greater than two weeks (your secret Iteration length), break it into multiple "phases" and rename them to Feature-Phase1, Feature-Phase2, etc. Explain it as an "internal thing for the development team".
Calculate the sum of all the Features. Mark the total in red, if it's over the Capacity, or green if under or equal.
Do the "Verify the Feature" step offline away from the Customers. For any questions, call a meeting.
Do the Feature to Task breakdown offline. Re-calculate the cost of the total project. Call a meeting if there's a significant discrepancy.
Add columns to calculate the status done/not done, the remaining Capacity in man-weeks, and the Features remaining all in man-weeks.
Meet every two weeks to "review the remainder of the project". Change any work-effort estimates. Negotiate any scope changes based on the fact that the remaining Capacity is less than the Features remaining total.
Make sure you do an "Engineering Build" every two weeks. Offer to show the Customers the completed Features at that time.
Make sure you have a Test Site or Test Station ready for them to play with the last Engineering Build at all times.
Make sure you do the Tasks and Features in priority order!

The XP Planning Game

2009-01-26T20:57:00.004-08:00

-- Jan 18, 2009 The XP Planning Game

Customers start with a set of Stories already written up.

Customers will initially need help writing up the Stories. Once they've gone through several iterations and are comfortable with the process, they usually "get it" and can do this work on their own.

Any partially done Stories from previous iterations are also brought into the set of Story cards.

Developers announce their Velocity from the previous iteration

The Velocity sets how much capacity the current iteration has. Using the previous iteration's Velocity for this iteration is a simple scheme to accommodate changes in how fast the team is completing Stories. Note that a previous iteration's Story count is added only if the Story was completed -- no partials allowed.

Customers read the Stories out loud one by one

Having the Customers read the Stories aloud gets the ball going and seems to have the psychological effect of making their request more real. The tone and speed with which they read seems to impart the urgency, importance, or not to the Story.

Customers usually ad lib something during the reading, filling in the gaps present in the raw sentences on the index card. Sometimes the ad lib verbiage, again, gives a tone of urgency or importance to the Story.

Developers ask questions to clarify what the Story means.

The Story is meant to impart a deeper understanding of what is intended by the Customer. Now is not the time to be shy. On some teams it makes sense to have the Coach act as a facilitator to make sure questions are asked and that everyone on the team is participating. It is important that Developers have their skin in the Story.

If any new information is found, notes are made on the Story card. This serves as a mnemonic during the Iteration to help guide the development.

Sometimes just the telling of a Story will suggest a new Story to the Customer. It should be written up immediately. Do not wait till after the Planning Game, it will be forgotten.

Verify the Story

Have the Development team have a discussion around the potential development - 1 to 3 minutes tops. It will feel like a whiteboard session without the whiteboard. Usually any significant questions will pop out of this discussion.

Does the Story fit into the iteration? If not, split it into two or more Stories.

Is the Story testable? Have a quick discussion as to how the Story would be tested in the Acceptance Tests.

If the Development team is unsure of whether or not it can be done, a Spike is announced.

An estimate for the Story is given by the Development team. This is a fairly accurate estimate by this time because of the discussions and questions raised so far.

Have someone keep track of the total Story Estimates so far. If it goes over the Iteration length, stop.

Customers choose the Stories they want to do for the iteration based on the Story estimates and Velocity.

Developers can take a break at this point if they want. The Coach should not. Sometimes during these discussions, the Customer will again indicate why a Story has a particular priority.

Developers go through the Stories one by one

The Developers now have a set of Stories that the Customers want done next. Customers can take a break at this point if they want.

The Developers break down the Story into Tasks. This usually only takes a few minutes.

The Tasks are estimated. Keep the estimates fairly coarse, around a 1/2 or 1/4 day minimum.

Re-evaluate the Story estimate based on the sum of the Task estimates. If the total Tasks take significantly more or less than the original Story value, set that Story aside so it can be brought up to the Customer.

Customers and Developers re-convene

Any substantial changes to Estimates are reported to the Customers. This gives the Customers a chance to swap out a Story if it's cost has become greater than it's value.

Stories for the current Iteration are laid out and the Customers arrange the Stories in priority order. The Priority is marked on each Story index card.

The planning game is over.

The developers begin work on the Tasks based on Story priority. Tasks are done such that priority is given to completing a Story (versus just completing a task).

The End of the Middlegame, reprise...

2009-01-26T20:57:00.003-08:00

-- Jan 17, 2009 The End of the Middlegame, reprise...

I want to reiterate why I'm wading through all the pixie dust. A software development process or methodology exists to deliver something to your customers. So start there and work backwards. Nearly all of the practices you need will fall out of that analysis.

I say "nearly all" of the practices, because there are going to be practices that you do just to keep yourself sane. One of the things about non-trivial software development is that it quickly becomes chaotic and extraordinarily complex in the face of extraordinary pressures to generate perfect code. To keep that all under control, you need some confidence that certain activities are being done at the right time and in the right way.

The End of the Middlegame

2009-01-26T20:57:00.001-08:00

-- Jan 17, 2009 The End of the Middlegame

Un-sprinkle some more of that pixie dust.

We are now back to a point we are gathering up the deliverables ready for the final release, so we can create the DVD's from them.

Usually you've got to now compile the code. You may be delivering an interpreted language, i.e. the source itself, but that is only slightly simpler.

Let's start again. You've got to compile the code. But what code? You need a way to clearly identify the source code you want to compile. Is it a particular branch in your source control tree? Is it a label or tag in your source control? Is it another kind of identifier in your source control (e.g. Perforce changelist, Subversion revision number, etc.)? You are using source control... right?

You've identified the right code to compile. Now you perform the compilation. But you know that the compiler, and the environment on the computer doing the compilation is crucial. So you've carefully placed controls around the toolset and around the computer. You may have even assigned a specialized configuration management team to ensure it is a stable, well-controlled configuration.

You've identified the right code and the right toolset. Now you perform the compilation, using one or more commands to invoke the build. For any sizable project, this generally will be make, nmake, omake, rake, etc., or Ant, scons, or any of the other alternative build systems. Even MSDEV can be invoked from the command line.

The nagging thing about everything so far is that it is manual. Someone has to know a bunch of things. Someone has to be sure to type it all in correctly. But I will stick to my premise: when something is not exciting, people make errors and when something is too exciting, people make errors. The range of tasks that people can repeatedly do well without making errors is very small. So my solution again is automation.

A continuous build system (CBS) that correctly identifies the source version to compile, that has a known and stable configuration and that will invoke the correct commands repeatedly without fail, without error is best. The "Continuous" part of CBS is important, because, again practice makes predictable. The automation ensures there is no deviation from the build process and so as long as it's not garbage in, you won't get garbage out.

"Continuous" means that it is integrated with your source control. For example, have the build system periodically, and often, check if any code has been submitted to your source control. If it has, it should automatically submit a build.

One very large benefit to this, is if someone has "broken the build", you will know about it very quickly and can take steps to fix it very quickly too. One strong recommendation: send both successful and failing build notification emails to everyone on the development team.

Do not just send failing notifications, send both success and failures. There is a positive psychological effect to seeing the steady stream of successful notifications and, less subtly, it lets everyone on the team know that a change has just occurred to the common source branch.

Do not try to send failure notification emails to the responsible party only. The software development team, is a team. If one member makes a mistake, everyone needs to know about it. You are already sharing successes, let everyone share in the failures too.

One extension is to add build documentation to it. Let it analyze the build environment and report any deviations from expectations. Let it keep a log of what it did for every build. Is there anything else you need to generate, then do it!

Another great extension is to get the CBS to automatically generate a unique build identifier. Any software image emitting from the CBS machine can then be identified back to build time. Add, as part of the deliverables, the source control identifier and you can trace back any software image to the source files that generated it.

Another beneficial extension is to get the CBS to publish the deliverables, ready to burn onto the DVD. Not every build is an important release build, but it's good practice to pretend it is.

Do you have a special process for release builds? Then encapsulate the special process in a special machine machine, and you will never have to worry if someone "got it right". You've been generating builds from that build machine continuously...

More books

2009-01-26T20:56:00.005-08:00

-- Jan 14, 2009 More books

More about design than software engineering, but still of invaluable service when it comes to the core of the human side of software:

The Design of Everyday Things, Donald Norman, http://www.amazon.com/Design-Everyday-Things-Donald-Norman/dp/0465067107/ref=pd_bbs_sr_1?ie=UTF8&s=books&qid=1231987133&sr=8-1
The Visual Display of Quantitative Information, 2nd edition, Edward Tufte, http://www.amazon.com/Visual-Display-Quantitative-Information-2nd/dp/0961392142/ref=sr_1_1?ie=UTF8&s=books&qid=1232005637&sr=1-1

Predictable vs. Repeatable

2009-01-26T20:56:00.003-08:00

-- Jan 9, 2009 Predictable vs. Repeatable

I've often been told that the purpose of a software development process is to ensure repeatability. I think there's a bit of a muddle in there.

I think we really want predictability, not repeatability. But something repeatable is predictable. They're synonyms right?

Hmmm. How about this: I sure don't want to repeat a past nightmare project, but I sure would like a predictable outcome for any given steps I need to take in my current projects. If I take stepA, followed by stepB, all the way to stepZ then I predictably get outcome1, and I sleep well at night.

I want predictability in outcome given the steps I take, but I don't want to necessarily repeat the steps I took in the last project. Repeating the steps I took in the last project might not be applicable in the current project and so I may not get the same outcome.

Unfortunately, to most people the use of the word "repeatable" implies repeating the same steps. So the idea of writing down a prescribed set of steps to be followed verbatim for each and every project is what tends to happen, but I don't think it's a very successful strategy. Yes you can repeat the project, but the outcome of those steps may not be predictable.

I teased my cat 3 days ago, and it ran away. I teased my cat 2 days ago, and it ran away. I teased my cat yesterday, and it ran away. I teased it today, and it scratched the hell out of me. Same repeatable steps, different outcome. Yes, a strangely satisfying, predictable outcome. Why did the cat fight back? Because it's not the same cat today as it was three days ago.

Now you could argue that three out of four times we got the outcome we wanted, i.e. a 75% success rate! Gotta love statistics sometimes.

So let's keep going: tomorrow I tease the cat, it scratches me again. The day after that, I tease the cat, it bites me. The day after that, I tease the cat, it bites me again. The day after that, I tease the cat, the neighbors call the SPCA and I'm thrown in jail. And so on. Now what do the statistics look like?

Compare all that to:

I teased my cat 2 days ago, I predicted it would run away, and it did. I teased my cat yesterday, I predicted it would run away, and it did. I teased my cat today, I predicted it was going to scratch me and it did. The difference is a window of opportunity has opened. I can use my predictions to change my actions in anticipation of expected outcomes, i.e. I'll stop teasing the cat.

BTW I don't own a cat, nor do I tease other people's.

Automation Makes Predictable

2009-01-26T20:56:00.001-08:00

-- Jan 9, 2009 Automation Makes Predictable.

You've heard the expression "Practice makes perfect"? Well it's wrong.

Practice the same imperfect steps over and over again and you will surely get imperfection. Sometimes this is called "If you always do what you've done, you'll always get what you've gotten!" Practicing a given set of steps just ensures we can repeat those steps absolutely the same. They may be the wrong steps or they may be right, but all you can do by practicing them is ensure they are the same steps.

To get perfection, you must use intelligence and insight to understand the steps you want to follow... and then practice them. If you practice perfect steps, you will get predictable perfection.

Ok. Great. So practice makes predictable, but once it's predictable, it's kind of boring, and once it's boring it becomes error-prone if you have to do it manually. All of which points to automation as the right direction to enforcing a predictable process. Automation makes predictable.

Some caveats:

Do we want boring? Yes, we do. Good. Just checking. We don't want surprises, especially last minute surprises. We want predictability.
don't start automation too early. You miss out learning some very valuable things about the process. You need a motivated, intelligent person initially running through those steps to ensure there is a deep understanding of what is going on and why it needs to be done.
Automation makes predictable, which is it's strength. But it's strength is it's weakness. The predictability of it's outcome is had from the rigid adherence of the scripts to their contents. When the scripts follow their algorithm down into oblivion, you won't necessarily know why. It pays to have a trace of what exactly happened.
Automation is great for those parts of the software development process that are fairly inflexible. If there is a lot of complexity, it may still call out for manual processing or perhaps a blend of automation assisting a manual process.

The Penultimate Endgame

2009-01-26T20:55:00.003-08:00

-- Jan 9, 2009 The Penultimate Endgame

Un-sprinkle some of that pixie dust.

We are now back to a point where we are creating the DVD's. How do we do that? Simple: burn the DVD from the software images.

The main question is how do we ensure that the image on the DVD is the one we want? In short the software image has to have some identifying mark that we can detect on the image in the DVD.

One part of this is to designate a build machine that is used only for the creation of final released images. This is the only machine that creates the image and creates the DVD. If the total process is automated, the image on the DVD is the image that was created on the build machine. Even still there is a need to uniquely identify an image on a DVD. I have two DVD's, both with my_software.exe on it. How do I tell which exe is the latest?

One way is to embed a version string. The version string is not a no-brainer. As it turns out the versioning of a software release has become part of marketing. Since the they have been talking about v2.0 for the last 8 months to the customers, you can't rename the image "v2.001" at the last minute. One way around this is to have a rather long version string of which only the first few characters are "v2.0", the rest are for our benefit.

The build machine will automatically (or manually) increment the version string for every build coming off of it. So unique builds have unique strings.

A better way is to use add a build info string into the image. But this isn't a no-brainer either. It only works if every build on every machine is unique in some way. This can be done if every developer's machine has a unique name and a counter. That counter is incremented each and every time a build is kicked off. The counter and the developer's machine name is stamped into the image via the linker (or some language appropriate method) and maybe add some other data for convenience, say the date time stamp of the final link. This way, every machine, every build is unique and it is easy to tell the pedigree of the build from my developer machine vs your developer machine vs the release build machine.

One little twist that is quite appealing is the use of the version control numbering. Some version control systems (e.g. Perforce and Subversion) have a unique number for every submitted change. If that number could be embedded in the image then you have a direct link from the image on a DVD to the source that was used to create it. (Note we always assume there are no errors in the makefiles/Ant/scons/rake scripts!)

And finally, the creation of the installation scrips and the transfer of those scripts and software images to the DVD. In this, practice does make predictable, and once predictable, boring. So automation is the right direction to follow.

I've been using "DVD" as if this is the only way to deliver software. Not true of course. You will have to replace your delivery mechanism as appropriate.

Books

2009-01-26T20:55:00.001-08:00

-- Jan 6, 2009 Books

Quick go out and buy these books immediately:

Waltzing with Bears: Managing Risk on Software Projects, Tom DeMarco, Timothy Lister http://www.amazon.com/Waltzing-Bears-Managing-Software-Projects/dp/0932633609/ref=sr_1_12?ie=UTF8&s=books&qid=1231309159&sr=8-12
Are Your Lights On?: How to Figure Out What the Problem Really Is, Gerald Weinberg, http://www.amazon.com/Are-Your-Lights-Figure-Problem/dp/0932633161/ref=sr_1_1?ie=UTF8&s=books&qid=1231309355&sr=1-1
Sources of Power: How People Make Decisions, Gary Klein, http://www.amazon.com/Sources-Power-People-Make-Decisions/dp/0262611465/ref=sr_1_1?ie=UTF8&s=books&qid=1231309472&sr=1-1
The Logic Of Failure: Recognizing And Avoiding Error In Complex Situations, Dietrich Dorner, http://www.amazon.com/Logic-Failure-Recognizing-Avoiding-Situations/dp/0201479486/ref=sr_1_1?ie=UTF8&s=books&qid=1231309520&sr=1-1
Managing Humans, Michael Lopp, http://www.amazon.com/Managing-Humans-Humorous-Software-Engineering/dp/159059844X/ref=sr_1_1?ie=UTF8&s=books&qid=1231309280&sr=1-1
Made to Stick: Why Some Ideas Survive and Others Die, Chip & Dan Heath, http://www.amazon.com/Made-Stick-Ideas-Survive-Others/dp/1400064287/ref=sr_1_1?ie=UTF8&s=books&qid=1231310167&sr=1-1
How to Win Friends & Influence People, Dale Carnegie, http://www.amazon.com/How-Win-Friends-Influence-People/dp/0671027034/ref=sr_1_1?ie=UTF8&s=books&qid=1231310218&sr=1-1

That was close. No one should go into battle without them.

These books are rich. Read them once, you come away with insight. Read them again, and your insight is deepened. Read them again, and it's deeper still. Repeat.

The Endgame

2009-01-26T20:54:00.002-08:00

-- Jan 6, 2009 The Endgame

Why are we doing all of this? Why the angst over this process or that? Why have a Software Development Process at all? Why don't we just start coding until we're done?

It's all in the endgame.

Let's start there, start at the end of the endgame. You're at the customer's site. Sprinkle some of IBM's magic pixie dust and the DVD's are in your hand and you are ready to install! That was easy.

This is the moment that you've been working so hard for during the last 8 - 12 months. Everything on that DVD is a great big neon sign about your company. It screams at your customer whether you're a professional, seasoned company or "just a couple of guys in a garage" kind of operation. The installation software and the ease of installation is the first concrete clue you are giving to your customer that you've done all of your homework and your software is worth the bazillion dollars they just paid for it. It is your face.

If you do the installation for them, if better be foolproof. You do not want your installation software making fools of your installers. They are your face. If they are embarrassed, your company is embarrassed. If they look like idiots, your company looks like an idiot.

If your customers do the installation themselves, the installation software better be even more foolproof. The customers don't know and don't care about your installation. You can't trust the customer to not screw it up, so your installation has to be even smarter. If it's stupid, your company looks stupid.

Ok, let's try it. Slip the DVD into the drive and whirrrr... bump.

Wait a minute. You tested your installation software right?! You know it runs on the customer's equipment sitting in front of you... right? It runs on that OS, right? It needs how much disk space? Is it an upgrade or a fresh install? Doesn't the software figure that out for you?

So when should you test the installation software? Good rule of thumb is "Practice makes predictable". So practice the installation on every development build. If there's a glitch on an upgrade or a new installation, you'll know that whatever development occurred during the previous build and the current one is most likely the culprit.

You'll need a full set of target machines that you can reset to an original state quickly and easily (kind of screams for a VM solution doesn't it?). After doing it a few hundred times, you'll most likely want to automate the installations. Hey and that's a nice feature for your larger customers too.

And you'll most likely want a developer dedicated (targeted?) to maintain the installation software. Oh and a good mitigation is to have code reviews of the installation software by a couple of other folks. If your main developer leaves, you have some scrambling to do but at least they'll have seen the code a few times before they're thrown in the fire.

At this point, you'll have very, very few surprises. If there is one, you have a whole bunch of data and a few tools at your disposal to fix it quickly.

That's why we have a Software Development Process, to give us the data and tools we need to fix problems quickly.

Adoption of XP (Extreme Programming)

2009-01-26T20:54:00.001-08:00

--Jan 5, 2009 Adoption of XP (Extreme Programming)

The question was:

Is it better to adopt a gradual transition to agile development or commit to full XP, for example, and adjust later?

My answer was, and is: Gradual transition.

The Big Bang approach does not work unless:

your team is already working well together. They are sincere about trying XP, they are technically and emotionally competent. e.g. if there is a weak member on the team or a passive-aggressive on the team, his/her attitude will show up in with a big loud bang and you will blame XP for it.
your development team is really on board with XP. They must not have a "say-yes-to-management-because-I'm-scared-to-get-fired" attitude. XP requires capital-P Participation from everyone. Especially when it is being initially rolled-out. e.g. They can't say "I'll refactor" and then pull out 2 lines out of their 150 line function and claim "I refactor!"
your development environment is already up to snuff. XP stresses development environments. e.g. if you don't have your source control under control it will show up with a big loud bang and you will blame XP for it. In my experience, there is a tremendous amount of political and technical inertia in the build system. I guess the difficulty in getting the thing to work at all and the criticalness of it causes the inertia. You might not be able to change it because too many people refuse to allow you to.
management is really on board with XP. They must not be doing "airplane-magazine" management. e.g. they say yes we want emergent design, no BDUF, etc. and then call week-long design sessions with "the boys". They can't say "1 week iterations except this week and next week and that week"
management is patient. They must recognize that your development environment and your development team are tightly integrated as a symbiotic whole, changing an aspect of one will affect the other. Those changes perturb other aspects of the whole and so on. It is a dynamic system that is held in equilibrium by a lot of effort by a lot of people, changing any part of it requires those people to understand at a fine level what is required for them to do in the new setup. Your management has to recognize that it will take time to change all these things and have it run as smoothly or better than it did before.

Now assuming you have all of these things in place, why do you want to convert to XP?

You have a well-run shop with intelligent, coherent management and developers! :) My point here is that chances are you are considering XP because you are having problems not because things are going well. Because of that you will face a lot of political, technical and personnel problems. I suggest, then, going slowly, i.e. gradual transition.

Here are some guidelines to achieve this gradual transition.

First step, analyze the company's context. What are they currently doing? what are they competent at? what are they most amenable to changing?

Second step, determine which one of the XP practices is most likely to be acceptable to them. Which one requires the least amount of change for the company? Which one are they already doing or almost doing?

Third step, introduce the practice to them and then "predict" where there will be failure points. For example, say 1 week iterations are the most likely to be accepted. Some failure points with 1 week iterations are:

a task can take longer than a week, how do we merge that task in?
the integration time for only a week's worth of work is pretty small, but as a percentage it can be fairly high. How can we reduce that integration time?
it seems that Fred likes to claim he's "done" but every week we have to go back and "fix" his stuff. What can we do about that?
etc.

If this step is done well you should have a bagful of credibility at this point. You have to sell this to developers and managers alike. If the developers don't buy-in, they will sabotage your efforts. If the managers don't buy in, they will override your efforts of just cancel them altogether. Beware of the manager who wants a complete process revamp "yesterday", this stuff takes time.

Fourth step, use that credibility to introduce the other practices to fix the failure points that you "predicted".

Einstein and Thought Experiments

2009-01-26T20:52:00.000-08:00

-- Jan 5, 2009 Einstein and Thought Experiments.

Started this Weblog today.

Einstein used thought experiments.

When I was in school, one of my professors said "Think the unthinkable!" How do you think something unthinkable? Perhaps he was just saying it for the managementese of it. Often management-speak can have some interesting weirdness on the outside, with a soft melty center on the inside. In this case, it may have been a rehash of "Think out of the box", but knowing the professor, the person himself, I don't think so. I think he meant it as a sort of dare, as a challenge. A challenge which is also a conundrum makes it pretty odd.

But back to Einstein. He thought the unthinkable. But in this case it was a different connotation of "thinkable". He used thought experiments to imagine semi-real scenarios. Take a train and run it at near the speed of light ("Relativity: The Special and the General Theory" by Einstein himself). It's clearly possible, although highly improbable. But he used the questions it raises to his advantage. He could explore an unknowable (unthinkable?) scenario and use it to flesh out what he wanted to study. What would it be like? What would happen? Why would this or that happen and not that?

Strangely enough, I read in "Sources of Power: How People Make Decisions" (Gary Klein) this is exactly what normal everyday people do. Although perhaps not in an analytical or methodical way as Einstein did. They run simulations in their head of what could happen if they made this decision or that. They do informal thought experiments to themselves.

But out loud, this freaks people out. I've tried it out in conversations with people, and it makes them uncomfortable. It's like imagining something which isn't quite "every day real" makes the analysis and results just not quite right. It's suspect. And besides, how can you trust someone who makes up a story about trains going the speed of light?

But I'm not sure why. A thought experiment is like an Analogy on Steroids. People are ok with analogies, although they do spend a lot of time picking apart the analogy rather than understanding the principle the analogy is trying to reveal. In a thought experiment the analogy is there but it isn't one thing like another, it is reality, but perhaps just a rare edge case of it. Einstein didn't really make up a story, he laid out a very possible, realistic scenario. Not something he (or anyone else) could realistically achieve, but the scenario is not fantasy, everything he said could possibly occur. On the other hand, analogies aren't that realistic.

In any case, I'm going to reverse engineer Einstein and use thought experiments.