Do NOT use Agile! (sometimes)

A good friend of mine received some distressing news recently. Her client was "migrating" to Agile methodologies, and wanted to use Agile to run their project. She had a theoretical understanding of what Agile is, but no practical idea of how to implement it. She asked if we could talk through it, and I agreed.

The discussion was really frustrating. About three minutes into the description of her project (including the people and politics involved), I realized that an Agile approach to their project was fraught with peril. It reminded me of something that you won't hear me say very much: there are times when it is a BAD idea to use Agile, or at least, unnecessary to do so if there's any organizational inertia.

Let's cover a couple of non-negative situations where Agile isn't a necessity to successful project delivery.

Short Projects

Projects that would typically encompass only one or two Agile iterations are probably easily handled via any methodology. As such, don't feel like you have to jump through all the ceremonial and artifact-centric hoops that are required for most Agile approaches. To be clear, you CAN run these projects in an Agile manner, but it's not necessary. Note that software package/solution spikes fall into this category for me (since they take anywhere from a day to a couple of weeks at most).

Simple Projects

A short project isn't necessarily simple, and a long project isn't necessarily complex. I'm defining "simple" projects as projects that are solving problems that have already been solved in well-known ways, or projects whose outcomes aren't very important. R&D projects might fall into this category. For me, secondary ETL projects are commonly in this category: a client needs to add a new data feed to an extant data stream, and the input mechanism is going to be the same as the original one. This is probably pretty simple, and doesn't mandate an Agile approach.

Fixed Triple Constraints Projects

For projects where each side of the triple constraint triangle are truly fixed, Agile approaches aren't necessary. Contracts with governmental agencies might be this way, for example. Projects where the budget is fixed with a hard deadline and a specific scope can succeed with a Waterfall approach. I would still argue that the team *could* improve delivery iteratively via introspection and subsequent changes, but the project could still be successfully delivered without this.

Alright, now let's get into the less happy reasons to avoid Agile. Note that none of the following reasons are indictments -- Agile practitioners should also be pragmatists, and that's where these observations come from.

Teams Lacking An Agile Spirit

Not everyone can embrace change. Not everyone is confident enough to be kindly yet firmly honest with their teammates. Not everyone cares enough to do more than "their share." Not everyone wants to take on the responsibility of self-direction and organization. That's all OK -- Agile really isn't for everyone. If you have team members that are REALLY against Agile practices, it might not be worth trying. Or, you could see if there is another team on which those folks would fit better (which would be my advise). However, sometimes there aren't *any* teams that want to do Agile, because...

Organizations Lacking An Agile Spirit

Some organizations hear about Agile, send people to training, and then try to become Agile by using all of the Agile tools without understanding that Agile is a way of thinking about solving problems and interacting with people. Most of the organizations I've gone into trying to do Agile transformations (moving from traditional to Agile approaches) fall into this unfortunate category. They try to fit their current roles into Agile role titles. They try to create a Gantt chart on an Agile information radiator. They don't understand that Agile is about doing something fundamentally different, and instead fall into the trap of doing something operationally familiar with new tools.

Organizations Where Agile Has No C-Level Support

If you want to make a transformation from traditional to Agile project approaches, you *have to* have high-level executive support. The change is painful for most organizations, and there is typically incredible inertia in every aspect. Having someone high up that can mandate the changes until they are done makes this more possible. The fundamental problem with Agile approaches when no muckety muck is directing the change is that too many people/things in the organization can accidentally or intentionally sabotage the efforts.

Back to the discussion with my friend...

We wound up going through the twelve principles described in the Agile Manifesto. The groups involved with her project could legitimately enact a couple of them, had a chance of adhering to a couple more, but would absolutely fail to adhere to at least half of them. If you an go through that list and be honest about your ability to live them out, you should have a pretty good idea of whether your project is a good candidate for an Agile approach. At the very least, you'll have lots of good talking points about places/things you'll need to change in order to do Agile projects in the future.

In my friend's case, I suspect that the client mainly wanted to "do Agile" with them because they wanted visibility into why it took so long and cost so much to do what I would classify as "simple" projects. I advised my friend to talk to the client and ask if they simply wanted to be on some daily standups to see where the churn happens. Something as simple as that might satisfy the client without having to do a full-blown Agile implementation. I haven't heard back yet, but suspect that this is the track they will follow.

So the next time someone unexpectedly wants to do an Agile project with you, take the time to make sure it's a good candidate. Look for the pitfalls of team and organizational challenges. If the project clears the Agile principles pretty well, go for it. If not, it's worth more discussion before launching into an Agile approach that might've been doomed from the start...

June's Lone Entry

By this time last year, I had written 18 blog entries for the year, with the single entry for June being a book review of "The Magic of Reality" by Richard Dawkins.  I'm behind my mark this year, only having submitted 12 so far.  I would apologize for this, but my hope is that the quality of the content this year has improved.  If it hasn't, feel free to make suggestions in the comments section.  I'm happy to take requests on blog entries.  Then again, if you're already upset by the quality of my posts, I'm not sure you'll like anything I have to say about a topic that might be near and dear to you.  Wow, what a Catch-22 you're in!

With the above said, I don't have any reasonable excuses as to why I haven't written more entries.  I could say that I've been busy, but I don't think I've been any busier than I was at this time last year.  I will say that we've picked up a couple of activities since last we spoke (or I typed and you read).  First of all, we've taken up gardening (box and tomato) and bread-making.  You may recall my earlier blog entry about tomatoes.  That entry was inspired by the fact that Tanya is growing four different kinds of tomatoes in the garden, in addition to rosemary, basil, mint, and stuff in the box.  Overall, the experiments have been fruitful (or leaf-ful, in the case of lettuce).

Bread-making is worthy of a post in and of itself, so I won't comment on it further here other than to say: if you're not making bread for yourself, you should be.  It's delicious, and better for you than almost anything you can buy in a store.  :-)

The second activity is swimming.  Beginning June 1, we started swimming on Monday and Wednesday evenings with North Texas Masters Swimming, coached by Dave Young.  It's only been a month, but I really can't say enough good things about this organization.  First of all, Dave is incredibly knowledgeable about all things swimming.  He's fantastic with instruction, going so far as to record individual laps on his iPad and playing them back to us during the workout for immediate feedback.  He's also funny, which helps when you're struggling with all of the mechanics and techniques.  Secondly, the facility is great and pretty close to our house.  Third, the group itself is friendly and approachable, with lots of different people at different levels offering words of encouragement and advice.  When it gets to be Monday and Wednesday afternoon after work, I'm typically dreading going.  Tanya always talks me into it, and I'm always glad I go.

Beyond that, it's been a pretty normal start to the summer.  The boys are here, and this year we're actually organized about summer activities that will (hopefully) lead to a more productive time than last summer.  We've listed modest goals for them to achieve by the end of the summer, gone through the process of enumerating what it will take to achieve those goals (at a high level), and actually written it all down.  We further spent some time planning how much stuff needs to be done each week to achieve the goals as well as thought about rough dependencies and contingencies.  I've gone so far as to put up tiny scrum boards for them.

As you can see, Garrett is researching colleges and learning touch typing (which apparently he missed in years 1-3 of high school).  Gabriel is going through scripting courses at codecademy.  Garrett will have tasks in flight from multiple goals per week, while Gabriel will be working through one at a time.

I'm actually pretty interested to see how the boys like this approach, since it's very similar to what I do at work every day.  And for those of you that were wondering -- yes, I considered using a Trello board, but no, I do not sing at the daily stand-up with the boys.  I'm not sure they could take that level of energy that early in the morning (10 a.m. CT).

So that's what's going on so far this summer.  Aside from all this, I'm getting back into the game development that I'd meant to do at least a year ago.  I'm working on the Dark Tower board game in clojure/clojurescript.  It's coming along nicely so far, but not far enough to comment on yet.  Stay tuned for that!

Another great outing...

I just wanted to give a quick shout-out to the great folks in the MIS 374 class at UT Austin.  The students and instructors (Clint and Bruce) are fantastic and gracious.  For those of you that don't know, I do a guest lecture for the class once a semester on Agile Principles, and always enjoy the experience.  This semester's lecture was last Monday, and I wanted to touch on something cool.

This year was a new experience, both for the students and for me.  The MIS 374 class has moved into a new classroom, one of the Active Learning Classrooms at the McCombs School of Business.  Instead of being set up as a lecture classroom, the students sit around mobile tables with white boards and displays on every wall.  It made the presentation quite a bit more interactive, as I could walk around the room while I talked.

It also provided an opportunity to trim down the lecture to the bare-bones, this-will-help-you-on-you-project things while making a class-participation exercise possible.  We wound up doing a prioritization exercise, where the instructor took the role of the product owner and did an initial prioritization of the product backlog.  The students then stood around the backlog and asked the instructor questions about why they had prioritized the cards thusly.  They then pushed back and suggested that a different order would be more sensible, or less risky, or higher value based on how the instructor answered the questions.

Hopefully, the exercise made the lecture more real for the students than it was in lectures past.  In either case, I'm really pleased with the new format and the new classrooms, and look forward to doing it again.  Only next time, there will be even LESS lecture and MORE group exercises!

Agile guest lecture series: Methodologies

I'm going to wrap up this series with a quick comparison of the two most common Agile methodologies: Scrum and eXtreme Programming (XP).  If you missed any of the other three entries, you can find them at: Manifesto, Principles, and Pragmatism.

So... methodologies.  In my opinion, the biggest difference between Scrum and XP is in their prescriptions.  Methodologies prescribe things, meaning they have sets of rules about things that you must follow in order to declare that you're adhering (cohering?) to the methodology.

Scrum

Scrum prescribes three things: roles, ceremonies, and artifacts.  Roles include Product Owner, ScrumMaster, and Scrum Team.  Ceremonies (meetings) are Iteration Planning, Daily Scrum, Review, and Retrospective. Artifacts include Backlogs, Burndowns, and (most importantly) Working Software.  Wikipedia has a decent overview of the software development life cycle using Scrum.

I spend a good deal of time talking about Product Owners, Backlogs, and Working Software during this part of the lecture.  It's important that the students understand the Product Owner role so that they know how to manage expectations -- both their own and those of the client sponsor that is filling the role.  I also remind them that they have another Product Owner in the class instructor (see the Principles lecture notes).

Backlogs are important to the students because they will become communication tools as well as information radiators.  Working with the instructor and with their client sponsor to elaborate and prioritize stories makes the project more visceral and manageable for them.  Also knowing that you don't have to keep the entire project in your head all at once is relieving -- you only need a lot of information about the stories near the top of the backlog.

Working software is the most important thing the students can produce as a result of their efforts.  I remind them that in order for it to be declared "working" it ought to be in the client's production environment.  It can't be working on their laptop only unless they intend to give their laptop to the client and help the client wire the laptop into their production environment.  Most of the students look at me like I'm crazy at this statement, and I'm glad they realize the absurdity of the scenario. :-D

eXtreme Programming (XP)

XP prescribes practices.  In fact, it prescribes lots of practices, but I generally only mention 4: test driven development, pair programming, continuous integration, and collective code ownership.  Again, have a look at handy dandy Wikipedia for decent overviews on the definition of each.

I typically only spend a couple of sentences on each of the above four practices.  The main reason for this is that software development as a craft is not the focus of this class -- project management is.  However, I make easy suggestions around TDD and pairing to make sure the students understand that these practices can help them make decisions, keep moving forward, and increase their code quality without a lot of effort.

The final point I leave them with is that they are very likely to encounter prescriptions from both of these methodologies being enacted at the same time without giving credit to each other.  There are a lot of high-functional cross-functional project teams that are gathering around an iteration backlog everyday to answer three questions that then break off to pair program using test-driven development so they can commit code to a continuous integration server that don't realize the last three things I said having NOTHING AT ALL TO DO WITH SCRUM.  Those are, in fact, XP practices that the team has adopted.  If they happen to get into a Scrum team in the wild that is not doing those practices, they have an immediate opportunity to improve the team by suggesting them. :-)

Agile guest lecture series: Pragmatism

Welcome back! In part 1, I introduced the Agile Manifesto. In part 2, I talked about the first four principles that help fulfill the manifesto. The next thing we cover with the students is pragmatism. Specifically, I tell them to identify and do the hard work first.

The good news for the students is that by this point in the lecture we've already talked about at least some of the hardest work that they haven't thought about at all up until this point: deploying their code to the client production system. It's especially difficult because there may be parts of it that are completely beyond their control. Coordinating with client personnel that aren't normally involved in the project, working through whatever release activities are necessary for the client to get code promoted to production, adding IP addresses to whitelists in firewalls, and the like are just some of the things that will be done primarily by people other than the student teams, and as such those activities might be hard or time-consuming from the calendar perspective.

Speaking of calendars, I like to back into a few dates to instill a sense of urgency in the students at this point.

(Note that the lecture was given on March 2nd.)

The calendar begins without the highlighted dates. The first question is: Who knows when everything is due? Typically only one student out of both sessions will know the due date (May 14th, for this year's students). The second question is: Who knows when you should be principally finished with all the work? When is the code complete date? No one ever remembers this date (April 22nd). A more interesting question is why is that the code complete date? It's mainly because the intervening weeks are full of meeting other graduating student obligations, so the instructors want to make sure the students have done most of the work before the nuttiness ensues.

Both of these dates, by the way, are in the syllabus. The instructors have gone so far as to make the entire schedule into a Google calendar so that the students can easily import the dates.

The final date (or first, chronologically), is a suggestion I make for the students. If there's still hard work to be done after that date, the project is in jeopardy. The scary thing about that date is that it's roughly a month after the date of the lecture. At this point, most of the students haven't actually started working on the project itself yet. I urge them to go ahead and start making a backlog of things that they know will have to be done (either because we've talked about it or because it's in the syllabus). From that list, make a guess at what the hardest or riskiest things are and push those to the top. Go over the list with the instructors to vet it with them.

What's a backlog, you ask? That's a good question... one we'll cover in the next post. :)

Agile guest lecture series: Principles

In the first entry in this series, I talked a little bit about the Agile Manifesto. The manifesto deals with the spirit and intent of an Agile approach, but the authors didn't stop there. They also included a set of twelve principles that, when followed, fulfill the purpose of the manifesto.

For the students, most of these principles are not easily applicable. They have other classes and extra-curricular activities. This project is not their full-time job. Likewise, if you're trying to do an Agile transformation, enacting all of the principles all at once can be very challenging. There are lots of personnel and organizational resistances to change. As such, I suggest digesting these principles in smaller chunks. For the students, the first four principles can typically be applied to their projects in fairly straight-forward ways. If you're interested in taking an incremental, iterative approach to Agile adoption, you can do the same thing. Practical application of the first four principles are covered here.

Principle 1!

There are so many important words in this principle. Satisfy. Customer. Early. Continuous. Valuable. While they are all important, the one I focus on the most is valuable. I ask the students, given what they already know about the manifesto, how they might achieve this. The suggestion I offer to them is to participate in and help the client prioritize the work.

Helping clients prioritize work has two net effects. First of all, you stand a much better chance of delivering more valuable things than less valuable things by simply asking the question "is this the most important thing we can be doing right now?" Secondly, and this is especially important for students, is that you have a chance to say "no, that's too much work." Students are in the habit of doing what they are told to do, especially if part of the assignment is "do what the client wants." It's important to realize that agreeing to a lot of work and then not delivering is far worse than being honest about your capacity, delivering a smaller set of work, and having that set of work be valuable.

Principle 2!

This one is a little more challenging to apply to students, mainly because the projects that are taken on during the semester have a deliberately small scope. However, I typically suggest that the students put thought into their change management process (or create one if they don't have one). When your scope is small enough, the likelihood of it changing small as well. However, priorities might change, especially if they are regularly asking the client "is this the most important thing we can be doing right now?"

The other reason priorities might change is because these type of projects always have a two-headed client master. One head is the client that is sponsoring the project, but the other head is the course instructor. The students don't always take into account that the instructors are in the business of selling the brand of UT to the participating project sponsors. As such, they might need the students to do things that they haven't adequately planned for. When that happens, they need to be able to communicate with the project sponsor in such a way that the project can continue to move forward and not have the change negatively affect the relationship with the project team or the class instructor. Serving two masters that might have competing agendas is something that the students typically haven't dealt with much, so having a mechanism to do so is very valuable.

Principle 3! Deliver working software frequently, from a couple of weeks to couple of months, with a preference to the shorter timescale.

Most of the students don't differentiate between "demonstrate" and "deliver." My advice here is to push code to the production environment more than once. 

Getting code off of a student's machine or a development environment and working properly in a production context is typically pretty hard work. Figuring out how to do that early on has lots of advantages. It exposes you to the pitfalls in the client production release process (like QA, deployment, and maintenance). It also introduces the students to the people that will likely have a hand in maintaining the software after the project is over (more on that later). Finally, it gets you into the habit of delivering working software prior to the project deadline, forcing a focus on quality that might be delayed until the end of the project otherwise.

Principle 4! Business people and developers must work together daily throughout the project.

This one is probably the biggest challenge for students. Most of these students don't even work together amongst themselves daily, much less consider bringing a client representative into the mix. Since it's not pragmatic for the students to fulfill this principle completely, I suggest two things here: set up a regular communication method and schedule, and set up a sensible escalation path.

Setting up a regular communication method and schedule is something that will already be mandated by the course instructor. I suggest to the students to make that schedule as frequent as they and their clients can tolerate, especially considering they won't be co-located in most cases. Most students are thinking "a weekly phone call is good enough," but I'll suggest to them that every time they meet to work on the project (including class time), they should pull the client in. Half-hours are actually pretty hard for clients to dedicate to student projects, but a five or ten minute phone call right after lunch is very consumable, since in the real world people like to ease back into work after lunch anyway. In any case, get a regular method (phone call, email, on-site visit) and schedule (every Monday and Wednesday from 1:10 - 1:15 pm) agreed to by all parties.

So what happens when you've got a decision to make or a question you need answered and you aren't anywhere close to your regular meeting time? Have an escalation path to cover communication outside of regular meetings. For lots of clients, having a one-off phone call, text, or email is just fine. What's missing from this communication is a closing feedback loop. Clients in the real world are notoriously unavailable when you really need them. I'll typically suggest the following:

• Start with an email or whatever is the least intrusive for the client, but which they are likely to respond to.
• Escalate to a phone call/text or whatever is somewhat intrusive for the client, that will demand more immediate attention.
• If the client becomes totally unresponsive, escalate to the course instructor. It's important information for the instructor, and they can facilitate with the client project sponsor (or higher ranking folks) as necessary.

The last component of the communication loop is to include timeboxes for responses and default actions with each communication. For the first attempt, include something like "...if I don't hear back from you by <X>, I'll <do the escalation communication>...". On the second attempt, include something like "...if I don't hear back from you by <Y>, we're going ahead and <doing a default action/making a default decision>...". At this point, it's far better to be moving forward, even in the wrong direction, than to get stuck. The client agreed to do their part in making the project successful, so they should be accountable for participation.

Typically by this point, I see looks of confusion on the students' faces. They aren't accustomed to having not only the right, but the responsibility to push back on clients for accountability. This is a very powerful concept that serves them well throughout the rest of the project, and hopefully as they get out into the working world.

The next thing we cover, however, immediately disheartens them. :) Stay tuned for that post!

Agile guest lecture series: Manifesto

A couple of times a year, I'm fortunate enough to be invited to give a guest lecture to the graduating seniors in the MIS department at the University of Texas at Austin. The lecture is typically over Agile principles. Given that it's only a single lecture, I try to focus on things that can be immediately applied to the projects that the students are working on. It occurs to me that the things the students and I talk about are things that might interest anyone that is looking for low-hanging fruit towards making a transition to or simply getting started with Agile methodologies.

 The lecture begins with a simple question: What is Agile? Agile methodologies, in their simplest forms, are popular alternatives to Waterfall methodologies in the software development life cycle. They allow us to adapt to changing technologies and requirements far better than traditional approaches to software development. Agile methodologies are inspired by the tenets of the Agile Manifesto (http://www.agilemanifesto.org/).

 The above question is, unfortunately, immediately followed by perhaps a more important question: *WHY* is Agile? As it turns out, the folks that originally wrote the Agile Manifesto did so because they were tired of doing software projects as if they were "fixed" projects. When I say "fixed," I mean that all three sides of the project management pyramid are fixed but the project is still expected to be successful.

These folks were (and are) pretty brilliant, yet despite their brilliance they still experienced far more failure in software projects (or re-definition of "success" such that it fit whatever was delivered) than actual success. The projects went over budget, or over deadline, or didn't deliver the full scope, or some combination of two or three of these three. And inevitably, the quality was never good enough. There had to be a better way to make software, specifically business software, more successful. When they put their heads together can came up with the original Agile Manifesto, they did so with an eye toward which precepts and principles would actually lead to working, valuable software in a reasonable amount of time which allowing the delivery team to learn, grow, and enjoy professional life along the way.

So what are these vaunted tenets? Adherents to the Agile Manifest value:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

Do you see the big, boldy words? That's for extra emphasis! 

To be clear, it's not that the words on the right aren't valuable -- they most certainly are valuable. The direction here is that you have to make a decision between the two choices, favor the choice on the left. Favor individuals and interactions over processes and tools. Favor working software over comprehensive documentation.

One of the most important things to recognize here is that the tenets are very much attitudinal and behavioral; they aren't meant to be only procedural. In order to have a healthy Agile environment, you need to have the right attitude and approach. Some teams can't make that change, and that's nothing to beat yourself up about. For the teams that CAN make the change, however... it's a brave, new world!

Stay tuned for the Principles in the next post!