Software Maintenance

Last year we had some bad performance problems in our system. The users decided to turn on some functionality which delay processing until night. It was a valid work around. But it did not cure the root cause of the problems. This year there was a worry that the problems would come back. So the big boss put together a tiger team to attack the problem. He chose one of the new members of our team to represent us in the tiger team.

There was one big problem with this plan. Our system is complex. No matter how competant a developer you are, it just takes time to learn the ropes. The new developer from our team attended the tiger team meetings. However she just ended up confused. She asked me and our team lead for help. I was chosen to assist.

The new developer provided a document that listed ideas from a recent brainstorming session that the tiger team had. The new developer could read the words from the document. But it was all Greek to her. I defined some of the key terms. Then I tried to paint the big picture of what the tiger team was kicking around.

Today our new developer is tasked with producing a proposal for changes to our apps in order to alleviate performance problems. Good luck. For grins I got sent a copy of the proposal. I knew we were in trouble when the main app names from our suite were mispelled. Ouch.

My team lead thought we could come in and rescue this effort. I advised against it. This thing is going to need to fail hard before people in charge understand that you just can't throw new resources at a tough problem and expect much good to come from it.

My team got some trouble tickets from the customer. The team lead assigned a high priority problem to me. I studied the error messages. Then I took a look at the code. I deduced that if all was set up correctly, there was only one way these errors could occur. I sent directions to the customer on how to reconfigure to avoid the error. Then I got tasked with helping out another team.

My recommendations were met with a lot of resistance from the customer. They questioned my logic. And they were not happy with my recommendations. The top manager from the customer organization asked me to investigate further. Then the top manager from my company told me to get busy on this.

Luckily I had finished helping that other team. I decided to not take on any other work while I worked this top priority problem. I looked deeply at the examples from the customer. I traced each SQL statement with the data from the production database. That's when I spotted some bed setup data.

My failure was that I assumed all the setup data was correct. Incorrect assumption. I also did not spend enough time on analysis because I was busy. Error number two. Now armed with the full picture, I can address the customer problem. Although they were unhappy with my first attempt, I am sure they will be relieved when I correct the underlying problems.

Today our customer reported a few high priority problems with our apps. My team lead had me look at one of them. Based on some business knowledge, alone with some screen shots of the error, I figured it out. After that I thought I deserved a lunch.

After I picked up some take out, I found that a meeting was called to talk about the trouble ticket assignments. I thought I was safe because I knocked out a problem in record time. Nope. A guy on another team was having a hard time and needed help. I wanted to talk with him. But he was stuck in meetings all afternoon. I went with the one email he was able to send out.

I tracked down the processing that had occurred on the data. There was a lot of processing that looked missing. That was a good data point. I traced back the properties of the data and found out we were not supposed to do any processing due to some initial requirements. After I talked with the developer needing help, I found there were more examples of the problems.

I analyzed the data and confirmed we erroneously skipped some processing. At first my findings matched those of the developer. There was no way the code was going to skip this stuff. Then I recalled helping another developer on this team. He had some code that was loading variables in the wrong order. He fixed the code, but not clean up the data. The fix was also late to make it into production. Ouch.

Once I explained the history of the code changes, the light came on in the developer's mind. He said this explanation matched some other weird errors that got logged. I am thankful I knew about the work of the other developer (who has since left the company). Sometimes a little knowledge of past work goes a long way in bug tracking.

Last year I needed an upgrade of my Rational Tools Suite. The system administrator who did the upgrade was unable to complete the task. I asked him to try again. He could not get the new software installed. I then asked him to escalate the problem to IBM. No dice.

My virtual desktop was without Rational Tools. Ouch. The system administrator recommended he redo the whole virtual desktop from scratch. Seeing how I just lost my Rational Tools, I decided against that. I did have one other virtual desktop that had the right version of the Rational Tool Suite. With the two virtual desktops, I am still able to do my job.

The plan from my manager was to get a new virtual desktop with all the software installed. A member of the customer organization was to get me the new desktop. Well eventually I got it. But nobody was scheduled to install all the software. Eventually somebody called to see if the trouble ticket to get my new desktop could be closed. No way. I don't have a working machine. Now my team lead has to sort through this mess. Oh boy.

There is a six week Ruby on Rails training class starting up in NYC. Developers are getting salty because it costs a whopping $2800. I guess if you are a starving developer, $2800 is way over your training budget. But wouldn't you want to get your employer to foot the bill.

The cost does not sound that bad based on what you are doing. I have gone to numerous one week classes that cost $2000++. Compared to those, the six week Ruby class seems like a steal. Then again, I got my employer to eat the full cost. Sure it got counted against my training budget at work. But no money flowed out of my wallet for the class.

The pricing model itself almost classifies the course as employer funded training. I am really glad my employer provides me with a guaranteed benefit of paid training each year. The funny thing is that my employer is loose with the money, but never wants me to be away from my day job. Weird.

These days I usually spend the training budget on college classes that meet in the evening. So it turns out to be a win win. However I pay a whole lot less for my college classes than the Ruby course. It does not matter. I can't get off from work for a week, let alone the duration of the six week Ruby course.