Tripping on my own feet

by Terry 8/14/2008 2:09:00 PM

Last night at work I made an error I do not think I have ever done in the last ten years of software development. I inadvertently overwrote production software with pre-production test code. I am a big advocate of using source control in software development for any size team, and I intentionally configured the system to force me to use distinct accounts, each with limited access and permissions. This normally serves me well, but not last night, particularly for one portion of my product. This one portion of code I inherited from another team and have had to rebuild it to fit into a new environment and new source control from a completely distinct origin.

I wear several hats at work; software developer, technical architect, system administrator, and change manager. I switch between these roles, sometimes frequently, during the course of a day, like when I release a new version of my suite of products into our acceptance test environment or production environment. Hence the reason for good source control, polished processes and compartmentalized access.

For this one small blob of code, what I have in production is not quite what I have in my source control and development and test environments, nor is any of what I have under source control history a match for the current production code. This is a bad place to be in my position.

As I was configuring a large change in our test environment last night, I made a final update to the test environment. I failed to notice the publication path was the production path (then one path without restricted access yet), which has a similar path as our test environment. I released the code and scratched my head for about three minutes until it sunk in what I had done.

The good news, what there is of it, is the code I released into test last night will complete the synchronization of the inherited code and place it completely into the better managed release cycles. Unfortunately, this new test code depends on changes not yet in production, so it out and out broke.

To make matters worse, as part of another transition, my backups for this one portion of the production environment are not current, and may be gone. I recently change the hosting platform to new servers, twice in the case of the problem code, all with new backups. The old backup files are likely gone, or exceedingly difficult to recover. The new backups are not configured yet.

There is a happy ending. I do have source control and while it did not match exactly what is in production, it was close. I restored to the closest match I could find, made changes to match production functionality, rebuilt and published to my production environment. Life is good again, and life will be better yet in a couple weeks when I release a full version of all related tools with fully synchronized versions. The release processes also includes full backups of all affected parts, so I have a real recovery option should an error occur.

If there is a moral to the story, it is this; Source control is good, backups are better and access control is vital.

Currently rated 2.0 by 1 people

  • Currently 2/5 Stars.
  • 1
  • 2
  • 3
  • 4
  • 5

Tags: ,

computing | Personal

Powered by BlogEngine.NET
Theme by Mads Kristensen

About Terry Losansky

Terry Dee Losansky

I am a software architect, actively practice and teach martial arts and live in Snoqualmie, Washington. I have an amazing daughter who is the jewel of my life.

E-mail me Send mail
Terry's Facebook profile


<<  July 2021  >>

View posts in large calendar

Recent comments



The opinions expressed herein are my own personal opinions and do not represent my employer's view in anyway.

© Copyright 2021

Sign in