Ah the thrills of working a dream. Being my own boss. Creating a company "of the people, by the people, and for the people" so to speak. A company that's not cut from the corporate or venture capital mold where every decision is profit motivated. How fun it is. No two days are the same. Such is life trying to build a software company from scratch with nobody's money but your own. Many folks would call my company a Micro-ISV. A term coined by Eric Sink, founder of SourceGear. I am not a big fan of this term because of its ambiguous perceptions, but nevertheless, the term does accurately describe my available resources. And with loads to do and micro resources, you roll up your sleeves and get busy. You learn to live by these ten words.

If it is to be, it is up to me.

Perhaps had I seen Neil Davidson's Micro-ISV survey before I started this gig I might have reconsidered and stayed marching to the beat of the corporate drum. Yet even in the face of undeniably bad odds, I am hopelessly optimistic because I firmly believe that with hard work, talented people, strong focus and persistence, success is inevitable. This belief doesn't help my resource problem though, and as a result, I do lots of things myself, both technical and non-technical.

Because I do so many different things, I've come to wonder exactly how I would title my work. Thinking purely technical, I would say I am a jack of all trades, which has me wondering… is knowing a little about lots of technical areas better than knowing everything about only one particular technical area?

It's interesting when I discuss software with corporate developers. Many of their issues have to do with people, opinions, and pecking order. Corporate, big money development involves lots of people, lots of specialties, and lots of individual interactions.

The list of specialties involved in creating high quality software is pretty long.

• Analyst
• Architect
• Application Developer
• UI Developer
• Technical Writer
• Graphic Designer
• Project Management
• Web Developer
• Tester
• Deployment Engineer
• Product Champion
• IT Support
• Technical Evangelist
• Database Admin

And the list could go on, even into sub-specialties. I was discussing web development the other day with a friend and I asked a question about JavaScript and AJAX. His response dumbfounded me when he told me he didn't do JavaScript. He writes Asp.Net. The browser guys write JavaScript. This is shocking when I contrast it with a day in my world.

I am not a web developer… but I write a lot of Asp.Net code.

I am not an analyst… but I write a lot of specs.

I am not a tester… but I do a lot of testing.

I am not an IT guy… but I do a lot of server stuff.

So what's better? I think it's less about what you are responsible for, and more about what you are interested in and pursue knowledge of. A person who knows only JavaScript, and is not interested in anything else would not be very useful here. However, if that person knows far more JavaScript because of previous duties and responsibilities, but also knows about Asp.Net, IIS, and Data Access, we might have a place for them.

Speed is what you get with a highly specialized technical expert. I could write anything in JavaScript, but it will take me longer than it might take "the browser guys". Depending on the task, far longer. At $185/hour, I expect that many companies value speed over everything else. It goes back to that profit thing. But take into account the cost of the human interactions between specialists, and those efficiency savings start to get lost.

With technical tasks success or failure is pretty easy to spot, regardless of how long it takes to complete. Whether or not it works is usually easy to judge. So if all I get is five people to create a new software application, I would take 5 jacks like me over five specialists any day.

With non-technical tasks however, it's not that simple. The harder it is to assess the success of the work product, the greater the need for a specialist. I used to think that a smart developer can do almost anything. I still do for the most part, but let's just say that in the 15 months since I launched Kinetic Jump, my opinion on certain non-technical tasks has changed. But more on that next time…

Brian

Can doing things right, bite?

We released our first product, AppLife Update, in late June. This product is an automatic updating solution for .Net that makes it really easy to add automatic updating features to any .Net application. Of its many features, one is that you can apply an update without needing administrative rights. We accomplish this through the use of a Windows Service that can execute an AppLife update package under the security context of the local system. To provide this feature securely we did a lot of work. We require administrative permissions to register any given application before it can use the service. We use RSA public/private key technology, we lock files, we double validate both inside and outside the service, we do lots of things. Somewhere, in all of that security decision making, we decided to sign all of our executable with an x509 Authenticode certificate. Seems pretty benign right? Heck, if you read the Windows Vista Software Logo Spec 1.1, you'll see that Microsoft thinks it's a good idea:

All executable files must be signed with an Authenticode certificate. This includes files with the following extensions: exe, dll, ocx, sys, cpl, drv, scr
Limited waivers may be available for 3^rd party redistributables when singed versions are not available.

Before going out and buying our certificate, I read lots of information about Authenticode signing and it just seemed like the right thing to do. Heck the only real decision I felt I had to make was whether to pay more for a Verisign certificate or go with Thawte. We didn't. (I don't really understand that difference, especially when Versign owns Thawte, but that's a topic for another day. Perhaps the power of a brand?) Signing our assemblies just seemed like the right thing to do. So we did. Carte blanche. All exe files were code signed, along with msi's and msm's.

And when we were all done, we tested, tested and tested some more. We covered just about every potential environment we could muster in-house, and we found beta testers out there "in the real world" to run updates. We were updating in corporate environments, in small business environments, in home environments. Our solution just worked. So we shipped.

So what's the problem?

Along about the end of July we received a support email informing us that a guy in Germany couldn't install our software. He had 64-bit hardware running German 64 bit Vista. Not knowing what his issue was, we went to work trying to solve it. Whenever he tried to install our software, our windows service would fail to start. Looking into the msi logs, we found:

Error 1920. Service 'AppLife Update Service' (KjsUpdateService) failed to start. Verify that you have sufficient privileges to start system services.

We tried hard to reproduce his issue and we couldn't. We did find some issues with our product and 64 bit operating systems, and our product is now better for that, but no matter what we tried, we just were not able to reproduce this. I also got really good at navigating to the control panel and event logs on non-english Vista. This might come in handy some day, if more than a handful of customers ever actually make the move to Vista. This person then discovered on his own that if he disabled his network card, the service could start. Inquiring about this, we learned that his company used a Ken firewall system that is common in Germany, but does not have an English version. After spending loads of time on this and seeing the evaluator's enthusiasm to get to the bottom of the issue disappear (and who can blame him), we gave up. Our best guess at the time was some incompatibility with their firewall/proxy system was causing our service to fail. We really couldn't do much more because after all, we kinda needed him since we were never able to reproduce it.

Then in September, another evaluator reported a similar issue. He too was in Germany, but didn't have any unfamiliar networking software, but they were behind a proxy server. It was then that we first knew that we had a real problem and good or bad, we were happy to have another "tester" since we knew we hadn't fixed it and still couldn't reproduce it. So we littered our service code with additional exception handling and sent him a custom version of the service. He still couldn't install and we weren't getting any information out of our exception handling. This was very odd. It just seemed like either our code was not executing or something was hanging. We didn't have to look through much code since the service really does very little work on startup. One thing it does do is create an IPC Remoting channel. The IPC Remoting channel is based on named pipes. So we went into dissecting every bit of the framework IPC Remoting channel, looking for anything that might fail when executing under the local system account, or in a way that hangs the process. This lead us nowhere, except a better understanding of the framework IPC channels.

This story ends with the final realization that our code was likely never even being entered and with the proclamation that Google really is magical. I really don't remember how, but I found this blog post using Google.

http://blogs.msdn.com/shawnfa/archive/2005/12/13/502779.aspx

And in there it talks about Certificate Revocation Lists and how the .Net framework attempts to verify a certificate and how this can be time intensive. BINGO! And I was not even looking for anything related to code signing.

The revelation was found.

Loading a code signed .Net assembly could take a long time if network access is being blocked to the security profile that is loading the assembly.

We rebuilt the service without signing the service executable and sent it off to our evaluator. Problem solved. After this, I went back and re-read about code signing, and the potential for this issue is just not presented anywhere in the educational literature that I could find. So file this little nugget and think about it anytime there is an inexplicable pause in loading a .Net assembly. Is it signed?

Through all this, I did try to enlist the help of others, but when you can't reproduce an issue, it's really hard to ask anybody to give the time of day to solving it.

And even after discovering this, we made boneheaded mistakes that lead to further issues by just not thinking. This week another evaluator, amazingly again in Germany, couldn't perform an elevated update using the service. It turns out that we missed removing the code signing step on assemblies that the service interacts with during an update process. I felt pretty stupid this week.

What's even more surprising is what the Framework does after the certificate validation fails. The assembly fails to get a specific attribute, but otherwise loads and executes just fine. In both of our problems, the actual reason for failure was timeouts waiting for activity that should not have taken so long. If our time out thresholds were longer, the process would have worked, but been unnecessarily delayed.

So finally, after almost six months since the first report of an issue, I think we have this problem completely solved.

We still code sign our primary executable and our installers, but that's it. We do this primarily because we distribute our software exclusively over the internet and we want to instill confidence in our products. Also, the worst that could happen is some users might experience a delay before our application starts.

The lesson learned here is a tough one. We decided to sign our assemblies based on a desire to follow Microsoft suggestions for trustworthy computing and doing the right thing. I now know that doing the right thing can bite. Hard.

Brian

If software specs are so important, why do so many software teams skimp the most on specifications? Here's the politically incorrect skinny.

• It's very difficult to do it well
• It requires fighting with other people for their time
• Those people you fight with don't think you listen to them, so they don't want to help
• It requires time and money that managers don't want to spend
• Nobody finds it very entertaining, so they are never "in the mood"
• Perception is reality, and creating software, any software, is perceived as progress. Creating documents is not.

We all do this. We all skimp on specifications. To what degree we skimp is the differentiator. For most organizations, about 50% of all software projects fail. Projects fail for lots of reasons, but a consistent theme in post mortem analysis is inaccurate or incomplete software specifications. Is there truth to this, or is it just easier than saying our ideas suck? If there is truth though, we should conclude then…

Improving specification activities is the single best way to improve your software development project success probability.

And this should make logical sense too, since no matter how good you are at creating software, if you are creating the wrong software, it's irrelevant. But… you could also logically conclude that too much effort in specification is also bad. After all, you do have to eventually build something. So seek to find the balance, but don't skimp.

Everything in Software is Iterative

I do not think that it is possible to fully specify a software product before it is built. And most people agree with me too, as nearly all respected software methodologies involve some level of iteration.

Spec, Design, Build, Test. - Do it over again until you run out of features to add or money to spend.

Everyone does this. The primary difference between methodologies is merely the size of the iteration. Some like them big (think waterfall), some like them small (think agile). But regardless of iteration size, are your spec docs changed during an iteration? Are spec changes controlled? Here's a test to help answer that question. When you finished your last iteration, did your specification match the product? If it does, it was either a really good spec or it was alive.

The Difference that a Living Spec Makes

If your spec has been properly maintained, then it is alive. It's a living document. It has an owner. Someone on the team who actually cares about it enough to feed it, water it, and make it grow. When you update a spec, then design and build against the change, your changes are more thorough. That's because when you open the spec, you are again immersed into the application as a whole, not just the specific issue that brought about the need for a change in the original spec. The cross references and related functionality are again brought to the forefront of your mind and you start to analyze the change in its entirety and how it affects all of the application stakeholders.

The importance of keeping a living spec increases with the size and complexity of your software. But even if your team is me, myself and I, you will create better software if you write specs and maintain them throughout the life of your application.

So don't neglect your specs! Give that doc a pulse and you will end up with better software.

Brian

Developers, Developers, Developers!

I wonder if Microsoft developers ever feel underappreciated. Having never worked there I can't say first hand, but judging from the antics of their chief exec, I'd bet not. Not every exec responsible for software endeavors share Mr. Ballmer's enthusiasm for our kind. If you bounce around long enough in this field, you will inevitably get a taste of both the yin and the yang of executive flavor for which I refer. In my case, the past employers that fall into the opposite spectrum of Mr. Ballmer are in that camp simply because they didn't know any better. Software, or should I say, the importance of software has a way of sneaking up on an exec. One day its pencils, ten keys, and typewriters, and then BAM! - You're in a meeting to negotiate this year's accounting system license agreement. Well, maybe I am exaggerating a bit, but for non-tech leaders, this creep happened, and some adapted better than others. And it's sometimes hard to fault them for their attitude towards developers. In the early days where their opinions were formed; it was all about their experiences. If their teams were good, and they successfully implemented software based systems that actually worked without first failing over and over, the bosses get the impression that it's easy and anyone should be able to do it. On the other hand, if their teams were lousy, and repeatedly failed, well, those developers are certainly not worthy of any kudos either. And then there is the yang for the yin. The exec that has presided over this technology creep with foresight. The creep was there for him too, but he creeped with intent. This exec recognized the competitive advantages that his software systems could achieve his company. This exec recognized the need to ensure that his development teams were better than his competitors and that this competitive advantage could significantly affect the organization's success. This guy is real and he's in Ballmer's camp.

For leaders who have "grown up" so-to-speak within the software process, their opinions are based on other factors, because their experiences were different. They understand far better than their non technical counterparts the great challenges (and great enthusiasms) of the software creation process. They understand developers, and understand that not all developers are created equal. They understand that while great coding might arguably be considered an individual endeavor, great software is definitely a team sport. The successful leaders in this group are also in Ballmer's camp.

In software, a good team of developers stand a chance of overcoming methodology weaknesses, dysfunctional project managers, and impossible schedules and yet still pull off miraculous success. But the best of everything else with bad developers… there's gonna be a big thud when the gantt hits the windshield.

So embrace your developers. Give em' a hug. Hell, stand center stage and at the top of your voice yell! Developers, Developers, Developer!

Brian

So the doorbell rings and I answer it. On the other side of the door is the Ice Cream guy. That's all that Schwans sells right? So I tell him before he says a word that I am not interested in buying any Ice Cream. "But wait" he says, while moving to stop the door from closing with his foot. "We sell more than just Ice Cream. Do you like pizza?" Well… yeah I like pizza, and so the discussion gets a new life. He then hands me his product catalog and lo and behold, they sell lots of things I like. So I start flipping through the pages and rattle off a few items. While I am doing this, he whips out his hand held device and starts punching 3 digit codes that correspond to the items I'm mentioning. When I finish, he goes out to his truck, brings back my items, and asks me how I'd like to pay. I say credit card, half expecting him to tell me that he can't take a credit card, I mean… he's at my front door for Pete's sake. "No problem" he says. I hand him my card and on the side of that same hand held device he slides my card through. And out from the very small wireless receipt printer strapped to his leg appears my receipt. I didn't even notice that printer at first, and I am usually pretty observant of electronic devices. He hands me my receipt and says his goodbyes. I close the door, and I am happy. And it's not because of all the good food I just bought that requires little to no cooking. That makes my wife happy. No, I am in la la land thinking about just how much fun it must have been to create the software application on that hand held. 3 digit codes, wireless printers, credit cards. Fun!

Yep, it happens to me all the time. I see software applications and I start wondering what it would have been like to create it, and how I might have done things better. This is because to me, few things are more fun than making software. Sure there are steps in the software process that are decidedly "funner" than others, but all in all, it's a blast, from specs to release. Fortunately I have been able to build lots of software through the years. The first time I got to write code that people actually used, I remember thinking, I can't believe I am actually getting paid to do this. For full disclosure, I wasn't paid much, but back then I probably would have worked for free if my boss had told me he couldn't afford to pay me.

I don't know how I would have ended up spending my days (and nights) had the solid state transistor never been invented, or had I been born 100 years earlier, but as a result of the advent of the modern computer and its solid entrenchment into all walks of business and commerce, I get to build great software for a living. So whenever the sun sets on my career and I take up the rocking chair, there is a very good chance that I too will get to udder the famous words of Thomas Edison.

"I never worked a day in my life. It was all fun".

And if the Schwan's guy ever shows up at your place, forget the pizza and go directly for the Black Jack Cherry Ice Cream. Product # 362.

Brian